a proposal for Samba 3.5

[tridge at samba.org]

This is a draft proposal for Samba 3.5. I am planning on presenting
this proposal as part of my Samba4 presentation at SambaXP next week,
but I thought it would be useful to get some feedback before then.

For those of you who don't know, the Samba4 tree is available via
anonymous cvs in the module "samba4" on cvs.samba.org. Use the
following command to take a look.

  cvs -z3 -d :pserver:cvs at pserver.samba.org:/cvsroot co samba4

Cheers, Tridge

-----------------------------------------

A proposal for Samba 3.5
------------------------

The Samba4 effort has been going for about 1 year now, and has made
very good progress, but so far has gained very little interest from
other than a few core developers. In fact, nearly all of the work on
Samba4 has been done by just a couple of people. I believe there are a
variety of reasons for this, but the core reason is that Samba4 is
seen as "too far off", and many Samba developers need more immediate
results.

To address this problem I would like to propose that we start a
Samba-3.5 effort, which will aim to integrate the parts of the Samba4
codebase into the Samba3 tree. The parts that I propose integrating
are those that are both already reasonably complete and are easily
separable.

In particular propose integrating the following parts of Samba4 into
Samba3, to form the basis of Samba 3.5.

   - the SMB client library
   - the RPC client library and client side IDL code generation
   - the ldb database
   - the smbclient code
   - the smbtorture test suite
   - the schannel code
   - some of the directory restructuring, but not all

I'll try to justify each part and give a rough explanation of what is
involved.

SMB client library
------------------

The Samba4 SMB client library is vastly better than the Samba3
library. The use of explicit structures for each type of request, with
every field of every request covered makes for a library that is much
easier to understand and much more complete is what it can do. 

Integrating the Samba4 SMB client library into the Samba3.5 codebase
should be quite easy. To do this I think we should _not_ use the
approach of creating shims that look like the Samba3 calls, although
that would certainly be easier to do. Instead I propse that we convert
the Samba3.5 code to directly call the libcli/raw/ raw SMB
interface. It is a much more verbose interface, but it is also an
interface that makes it extremely clear what wire calls are really
being made.

The major things that would need to be done before integrating with
Samba3.5 are the addition of the missing RAP calls (which are in
Samba3 but not yet in the Samba4 library) and the addition of kerberos
authentication routines. We could also consider skipping the RAP calls
and instead make things lile "smbclient -L" use the equivalent RPC
calls which are already in Samba4 (there are several advantages to
using RPC calls instead of the ancient RAP calls).

RPC client library
------------------

The RPC client library in Samba4 is much better designed than the
Samba3 equivalent, and is based on auto-generated code from IDL,
making it mch easier to develop further. It also fully supports RPC
over TCP and supports far more functions and information levels than
the equivalent Samba3 code.

It should be relatively easy to replace most uses of RPC in Samba3.5
with the Samba4 RPC library, except perhaps for the rpcclient tool. I
would suggest removing the rpcclient tool in Samba3.5, and instead
work towards writing a new tool with equivalent functionality
(possibly even basing the new tool on Tim's python or perl interfaces
to the library).

The major work that needs to be done is in the winbindd daemon and the
net tool, plus the domain authentication backend to smbd. It should be
a fairly straightforward conversion in each of these cases, although
certainly is not trivial. Once again I would suggest doing a full
conversion to the new interfaces rather than writing shim functions to
the old interfaces. We won't get the full advantage of the new
interfaces while we are still using the old syntax.

The ldb database
----------------

The ldb database provides a much cleaner and simpler way of managing
data that is naturally field/attribute based than tdb does. It also
provides us with a way to backend to either tdb or LDAP without
changing any application code. I would suggest replacing a number of
the existing tdb databases with ldb, especially the long term
read-heavy databases such as the printers database, secrets.tdb and
the default passdb database.

ldb is not complete yet, so this part of the Samba3.5 effort might
need to wait until ldb is finished, but I think that the gains from
using ldb are well worth the effort involved.

The smbclient code
------------------

The biggest user of the existing SMB client library is the "smbclient"
tool. I suggest taking the whole smbclient tool from Samba4 and moving
it as-is to Samba3. This also gains us a number of new and useful
smbclient commands, such as "deltree" and "allinfo".

We will need to do some work in adding some options that are currently
missing in the Samba4 smbclient, such as the -L option for browsing
(which I suggest we implement using RPC calls, possibly falling back
to RAP if needed) and the kerberos authentication methods.

The smbtorture test suite
-------------------------

This one should be a no-brainer. The Samba4 test suite is vastly
better than the Samba3 suite in every respect. Once we have the new
SMB and RPC client libraries then moving across the torture suite will
be very easy.

The schannel code
-----------------

The Samba4 schannel code is much cleaner and simpler than the Samba3
code, and uses an interface that is almost identical to our existing
NTLMSSP code. Using the Samba4 schannel code in Samba3.5 should be
quite easy, once the SMB and RPC libs are across.

Directory restructuring
-----------------------

I suggest we adopt the Samba4 directory structure for the client
library, RPC library and torture code, but perhaps not for the core
smbd code just yet. I think this will make maintainence between the 3
trees easiest.

What of Samba4?
---------------

I am not proposing that Samba4 be abandoned, and in fact I firmly
think that the server structure in Samba4 is a huge improvement over
the Samba3 structure and that we should move to Samba4 as soon as we
can. I think that by doing a Samba3.5 release that Samba4 will
actually happen sooner than if we don't, because more developers will
be exposed to the way that some key parts of the Samba4 code is
structured, allowing them to contribute to Samba3 more easily.

Timescale?
----------

I would hope that we could have a alpha of Samba 3.5 out within the
next 6 months with the above changes, but this critcally depends on
whether some more developers get involved.