proposal: merge waf build of s4 to master

[tridge at samba.org]

Hi All,

A couple of weeks ago I proposed that any new build system for Samba
be adopted in stages:

  http://lists.samba.org/archive/samba-technical/2010-March/070065.html

I'd now like to propose that the waf build of Samba4 move to stage 2,
which means merging it into master.

As is explained in the staging proposal, at stage 2 the existing build
system would still be the primary build system. Developers who are not
interested in using the waf based build would have no responsibility
to update the waf build scripts. I will keep the waf scripts up to
date, along with anyone else who is interested in helping out (I have
received patches from several people for the waf-wip branch, so it
seems there is interest).

The merge would change no files that affect the current build of
Samba3 or Samba4. Only additional files are added (primarily the
wscript files), and those are what allow the waf build to work.

Once merged, developers would choose which build system to use by
either using ./autogen.sh for the old build system, or
./autogen-waf.sh for the new system.

Once the build system is chosen, then builds proceed as normal, with
"./configure.developer && make" working fine with both build systems.

The build farm would continue to build with both build systems, as it
does now. Once the s3 waf build that Kai is working on has progressed
to the point where it is useful, I expect we will add a s3-waf build
in the build farm as well.

The reason I am proposing to merge into master now is that the waf
based build has gotten to the point where it is very usable for
general Samba4 development and usage. In fact, I've found using the
old build system to be quite cumbersome by comparison, and I really
miss the accurate dependencies, fast builds and small binaries when I
do work in master.

It also has become clear that the primary problems remaining with the
waf based build are in fact bugs in the old build system, so it seems
most appropriate to try to fix these while working in master.

The sorts of bugs I am talking about are:

 - the hiding of debug messages. We in fact have quite a few things
   failing in "make test" that are never shown. This is because with
   the old build system debug.c gets linked into programs multiple
   times (especially via python modules). This results in DEBUG(1)
   messages not being shown, as the DEBUGLEVEL global gets
   instantiated multiple times in the code but only set once. That
   means some error messages just don't get shown even though they
   happen.

 - duplicated talloc contexts. Andrew and I have found bugs due to
   duplicate globals such as the iconv convenience globals. We have
   fixed some of these in master, but more remain. The python.winreg
   failure in the build farm for the waf-wip branch seems to be one of
   these sorts of bugs, and I am still trying to chase it down.

You can see these types of bugs by this message in the waf-wip build
when running make test:

    ndr_pull_error(11): ndr_pull_relative_ptr1 rel_offset(758459697) > ndr->data_size(746)

that is caused by an ndr_pull of the text form of a security
descriptor. We never see this message with the old build system as
debug.c duplication means it is not printed. If you attach with gdb to
a Samba built from the old build system you'll see that the error does
happen, it just doesn't get reported.

Additionally, with the waf build, we end up with a much smaller set of
binaries (a bit over 100M, as opposed to about 1.1G with the old
system). Part of this is the much more extensive use of shared
libraries, but part of it is because the waf build goes through a
"object reduction" stage where it removes duplicate object files from
libraries and binaries. This means these hidden bugs now surface, and
we have a chance of fixing them.

The waf based build is certainly not complete yet. For example, I have
yet to do the compiler hooks to use the Irix and Tru64 compilers, and
there is work to do on packaging to make is easier for distros to
build with waf. I also need to document the "wafsamba" API that is
used to define the SAMBA_BINARY(), SAMBA_LIBRARY() and related
functions. I have started using pydoctor to produce html documentation
for those, but it is not anywhere near complete. We also need more
developer documentationo how to get the most out of the waf build
system.

I don't think any of these things are reason enough to delay a merge
into master however, with the proviso that the old build system is
still the 'reference' build system that we recommend.

I'd like to thank everyone who has helped get the waf build to this
point. Quite a few people have done testing, and have suggested useful
improvements. I've also had a lot of help from Thomas Nagy, the author
of waf, who has been very generous with his time.

Cheers, Tridge