proposal: merge waf build of s4 to master

[Matthieu Patou]

Tridge,
On 02/04/2010 09:13, tridge at samba.org wrote:
> Hi All,
>
> A couple of weeks ago I proposed that any new build system for Samba
> be adopted in stages:
>
>    http://lists.samba.org/archive/samba-technical/2010-March/070065.html
>
> I'd now like to propose that the waf build of Samba4 move to stage 2,
> which means merging it into master.
>
> As is explained in the staging proposal, at stage 2 the existing build
> system would still be the primary build system. Developers who are not
> interested in using the waf based build would have no responsibility
> to update the waf build scripts. I will keep the waf scripts up to
> date, along with anyone else who is interested in helping out (I have
> received patches from several people for the waf-wip branch, so it
> seems there is interest).
>
> The merge would change no files that affect the current build of
> Samba3 or Samba4. Only additional files are added (primarily the
> wscript files), and those are what allow the waf build to work.
>
> Once merged, developers would choose which build system to use by
> either using ./autogen.sh for the old build system, or
> ./autogen-waf.sh for the new system.
>
> Once the build system is chosen, then builds proceed as normal, with
> "./configure.developer&&  make" working fine with both build systems.
>
> The build farm would continue to build with both build systems, as it
> does now. Once the s3 waf build that Kai is working on has progressed
> to the point where it is useful, I expect we will add a s3-waf build
> in the build farm as well.
>
> The reason I am proposing to merge into master now is that the waf
> based build has gotten to the point where it is very usable for
> general Samba4 development and usage. In fact, I've found using the
> old build system to be quite cumbersome by comparison, and I really
> miss the accurate dependencies, fast builds and small binaries when I
> do work in master.
>
> It also has become clear that the primary problems remaining with the
> waf based build are in fact bugs in the old build system, so it seems
> most appropriate to try to fix these while working in master.
>
> The sorts of bugs I am talking about are:
>
>   - the hiding of debug messages. We in fact have quite a few things
>     failing in "make test" that are never shown. This is because with
>     the old build system debug.c gets linked into programs multiple
>     times (especially via python modules). This results in DEBUG(1)
>     messages not being shown, as the DEBUGLEVEL global gets
>     instantiated multiple times in the code but only set once. That
>     means some error messages just don't get shown even though they
>     happen.
>
>   - duplicated talloc contexts. Andrew and I have found bugs due to
>     duplicate globals such as the iconv convenience globals. We have
>     fixed some of these in master, but more remain. The python.winreg
>     failure in the build farm for the waf-wip branch seems to be one of
>     these sorts of bugs, and I am still trying to chase it down.
>
> You can see these types of bugs by this message in the waf-wip build
> when running make test:
>
>      ndr_pull_error(11): ndr_pull_relative_ptr1 rel_offset(758459697)>  ndr->data_size(746)
>
> that is caused by an ndr_pull of the text form of a security
> descriptor. We never see this message with the old build system as
> debug.c duplication means it is not printed. If you attach with gdb to
> a Samba built from the old build system you'll see that the error does
> happen, it just doesn't get reported.
>
> Additionally, with the waf build, we end up with a much smaller set of
> binaries (a bit over 100M, as opposed to about 1.1G with the old
> system). Part of this is the much more extensive use of shared
> libraries, but part of it is because the waf build goes through a
> "object reduction" stage where it removes duplicate object files from
> libraries and binaries. This means these hidden bugs now surface, and
> we have a chance of fixing them.
>
> The waf based build is certainly not complete yet. For example, I have
> yet to do the compiler hooks to use the Irix and Tru64 compilers, and
> there is work to do on packaging to make is easier for distros to
> build with waf. I also need to document the "wafsamba" API that is
> used to define the SAMBA_BINARY(), SAMBA_LIBRARY() and related
> functions. I have started using pydoctor to produce html documentation
> for those, but it is not anywhere near complete. We also need more
> developer documentationo how to get the most out of the waf build
> system.
>
> I don't think any of these things are reason enough to delay a merge
> into master however, with the proviso that the old build system is
> still the 'reference' build system that we recommend.
>
> I'd like to thank everyone who has helped get the waf build to this
> point. Quite a few people have done testing, and have suggested useful
> improvements. I've also had a lot of help from Thomas Nagy, the author
> of waf, who has been very generous with his time.
>
> Cheers, Tridge
>    
Well, the reduction of the size plus undercovering hidden bugs makes for 
me a very plus I'm eager to see it in master.

Matthieu.