[PATCH] Consolidate build groups to reduce time taken on single-target builds

[Andrew Bartlett]

On Sun, 2013-05-26 at 16:29 +0200, Stefan (metze) Metzmacher wrote:
> Hi Andrew,
> 
> >> So to summarize, I have a few complaints:
> >>
> >> Long startup time: No idea where the time is spent, but 6
> >> seconds on a clean build seems really excessive.
> >>
> >> Always compile everything before linking: I'm told that this
> >> is due to broken dependencies, but I haven't followed this
> >> exactly. I had thought that the main advantage of waf over
> >> autoconf is that it is perfect with dependencies, so I
> >> probably have just not listened right. If waf's way to
> >> dependency perfection is to compile almost everything
> >> always, I think that should be doable with autoconf as well.
> >>
> >> Compile individual C files: It seems not possible without a
> >> gross hack to compile just one file on demand without all
> >> that long startup phase etc.
> >>
> >> Change CFLAGS between runs: Nobody has commented on this one
> >> so far.
> >>
> >> The first two are certainly the most annoying ones, but
> >> probably the two that are hardest to fix. If Tridge failed
> >> to fix something (startup time), this is my very definition
> >> of "unfixable". So I can only ask everybody to keep the old
> >> build system around.
> > 
> > Volker,
> > 
> > I think I may have a solution that should address your first three
> > complaints, at least in part. 
> >  
> > This patch seems to dramatically reduce the time spent on single target
> > builds.
> > 
> > Groups were added to our waf build for stability during early
> > development, as dependency information was first imported from the
> > previous autoconf/perl based build system. 
> > 
> > I don't think we need this distinction in the main build of C files
> > into .o, and when linking these into binaries, because the invocation of
> > these tools is very well defined, and we will find any missing inputs
> > very quickly. 
> > 
> > As such, I've removed the libraries and binaries targets, consolidating
> > them with 'main'  
> > 
> > By making this change, a build of smbtorture only on a clean tree drops
> > from 3778 to 2489 targets, and much of the expensive linker stage is
> > skipped.  The time for a null build of smbtorture only also drops from
> > 4.673s to as low as 2.499s on my laptop.
> > 
> > As your work pattern is to work on a single binary and recompile it,
> > rather than they entire project, this should benefit you in particular,
> > and I hope it resolves to some extent your first complaint.  It also
> > addresses your second complaint, because you are correct we were doing
> > way to much when asked to compile a single binary.  
> > 
> > While I don't directly assist you with your third issue in being able to
> > directly compile only 1 C file, by allowing targeting of a single object
> > this should be at least a little more practical, as only files that the
> > target binary or library use will be recompiled.
> > 
> > I can't offer a simple solution on the CFLAGS issue, I think I have made
> > good progress towards your primary concerns, and I hope you can confirm
> > that these have assisted.  I certainly was surprised to see how much it
> > helped!
> > 
> > Jelmer/Metze:
> > 
> > Please review/push if appropriate.
> 
> It fails autobuild as the source3 make test requires the python bindings,
> but now source3/Makefile-smbtorture only generates smbtorture.
> 
> Traceback (most recent call last):
>   File
> "/memdisk/metze/W/b668620/samba3/source4/scripting/bin/samba_dnsupdate",
> line 41, in <module>
>     import samba
>   File "/memdisk/metze/W/b668620/samba3/bin/python/samba/__init__.py",
> line 27, in <module>
>     import samba.param
> ImportError: No module named param
> Unable to update hostname into
> /memdisk/metze/W/b668620/samba3/source3/st/dns_host_file at
> /memdisk/metze/W/b668620/samba3/selftest/target/Samba3.pm line 1175.
> make: *** [test] Error 1
> 
> Otherwise it looks good to me.

The attached patch fixes the issue, by simply building everything in the
autoconf make test (as it is clear that this is what is now required,
and build groups were masking the missing dependency). 

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-build-Build-all-of-samba-in-autoconf-make-test.patch
Type: text/x-patch
Size: 1165 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20130527/3296612f/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-build-Remove-binaries-and-libraries-build-groups.patch
Type: text/x-patch
Size: 3604 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20130527/3296612f/attachment-0001.bin>