[PATCH] Consolidate build groups to reduce time taken on single-target builds

Stefan (metze) Metzmacher metze at samba.org
Sun May 26 08:29:08 MDT 2013


Hi Andrew,

>> So to summarize, I have a few complaints:
>>
>> Long startup time: No idea where the time is spent, but 6
>> seconds on a clean build seems really excessive.
>>
>> Always compile everything before linking: I'm told that this
>> is due to broken dependencies, but I haven't followed this
>> exactly. I had thought that the main advantage of waf over
>> autoconf is that it is perfect with dependencies, so I
>> probably have just not listened right. If waf's way to
>> dependency perfection is to compile almost everything
>> always, I think that should be doable with autoconf as well.
>>
>> Compile individual C files: It seems not possible without a
>> gross hack to compile just one file on demand without all
>> that long startup phase etc.
>>
>> Change CFLAGS between runs: Nobody has commented on this one
>> so far.
>>
>> The first two are certainly the most annoying ones, but
>> probably the two that are hardest to fix. If Tridge failed
>> to fix something (startup time), this is my very definition
>> of "unfixable". So I can only ask everybody to keep the old
>> build system around.
> 
> Volker,
> 
> I think I may have a solution that should address your first three
> complaints, at least in part. 
>  
> This patch seems to dramatically reduce the time spent on single target
> builds.
> 
> Groups were added to our waf build for stability during early
> development, as dependency information was first imported from the
> previous autoconf/perl based build system. 
> 
> I don't think we need this distinction in the main build of C files
> into .o, and when linking these into binaries, because the invocation of
> these tools is very well defined, and we will find any missing inputs
> very quickly. 
> 
> As such, I've removed the libraries and binaries targets, consolidating
> them with 'main'  
> 
> By making this change, a build of smbtorture only on a clean tree drops
> from 3778 to 2489 targets, and much of the expensive linker stage is
> skipped.  The time for a null build of smbtorture only also drops from
> 4.673s to as low as 2.499s on my laptop.
> 
> As your work pattern is to work on a single binary and recompile it,
> rather than they entire project, this should benefit you in particular,
> and I hope it resolves to some extent your first complaint.  It also
> addresses your second complaint, because you are correct we were doing
> way to much when asked to compile a single binary.  
> 
> While I don't directly assist you with your third issue in being able to
> directly compile only 1 C file, by allowing targeting of a single object
> this should be at least a little more practical, as only files that the
> target binary or library use will be recompiled.
> 
> I can't offer a simple solution on the CFLAGS issue, I think I have made
> good progress towards your primary concerns, and I hope you can confirm
> that these have assisted.  I certainly was surprised to see how much it
> helped!
> 
> Jelmer/Metze:
> 
> Please review/push if appropriate.

It fails autobuild as the source3 make test requires the python bindings,
but now source3/Makefile-smbtorture only generates smbtorture.

Traceback (most recent call last):
  File
"/memdisk/metze/W/b668620/samba3/source4/scripting/bin/samba_dnsupdate",
line 41, in <module>
    import samba
  File "/memdisk/metze/W/b668620/samba3/bin/python/samba/__init__.py",
line 27, in <module>
    import samba.param
ImportError: No module named param
Unable to update hostname into
/memdisk/metze/W/b668620/samba3/source3/st/dns_host_file at
/memdisk/metze/W/b668620/samba3/selftest/target/Samba3.pm line 1175.
make: *** [test] Error 1

Otherwise it looks good to me.

metze

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20130526/d63b39d9/attachment.pgp>


More information about the samba-technical mailing list