[distcc] when recursive make becomes the bottleneck...

J.T. Conklin jtc at acorntoolworks.com
Tue Jan 28 21:54:56 GMT 2003

I'm in the midst of converting the build infrastructure at work from
using recursive gmake to scons (www.scons.org), and thought it might
be useful to share my experiences.

The gmake build scheme we have been using is much like many others I
have encountered over the years.  A list of subdirectories are built
sequentially by iterating over the list and spawning a new make for
each.  One problem with this approach is that the end of each module/
subdirectory becomes a synchronization point.  For example, when there
are fewer files to compile (in a given module) than you have processes
in your pool, you're not taking full advantage of your build farm.

This deficiency can be addressed by rewriting your makefiles to avoid
recursion (Cf Peter Miller's paper "Recursive Make Considered Harmful"
(http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html)) or by
using another tool (like scons, jam, etc.) that constructs a full
dependency graph before building.

Here's a real-world example that demonstrates the advantages.  Our
source code is about 600,000 lines of C++, with a small amount of C
and assembly thrown in.  There are sixteen libraries, a dozen or so
executables (including four very large ones), etc.

On a AMD Athlon 2000+, using our old recursive gmake with -j4 (mainly
to hide i/o latency):
        real:   21m26.052s
        user:   19m44.063s
        sys:    1m28.127s

On a distcc build farm including the above AMD machine, plus 2 1.0GHz
PIII machines and 2 Dual 1.0GHz PIII machines with gmake -j20:
        real:   10m33.496s
        user:   5m34.436s
        sys:    1m0.731s

Our scons build on the single AMD machine with -j4:
        real:   13m52.052s
        user:   12m54.470s
        sys:    0m43.463s

And finally, scons -j20 on the distcc build farm:
        real:   3m47.938s
        user:   3m16.116s
        sys:    0m35.347s

While distcc alone more than doubled the performance, distcc plus a
build scheme that takes full advantage of build parallelism really
lets it shine by speeding up the build more than 5 times.


J.T. Conklin

More information about the distcc mailing list