[distcc] More efficient multiple file compiles?

[Benjamin R. Haskell]

On Tue, 28 Apr 2009, Tom Holly wrote:

> Those assumptions are correct. I've had distcc built and installed for a 
> while and it's working wonderfully. Now I'm trying to incorporate make 
> -j and here is the problem:
> 
> I ran "pump make -j12 CC=distcc\ gcc all" in the directory where my 
> Makefile and distcc object files reside. Then I did a make install the 
> same way and that all  worked fine.
> 
> Now I want to test this on my own C files and that is what I cannot seem 
> to get working because I don't notice the files being compiled 
> simultaneously.

I'd recommend using environment variables to tell distcc to output more 
useful debugging information for your case.  Specifically, I'd look at 
using DISTCC_VERBOSE, DISTCC_LOG, and maybe DISTCC_SAVE_TEMPS[1].

For more general 'parallel make' advice, a thread from 
help-make at gnu.org[2] had some relevant discussion.  Specifically, one of 
the suggestions[3] was to augment your targets with timestamp-generating 
commands:

e.g.:

AT = @date +"$@: %s" ;

all:	foo bar

foo:
	$(AT)sleep 5

bar:
	$(AT)sleep 10


But, maybe that could be expanded upon further, by creating a small 
wrapper for distcc that outputs timing information (tested):

==== put this in your path as 'distcc-with-timing' ====
==== (at http://benizi.com/distcc-with-timing ) ====
#!/bin/zsh
_log=${DISTCC_TIMING_LOG=$DISTCC_DIR/timing-log}
_args=($argv)
function _tag () { 
    printf "%s %s%s\n" `date +%s` $1 "${DISTCC_TIMING_VERBOSE+ $_args}" >> $_log
}
_tag START-JOB
distcc $argv
_tag FINISH-JOB
==== end of distcc-with-timing ====

$ make -j4 CC=distcc-with-timing
# then inspect $DISTCC_DIR/timing-log

$ DISTCC_TIMING_LOG=/tmp/put-timing-log-here DISTCC_TIMING_VERBOSE=1 make -j4 CC=distcc-with-timing
# then inspect /tmp/put-timing-log-here

If you find more than one START-JOB line in a row, it's running in 
parallel.

Best,
Ben

[1] http://distcc.googlecode.com/svn/trunk/doc/web/man/distcc_1.html#TOC_20
[2] http://www.mail-archive.com/help-make@gnu.org/msg07297.html
[3] http://www.mail-archive.com/help-make@gnu.org/msg07304.html
[4] http://benizi.com/distcc-with-timing