[distcc] Re: --enable-gnome

[Martin Pool]

On 18 Jun 2003, Dimitri Papadopoulos-Orfanos <papadopo at shfj.DECOY.cea.NOSPAM.fr> wrote:
> I wrote:
> 
> >I've tried "make -j5" and still get lots of "Starved" messages although 
> >we have more than 20 hyperthreaded dual-Xeon machines, that is, more 
> >than 80 "processors".
> >
> >An yet I'm pretty sure compilations are correctly distributed. We had 
> >tested that a few weeks ago, and we still see tremendous acceleration 
> >when using distcc.
> 
> I've attempted to build distcc-2.7 with 'make -j30'. I'm running 
> distccmon-text this way, in another terminal window:
> 	distccmon-text 1
> 
> Please find attached the output of distcc with DISTCC_VERBOSE defined in 
> file distcc.log.
> 
> Also find attached the output of distccmon-text in file distccmon.out, 
> you'll see lots of "Starved" messages.

Thankyou for sending the log. 

Many of the commands are invocations of -E to run just the
preprocessor.  This is happening because of ccache wanting to look at
the .i file to determine if the compilation is cached.

You can see that all the occurrences are correlated in this way:

distcc[21482] (dcc_scan_args) scanning arguments: /usr/bin/gcc -DHAVE_CONFIG_H -D_GNU_SOURCE -I./src "-DSYSCONFDIR="/usr/local/etc"" -g -O2 -W -Wall -W -Wimplicit -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -c -E src/compile.c
distcc[21482] (dcc_scan_args) -E call for cpp must be local
distcc[21482] (dcc_lock_host) /tmp/distcc_marehaut.shfj.cea.fr_16044/lock_cpu_localhost_0 is busy
distcc[21482] (dcc_lock_host) /tmp/distcc_marehaut.shfj.cea.fr_16044/lock_cpu_localhost_1 is busy
distcc[21482] (dcc_lock_host) /tmp/distcc_marehaut.shfj.cea.fr_16044/lock_cpu_localhost_2 is busy
distcc[21482] (dcc_lock_host) /tmp/distcc_marehaut.shfj.cea.fr_16044/lock_cpu_localhost_3 is busy
distcc[21482] (dcc_note_state) note state Starved, file , host 
distcc[21482] (dcc_lock_pause) nothing available, sleeping 2s...

Invocations with -E can only be run locally, and they count against
the process limit of localhost.  Therefore when more than 4 (by
default) preprocessors try to run, some of them are starved of CPU
time. 

In particular this will happen when you run with a high -j level and
there are many ccache hits.  There is just not much work that *can* be
distributed, and so most of the time will be spent trying to get all
the preprocessing done locally.

I'm not sure what, if anything, should be done.  You already seem to
have localhost not in the host list, so it will only be used when
absolutely necessary.  Everything that can be distributed will be.

You could raise the process limit on localhost, but I don't suppose
going to 30 would help.

I suppose distcc could disregard the limit when preprocessing locally,
on the grounds that cpp is not CPU-intensive, but I'm not sure that's
a good idea.

Perhaps the backoff from CPU starvation should be better -- at the
moment it is a fairly dumb sleep/busywait.  This is adequate and
simple for the case I had in mind of just limiting the link phase, but
for the situation you discovered where there will often be far too
many processes it may be a bit wasteful.

Thanks,

-- 
Martin