[distcc] [Fwd: "Busy" waiting when using distcc/ssh (and SMP question)]

Marcelo Matus mmatus at dinha.acms.arizona.edu
Sun Jul 6 09:51:47 GMT 2003


Hello:

First, let me say that I find distcc amazing, I am using it
with three machines, including a laptop, a single CPU and
and dual smp machine... it works great!

However, I have noticed that when I use the 'ssh' connection,
the 'master' machine shows several very busy 'distcc' processes
which seems to be waiting for the remote 'ssh' compilation process
to finish, ie, it seems 'distcc' is doing.

1.- preprocess the .cc file.
2.- send it through ssh (I can see the new 'ssh' processes starting)
3.- start the compilation in the remote machine (I can see it remotely)
4.- wait in 'busy' mode, using as much CPU is available, while the
     remote process is compiling....
5.- read back the compilation result.

The problem is not that evident when you compile plain C or simple C++
code, but when you compile very ugly  C++ code, with lot of templates,
then the remote compilation takes a while (very long while), and then
you can see the local 'distcc' processes starving the master machine,
and competing with the local compilations :(.


Note that this does not happen when I use the 'distccd' server, only when
I use the 'ssh' connection, and it is mainly visible if the remote 
compilation
is long enough.

I also noticed that in the source code the 'distccd' connection use
non-bloking files and the non-blocking 'dcc_select_for_read' routine,
but not the 'ssh' one.... can this be the  problem?, or there is
a intrinsic 'blocking' behaviour in the ssh tunnel...

One way to test this, is to wrap the gcc/g++ compile with a command 
like this:

-------------  /usr/local/bin/slowgcc ---------

#!/bin/sh
sleep 5
gcc $*

-------------  /usr/local/bin/slowgcc ---------

and run some code using

time make CC="distcc /usr/local/bin/slowgcc" -j4

and compare both settings:

1.-  export DISTCC_HOSTS="localhost remotehost" 
2.-  export DISTCC_HOSTS="localhost @remotehost"

In the first case, only the local compilations using 
the localhost CPU.

In the second case, in the localhost you must see one 
or two extra 'distcc' processes competing very hard
for the CPU.


And now, the SMP question:

As I mentioned, I have a dual machine, and I tried the following

export DISTCC_HOSTS="singlecpu dualcpu dualcpu laptop"

ie, I put the name of the dual machine twice in the hosts list to force
twice allocation of jobs, but it didn't work.

'distcc' always send the same number of task to every machine.

Is there another way to tell 'distcc' to send more jobs to a given machine?


Marcelo












More information about the distcc mailing list