[distcc] very slow with many machines

[Jean Delvare]

On Wed, 15 Sep 2004 03:56:42 +0200, Stefan Heinrichsen wrote
> Hello,
> 
> I try to use distcc in a group of about 40 PCs. All PCs have a P4 2.6
> Ghz with 1GB Ram and are connected with 100MBit cards.
> I tried different values for -j.. from 70 to 150 but the compilie 
> time always is the same. Without distcc it is about 10% _faster_. In 
> the logs of the machines I can see that all hosts are used.
> 
> Where could be the problem? Is the network to slow to use so many hosts?

I think it is a known fact that distcc doesn't scale very well with such a
high number of hosts, because of the fact that a single host has to handle the
task distribution, which constitutes the bottleneck.

http://distcc.samba.org/doc/lca2004/distcc-lca-2004.html#scalability

"Performance tends to plateau between ten and twenty machines."

So you probably shouldn't use a value of -j above 40, and lower values down to
20 may even give better results. At any rate, make sure that the machine you
are running the compilation on is not listed as a compilation server (i.e.
DISTCC_HOSTS doesn't contain "localhost") so that it can focus on job
distribution and not actual compilation.

As a side note, if your 100Mbps network is not switched, it may act as a
bottleneck as well. I don't have any figure, but there are plenty of network
usage monitoring tools out there so you should be able to check if this is the
case. If on the other hand your network is fully switched, I'd say it cannot
possibly slow down the compilation (unless something else is saturating it at
the time you run your compilation).

Hope that helps,

-- 
Jean Delvare
http://khali.linux-fr.org/