[distcc] Multiple people running distcc at once, using one DISTCC_DIR

[Jean Delvare]

On Mon, 27 Sep 2004 08:04:42 -0700 (PDT), Victor Norman wrote
> I have taken Dan up on his challenge (sort of).  I ran 6 compiles
> simultaneously, each with its own DISTCC_DIR, and with host list
> randomization turned on, and -j20.  What I found is that the compilation
> times increased pretty significantly: from 39 minutes to 60 minutes.
> (...)
> As it was, I had up to 120 
> distcc compilation requests (6 builds times -j20) choosing from 44 
> CPUs.  If I eliminated my slower machines from the list I'd get (up 
> to) 120 simultaneous requests hitting some 22 CPUs.

It's quite obvious why it was slow. You try to run three times as much
processes as you have CPUs available. Since the randomization may be unfair to
some hosts, you can expect some CPUs to get 4 or 5 jobs at any given time. And
you are even thinking of running of half the nukber of CPUs!

Your first try should have been with a 1:1 ratio between CPUs and processes.
44 CPUs, 6 compilations -> 6 or 7 jobs/compilation. Try with -j6 or -j7 and
see how it goes.

> NOTE: I'm not using 2.17 with its timeouts, etc.  Should I use it? 
>  Will it affect my results, you think?

Timeouts should help much, although I am not certain that it is properly
implemented in 2.17. Connections being refused should lead to the next
(ramdom) host being tried, while 2.17 will fallback to localhost IIRC.
Implementing the former method would probably help in your case, and I guess
that Martin would enjoy such a patch.

As a side note, I'm really astonished by the number of people trying distcc
with -j20, -j40 or even more and complaining that it is slower than their
regular compilation. The big win is with the low values of -j. New users
should _always_ try with -j2 and -j3 first, even if they have a large number
of compilation clients. This is the best way to realize how efficient distcc
is. Then, and only then, they should try greater values of -j.

As for most things, increasing numbers and expecting a linear improvement in
return without first thinking of what these numbers really mean is not the
clever choice. Things are never that simple.

-- 
Jean Delvare
http://khali.linux-fr.org/