[distcc] Multiple people running distcc at once,
using one DISTCC_DIR
Victor Norman
vtnpgh at yahoo.com
Mon Sep 27 15:04:42 GMT 2004
Dan, Jean, et al.,
I have taken Dan up on his challenge (sort of). I ran 6 compiles
simultaneously, each with its own DISTCC_DIR, and with host list randomization
turned on, and -j20. What I found is that the compilation times increased
pretty significantly: from 39 minutes to 60 minutes. And, here is why: often a
compilation was trying to compile a file on a machine that was already
compiling the max number of compiles possible. So, the compilation would block
in the TCP listen queue. And it would block, and block, and block. Here is
some example output from 'distccmon-text 3', with eparker's modifications so
that the time is shown with each line of output. What we see is that a file
that takes less than 3 seconds to compile (resolver_stub.c) sits in the TCP
listen queue for more than 2.5 minutes before it is finally compiled (which
takes less than 3 seconds).
[17:01:35] 16445 Compile pkt_fwd_stub.c
angstrom[1]
[17:01:35] 16481 Send resolver_stub.c
marionberry[0]
[17:01:38] 16481 Send resolver_stub.c
marionberry[0]
[17:01:41] 16481 Send resolver_stub.c
marionberry[0]
[17:01:44] 16481 Send resolver_stub.c
marionberry[0]
[17:01:47] 16481 Send resolver_stub.c
marionberry[0]
[17:01:50] 16481 Send resolver_stub.c
marionberry[0]
[17:01:53] 16481 Send resolver_stub.c
marionberry[0]
[17:01:56] 16481 Send resolver_stub.c
marionberry[0]
[17:01:59] 16481 Send resolver_stub.c
marionberry[0]
[17:02:02] 16481 Send resolver_stub.c
marionberry[0]
[17:02:05] 16481 Send resolver_stub.c
marionberry[0]
[17:02:08] 16481 Send resolver_stub.c
marionberry[0]
[17:02:11] 16481 Send resolver_stub.c
marionberry[0]
[17:02:14] 16481 Send resolver_stub.c
marionberry[0]
[17:02:17] 16481 Send resolver_stub.c
marionberry[0]
[17:02:20] 16481 Send resolver_stub.c
marionberry[0]
[17:02:23] 16481 Send resolver_stub.c
marionberry[0]
[17:02:26] 16481 Send resolver_stub.c
marionberry[0]
[17:02:29] 16481 Send resolver_stub.c
marionberry[0]
[17:02:32] 16481 Send resolver_stub.c
marionberry[0]
[17:02:35] 16481 Send resolver_stub.c
marionberry[0]
[17:02:38] 16481 Send resolver_stub.c
marionberry[0]
[17:02:41] 16481 Send resolver_stub.c
marionberry[0]
[17:02:44] 16481 Send resolver_stub.c
marionberry[0]
[17:02:47] 16481 Send resolver_stub.c
marionberry[0]
[17:02:50] 16481 Send resolver_stub.c
marionberry[0]
[17:02:53] 16481 Send resolver_stub.c
marionberry[0]
[17:02:56] 16481 Send resolver_stub.c
marionberry[0]
[17:02:59] 16481 Send resolver_stub.c
marionberry[0]
[17:03:02] 16481 Send resolver_stub.c
marionberry[0]
[17:03:05] 16481 Send resolver_stub.c
marionberry[0]
[17:03:08] 16481 Send resolver_stub.c
marionberry[0]
[17:03:11] 16481 Send resolver_stub.c
marionberry[0]
[17:03:14] 16481 Send resolver_stub.c
marionberry[0]
[17:03:17] 16481 Send resolver_stub.c
marionberry[0]
[17:03:20] 16481 Send resolver_stub.c
marionberry[0]
[17:03:23] 16481 Send resolver_stub.c
marionberry[0]
[17:03:26] 16481 Send resolver_stub.c
marionberry[0]
[17:03:29] 16481 Send resolver_stub.c
marionberry[0]
[17:03:32] 16481 Send resolver_stub.c
marionberry[0]
[17:03:35] 16481 Send resolver_stub.c
marionberry[0]
[17:03:38] 16481 Send resolver_stub.c
marionberry[0]
[17:03:41] 16481 Send resolver_stub.c
marionberry[0]
[17:03:44] 16481 Send resolver_stub.c
marionberry[0]
[17:03:47] 16481 Send resolver_stub.c
marionberry[0]
[17:03:50] 16481 Send resolver_stub.c
marionberry[0]
[17:03:53] 16481 Send resolver_stub.c
marionberry[0]
[17:03:56] 16481 Send resolver_stub.c
marionberry[0]
[17:03:59] 16481 Send resolver_stub.c
marionberry[0]
[17:04:02] 16481 Send resolver_stub.c
marionberry[0]
[17:04:05] 16481 Send resolver_stub.c
marionberry[0]
[17:04:08] 16481 Send resolver_stub.c
marionberry[0]
[17:04:11] 16481 Compile resolver_stub.c
marionberry[0]
[17:04:14] 16517 Preprocess ablink.c
fatboy[0]
[17:04:14] 16519 Preprocess ablink_util.c
fatboy[1]
[17:04:14] 16527 Preprocess avail_stats_api.c
fatboy[2]
[17:04:14] 16540 Preprocess comm_cac_util.c
fatboy[3]
[17:04:14] 16554 Preprocess comm_ft_dtl.c
fatboy[4]
[17:04:14] 16556 Compile commsg_ncci.c
fatboy[5]
[17:04:14] 16573 Preprocess evpl.c
fatboy[6]
[17:04:14] 16580 Preprocess locks.c
fatboy[7]
[17:04:14] 16585 Preprocess ncci.cc sws-build2[0]
And, this happened over and over again. (Note: this wouldn't happen much, if
at all, if file locking worked over NFS and I had all 6 compiles point to one
DISTCC_DIR.)
AND: I'm using my version of distcc-2.16 with some pretty significant
modifications: the distcc code reads in a hosts-info file that is updated with
the load averages on the machines in the hosts file list. It takes the load
averages into account when choosing which host to compile on -- so it avoids
the heavily loaded machines.
So, I suspect the results would have been even worse if my modifications
weren't included.
And, if I had eliminated the slower machines from the hosts list, then I would
get more compilation requests hitting fewer cpus, thus increasing the "Send"
times even more. As it was, I had up to 120 distcc compilation requests (6
builds times -j20) choosing from 44 CPUs. If I eliminated my slower machines
from the list I'd get (up to) 120 simultaneous requests hitting some 22 CPUs.
And: my goal is to have 10 simultaneous compiles (at -j20) work well, so we
would have up to 200 requests simultaneously.
NOTE: I'm not using 2.17 with its timeouts, etc. Should I use it? Will it
affect my results, you think?
Any other ideas on how I can satisfy my requirements (listed in a previous
posting "My big ideas for distcc") -- especially, "The system performance
declines gracefully under heavy loads." I don't think the results I've shown
in this email satisfy that requirement.
Vic
--- Jean Delvare <khali at linux-fr.org> wrote:
> > --- Daniel Kegel <dank at kegel.com> wrote:
> > > I don't think you should be sharing a .distcc directory between
> > > multiple users. It's not a common configuration, and there are
> > > sure to be problems.
> >
> > Does "the community" agree? I thought this was the recommended setup.
>
> I do indeed agree :)
>
> > If I don't share a common .distcc directory then how do I solve my
> > worst-case scenario: 15 developers suddenly have to compile the entire
> > tree. They all put only the best hosts into their host files
> > (because, after all, they each want a really fast compile), and they
> > all start up their compilation. The result is a bunch of blocked
> > distcc tcp connects to machines with long tcp "listen" queues, and
> > ultimately, very slow compiles...
>
> I would hope that you users learn from their experiences. If "putting
> only the best hosts into their host files" leads to slow compiles, they
> are likely to get better results by using the supposedly slower hosts
> instead. All this should self-equilibrate after some time.
>
> --
> Jean "Khali" Delvare
> http://khali.linux-fr.org/
> __
> distcc mailing list http://distcc.samba.org/
> To unsubscribe or change options:
> http://lists.samba.org/mailman/listinfo/distcc
>
__________________________________
Do you Yahoo!?
Yahoo! Mail Address AutoComplete - You start. We finish.
http://promotions.yahoo.com/new_mail
More information about the distcc
mailing list