[distcc] Loadbalanced distcc

[Sean D'Epagnier]

On 1/5/06, Victor Norman <vtnpgh at yahoo.com> wrote:
>
> Sean, et al.,
>
> You do seem to understand my system correctly: dmucs is a wrapper around
> distcc which is a wrapper around gcc.
>
> Note that we here at marconi use pcons, not make to build our tree.  pcons
> is smarter than make because it calculates the entire tree of build
> dependencies before beginning to compile even the first file.  So, it can
> know which things to do in which order.  Now, in my opinion, pcons isn't
> quite smart enough, because it doesn't parallelize compiles across multiple
> directories -- that would REALLY speed up the compile, I think.
>

I think distcc should be able to have the option, where instead of compiling
immediately, it can store what the build will be in a file that is a sorted
tree of dependencies.  A very simple wrapper for make could take care of
this by replacing gcc with distcc and the specified option, then running
distcc on the tree file.  This way, builds can be in parallel even in
different directories.

I have thought of other ways to do this, with a make wrapper, executing
"make -n" then running every command, and if a given command fails (returns
1), move on to the next one, if a command fails, then move on to the next
looping around until everything is built, or everything left fails.  I know
this isn't pretty, but it should be a lot easier to implement.

The nice things about dmucs are:
>
> o if other engineers are using the compile machines for other tasks, the
> loadavg daemons will communicate the loads to the dmucs server, and those
> machines will be used less often for compiles -- so my system takes into
> account not just loadavg based on number of compiles on a machine, but on
> other factors as well.
>

Distcc already does this to some degree, if you send a job to a heavily
loaded computer, it will take a long time to compile.  More jobs will be
sent to computers that are not loaded (in the same amount of time)  If the
computer is at 100% cpu this would still be a major problem if distccd is
run with a lower priority (nice)

I think if a slow computer is working on the last file, and other computers
are ready, it should try to compile it on other computers as well, and
finish when it gets the compiled result from anyone.

o machines can be added and deleted from the compilation farm dynamically.
> For example, a powerful desktop machine on an engineers desktop can be added
> to the compilation farm when it goes into screen saver mode.  When the
> eningeer comes back to his/her desk, the machine can be immediately removed
> from the farm.  I've tried this, and it works well.
>

This is a pretty useful feature.  I don't think there was a really good way
to do this before.

o machines with differing "power" can all contribute to the compilation
> farm.  Each machine is assigned a "power index", and the most powerful,
> available, and not-overloaded machines will be given out for compiles.  If
> these machines become overloaded, then less powerful machines will be given
> out.  This provides  the best compilation times possible whenever the
> engineers does the compile.
>

This sounds like a good start, maybe the computers running should have a way
to feedback and update their "power index"

o the system scales really well.  I've tried it with 6 compilations going
> simultaneously, and the time to compile our code based went up from 44
> minutes to 56 minutes -- an increase of 27%.  That's not bad for a six-fold
> increase in the number of compilation requests happening.
>

So it takes 44 minutes to compile on one computer, and 56 minutes to compile
6 times using 6 computers?  Can you test the difference without your wrapper
but still using distcc?  Ideally we will get this down to 44 minutes and 44
minutes :-P

o ideally, it would be very nice to be able to know how long different files
> take to compile, and what order to do them in to produce the very best use
> of the compilation farm.  But that is probably an NP-complete problem, and I
> needed to bite off something I could chew.
>

I agree, but maybe we can set things up now so it won't be hard to do that
later?

Can we download and test it? I would like to see if it really speeds things
up, it's obvious it adds useful features.

-sean
-------------- next part --------------
HTML attachment scrubbed and removed