[distcc] Loadbalanced distcc

Thu Jan 5 14:07:26 GMT 2006

Sean, et al., 

You do seem to understand my system correctly: dmucs is a wrapper around distcc which is a wrapper around gcc.

Note that we here at marconi use pcons, not make to build our tree.  pcons is smarter than make because it calculates the entire tree of build dependencies before beginning to compile even the first file.  So, it can know which things to do in which order.  Now, in my opinion, pcons isn't quite smart enough, because it doesn't parallelize compiles across multiple directories -- that would REALLY speed up the compile, I think.

The nice things about dmucs are:

o if other engineers are using the compile machines for other tasks, the loadavg daemons will communicate the loads to the dmucs server, and those machines will be used less often for compiles -- so my system takes into account not just loadavg based on number of compiles on a machine, but on other factors as well.

o machines can be added and deleted from the compilation farm dynamically.  For example, a powerful desktop machine on an engineers desktop can be added to the compilation farm when it goes into screen saver mode.  When the eningeer comes back to his/her desk, the machine can be immediately removed from the farm.  I've tried this, and it works well.

o machines with differing "power" can all contribute to the compilation farm.  Each machine is assigned a "power index", and the most powerful, available, and not-overloaded machines will be given out for compiles.  If these machines become overloaded, then less powerful machines will be given out.  This provides  the best compilation times possible whenever the engineers does the compile.

o the system scales really well.  I've tried it with 6 compilations going simultaneously, and the time to compile our code based went up from 44 minutes to 56 minutes -- an increase of 27%.  That's not bad for a six-fold increase in the number of compilation requests happening.

o ideally, it would be very nice to be able to know how long different files take to compile, and what order to do them in to produce the very best use of the compilation farm.  But that is probably an NP-complete problem, and I needed to bite off something I could chew.

Vic

Sean D'Epagnier <geckosenator at gmail.com> wrote:    Again, my DMUCS solution provides load balancing.  DMUCS is basically a wrapper around distcc.  You run a "host-server" and you run very simple load-average-reporting tasks on each host (along with the distccd's).  Then, your compile contacts the host server for a host, places the result in DISTCC_HOSTS environment variable, and runs distcc. 

 I am in the process of getting my code onto sourceforge in cvs.  It may even happen today.

 Please let me know if you want me to send you the code directly.

 Vic      - Show quoted text -

 Let me try to understand this correctly:

 distcc is a wrapper around gcc, and dmucs (or whatever it is called) is a wrapper around distcc.  When dmucs is run, it reads data from the locally running host server for the DISTCC_HOSTS variable and runs distcc.  All of the distcc servers must run a distcc server and a client to the host server.  These clients send information about the load state of each server to the host server, so it has a good idea of who is idle, and will set the DISTCC_HOSTS variable correctly.

Did I understand that correctly?  Have you measured improved performance over distcc using this setup?  Why do you need a server and client setup, why can't the wrapper measure how long distcc takes for that given host and submit it to the host server itself?  I think there may be a few problems, but I'm not sure.  I don't see how this would speed up a compile.  distcc already, sends files to compile to the host that has the least jobs currently.  Because two files can compile on a host in parallel at nearly the same speed as serially with a multitasking os, as long as the makeopts are set high enough "-j8" then all of the cpus can be utilized well.  The problem is when there are less files left than cpus to compile them, then some cpus become idle.

Distcc slows down when a big file is given to a slow computer, and a small file is given to a fast computer, and there are no more files that can be build until the big file completes.  I don't see how you can solve this problem if you program is just a wrapper.  Distccd needs to measure the compile time as well as something to go with it (I suggest file size) to calculate the average speed of the server.

 The distcc client would look at all the files that can potentially be built in parallel,  There are a lot of ways to do this, the easiest is to send the files in order from largest to smallest, this is not perfect, but this way the longest you have to wait is the time it takes the slowest computer to compile the smallest file.  Another way would be to try to fit the files so that the computers will all finish as close to the same time as possible,  If it's faster to send two files to a fast computer and leave a slow one idle, it should be done.  Finally, if the compile has been done before, the distcc client can cache how long it takes to compile certain files.  Using this information instead of file size to determine how to send the files to various computers would further increase efficiency.

 Another thing to mention, files that open up other files to compile should have a higher priority and be compiled first.

 I realize these suggestions greatly complicate distcc, it would need to either interpret makefiles directly, or return immediately, and store the options passed to it in a build tree.  I think it makes sense to have distcc be a wrapper to make instead of gcc.

-sean

On 1/4/06, Victor Norman <vtnpgh at yahoo.com> wrote: Again, my DMUCS solution provides load balancing.  DMUCS is basically a wrapper around distcc.  You run a "host-server" and you run very simple load-average-reporting tasks on each host (along with the distccd's).  Then, your compile contacts the host server for a host, places the result in DISTCC_HOSTS environment variable, and runs distcc. 

I am in the process of getting my code onto sourceforge in cvs.  It may even happen today.

Please let me know if you want me to send you the code directly.

Vic 

Sean D'Epagnier <geckosenator at gmail.com> wrote:
  I'm also very interested in this subject.  I think there are a lot of advanced ways to do it, but a few simple ones would really help performance.  If anyone wants to contact me about working on this, or would like to share ideas, feel free to.

On 1/3/06, Patrik Olesen < patrik at famolesen.com> wrote: Hello,

I have seen that there where an old thread about developing some sort of 
loadbalancing for the deployment of the compiler jobs. What is the
progress of this work does anybody have any news?

Best regards, 
  Patrik
__
distcc mailing list             http://distcc.samba.org/
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/distcc  

 __ 
distcc mailing list             http://distcc.samba.org/
To unsubscribe or change options: 
 https://lists.samba.org/mailman/listinfo/distcc

---------------------------------
Yahoo! Photos
  Ring in the New Year with  Photo Calendars. Add photos, events, holidays, whatever. 

---------------------------------
 Yahoo! DSL Something to write home about. Just $16.99/mo. or less
-------------- next part --------------
HTML attachment scrubbed and removed