[distcc] Loadbalanced distcc

[Sean D'Epagnier]

There is no rush to get the code, I am interested whenever it's convenient
for you to post it.

I wasted a few hours of time writing a wrapper for make.  The program would
execute "make -n" plus any other command line options.  The program executed
each command make printed out in parallel (each in a different thread)  If
the command was successful (return 0) then it was deleted.  Otherwise, it
was re-executed the next round.  In this way I hoped to have as many of the
commands in parallel as possible, but if dependencies aren't satisfed, gcc
will just return non-zero, and try again the next round.  The program works
for simple make files, it even built distcc, but not so good for the complex
ones (make -n can fail when make doesn't!).  It also turned out to be slower
than just using make, distcc compiles in 4 seconds, and it takes 8 with this
program :(

I can try again with a recursive wrapper, and execute subdirectories in
order (but also parallel)  At this point I don't think it's very easy to
improve performance, since the slowest computer working on the largest file
is on average less than a second!



On 1/6/06, Victor Norman <vtnpgh at yahoo.com> wrote:
>
> oops: I mean that I'm trying to get the tree uploaded to sourceforge.net,
> but am having problem with cvs connectivity, due to our firewall, I think.
>
> Vic
>
>
> *Victor Norman <vtnpgh at yahoo.com>* wrote:
>
> Sean, et al.,
>
> *Sean D'Epagnier <geckosenator at gmail.com>* wrote:
>
> On 1/5/06, Victor Norman <vtnpgh at yahoo.com> wrote:
> >
> > Sean, et al.,
> >
> > < ... snip .... >
> >
>
>
> So it takes 44 minutes to compile on one computer, and 56 minutes to
> compile 6 times using 6 computers?  Can you test the difference without your
> wrapper but still using distcc?  Ideally we will get this down to 44 minutes
> and 44 minutes :-P
>
> [Victor:] No, it takes 44 minutes to compile our software branch using
> -j20 when there is no one else using the compilation farm, which consists of
> about 60 cpus of various strengths.  It takes about 48 minutes when 2 people
> compile with -j20, sharing the farm.  About 49 minutes with 3 people
> compiling with -j20; 51 minutes for 4 simultaneous compiles; 56 minutes for
> 5 compiles, 56.5 minutes for 6 compiles.
>
> o ideally, it would be very nice to be able to know how long different
> > files take to compile, and what order to do them in to produce the very best
> > use of the compilation farm.  But that is probably an NP-complete problem,
> > and I needed to bite off something I could chew.
> >
>
> I agree, but maybe we can set things up now so it won't be hard to do that
> later?
>
> Can we download and test it? I would like to see if it really speeds
> things up, it's obvious it adds useful features.
>
> -sean
>
> [Victor:] I'm trying to get the tree uploaded to cvs, but am having
> problems because of our firewall here at work.  I can send the tree to you
> if you want it.
>
> Vic
>
>
> ------------------------------
> Yahoo! Photos
> Ring in the New Year with Photo Calendars<http://us.rd.yahoo.com/mail_us/taglines/photos/*http://pa.yahoo.com/*http://us.rd.yahoo.com/mail_us/taglines/photos/evt=38087/*http://pg.photos.yahoo.com/ph//page?.file=calendar_splash.html&.dir=>.
> Add photos, events, holidays, whatever.__
> distcc mailing list http://distcc.samba.org/
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/distcc
>
>
> ------------------------------
> Yahoo! DSL<http://pa.yahoo.com/*http://us.rd.yahoo.com/evt=37474/*http://promo.yahoo.com/broadband/+>Something to write home about. Just $16.99/mo. or less
>
>
> __
> distcc mailing list            http://distcc.samba.org/
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/distcc
>
>
-------------- next part --------------
HTML attachment scrubbed and removed