[distcc] about distcc's scheduling

[Fergus Henderson]

Have you tried using pump mode?  That helps a lot.

Also, if the issue is that distcc isn't running enough jobs on each host,
try using the "host/JOBS" format in DISTCC_HOSTS,
e.g. DISTCC_HOSTS="foo.riken.jp/20,cpp,lzo bar.riken.jp/40,cpp,lzo".
This would allow up to 20 concurrent compiles on host foo.riken.jp and 40
concurrent compiles on host bar.riken.jp.
(The ",cpp,lzo" part is required for pump mode.)

At some point, depending on the size and structure of the software that
you're trying to compile,
and the way your Makefiles are arranged, you may run into limitations in the
parallelizability of your build graph.
For example, if your software has only three source files, then distcc won't
be able to speed up compilation by more than a factor of three at very best.
Similarly, if your software has 30 files, but they are split into 10
directories of three files each, and your Makefile(s) sequentialize the
compilation of each directory, then you still won't get more than a factor
of three speed up at very best.

On Mon, Nov 23, 2009 at 8:19 PM, Francois Berenger <berenger at riken.jp>wrote:

> Hello,
>
> Is there a way to use a pull scheduler instead of a push scheduler inside
> distcc?
>
> I have many compiling hosts and when they join a compiling job, they are
> not enough loaded, at least to my taste. Meaning: they could compile more
> things and I could wait less for them to finish.
>
> I think the problem is that the scheduler is pushing: the distcc server
> sends jobs to distcc compiling hosts.
>
> Instead, I would like this behavior: distcc compiling hosts ask and take
> jobs as long as they are not completely loaded with compilation jobs.
> The distcc server should distribute jobs to whom asks for.
>
> This should scale better.
>
> If there is no way now to do so but you can give me some hint on how to
> do so (where to look, what to modify, etc.), it might be helpful.
>
> Best regards,
> Francois.
>
> __
> distcc mailing list            http://distcc.samba.org/
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/distcc
>



-- 
Fergus Henderson <fergus at google.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20091123/42d346ee/attachment.html>