[distcc] randomized hosts + lock files for preprocessing

[Martin Pool]

Donohue, Michael wrote:
>
>
> Hi,
>
> We use distcc here at paypal, with many developers compiling, i.e.
> preprocessing and linking, on 4 machines with a shared disk.  The file
> locking seems work pretty well, even on a shared filesystem.   We have
> made some changes to distcc to cope with our environment, and to get
> better performance.   I’d like to start submitting the more general
> changes back to the source tree.
>
>
>
>    1. A change to make preprocessing grab a lock file before commencing
>       with the job.   This allows much more parallel make with minimal
>       performance impact, because the extra jobs just spin waiting for a
>       lock.   Without this change, the extra jobs all compete for cpu
>       time, and slow each other down, all while holding backend locks.
>       This greatly streamlines the process, and reduces the number of
>       locks held.
>
>
>
>    2. Randomizing the host selection algorithm.   This randomized based
>       on slots instead of hosts.  The advantage here is that a host with
>       more slots is more likely to be chosen.   Our backend farm has
>       about ten 4 processor machines, and about 30 single processor
>       windows boxes.   The existing algorithm always fills up the
>       windows machines before using extra capacity in the multi-cpu
>       boxes.   By changing how the hosts are selected, the
>       multi-processor boxes are now 4 times more likely to be selected,
>       which matches their 4x of capacity.
>
>
>
> I would like to get these  changes cleaned up, and parameterized so that
> they can be turned off and controlled, but once that is done, what is
> the preferred way of submitting patches?

I think that would be fine.

Please just send a patch (with diff -u); if possible include updates to
the manpages.

--
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 256 bytes
Desc: OpenPGP digital signature
Url : http://lists.samba.org/archive/distcc/attachments/20050324/ca1a9857/signature.bin