[distcc] randomized hosts + lock files for preprocessing
Martin Pool
mbp at sourcefrog.net
Thu Mar 24 02:52:50 GMT 2005
Donohue, Michael wrote:
>
>
> Hi,
>
> We use distcc here at paypal, with many developers compiling, i.e.
> preprocessing and linking, on 4 machines with a shared disk. The file
> locking seems work pretty well, even on a shared filesystem. We have
> made some changes to distcc to cope with our environment, and to get
> better performance. I’d like to start submitting the more general
> changes back to the source tree.
>
>
>
> 1. A change to make preprocessing grab a lock file before commencing
> with the job. This allows much more parallel make with minimal
> performance impact, because the extra jobs just spin waiting for a
> lock. Without this change, the extra jobs all compete for cpu
> time, and slow each other down, all while holding backend locks.
> This greatly streamlines the process, and reduces the number of
> locks held.
>
>
>
> 2. Randomizing the host selection algorithm. This randomized based
> on slots instead of hosts. The advantage here is that a host with
> more slots is more likely to be chosen. Our backend farm has
> about ten 4 processor machines, and about 30 single processor
> windows boxes. The existing algorithm always fills up the
> windows machines before using extra capacity in the multi-cpu
> boxes. By changing how the hosts are selected, the
> multi-processor boxes are now 4 times more likely to be selected,
> which matches their 4x of capacity.
>
>
>
> I would like to get these changes cleaned up, and parameterized so that
> they can be turned off and controlled, but once that is done, what is
> the preferred way of submitting patches?
I think that would be fine.
Please just send a patch (with diff -u); if possible include updates to
the manpages.
--
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 256 bytes
Desc: OpenPGP digital signature
Url : http://lists.samba.org/archive/distcc/attachments/20050324/ca1a9857/signature.bin
More information about the distcc
mailing list