[distcc] randomized hosts + lock files for preprocessing

[Donohue, Michael]

Hi,

We use distcc here at paypal, with many developers compiling, i.e.
preprocessing and linking, on 4 machines with a shared disk.  The file
locking seems work pretty well, even on a shared filesystem.   We have
made some changes to distcc to cope with our environment, and to get
better performance.   I'd like to start submitting the more general
changes back to the source tree.  

 

1.	A change to make preprocessing grab a lock file before
commencing with the job.   This allows much more parallel make with
minimal performance impact, because the extra jobs just spin waiting for
a lock.   Without this change, the extra jobs all compete for cpu time,
and slow each other down, all while holding backend locks.   This
greatly streamlines the process, and reduces the number of locks held.


 

2.	Randomizing the host selection algorithm.   This randomized
based on slots instead of hosts.  The advantage here is that a host with
more slots is more likely to be chosen.   Our backend farm has about ten
4 processor machines, and about 30 single processor windows boxes.   The
existing algorithm always fills up the windows machines before using
extra capacity in the multi-cpu boxes.   By changing how the hosts are
selected, the multi-processor boxes are now 4 times more likely to be
selected, which matches their 4x of capacity.

 

I would like to get these  changes cleaned up, and parameterized so that
they can be turned off and controlled, but once that is done, what is
the preferred way of submitting patches?

 

Michael

 

-------------- next part --------------
HTML attachment scrubbed and removed