[distcc] load management

[Wayne Davison]

On Tue, Apr 27, 2004 at 04:04:39PM -0700, Jake McGuire wrote:
> I am planning to start working on adding load management enhancements 
> to distcc for use at our company

Here's an idea I had been kicking around for awhile but never got to.
All systems that want to make distcc requests need at least one local
server process.  This server would be responsible for talking with the
other servers that were around (configured in some manner), keeping
track of each one's load (they would have either an open socket or get
regular UDP updates) and also how fast they have executed jobs (per K)
at various loads.  The idea is that this current status is maintained on
each local server (so no central server going down is a single point of
failure), but not by every client when it starts up.  I imagined this
server being the same one that handles the job requests from other
machines (if configured to accept remote jobs).  So, the local distcc
executable would just connect to the server at localhost, ask it to run
a job, and send it the output of the preprocessed input.  The local
server would be free to either send the job off to another host, or run
it locally.

The stats kept could start simple (with job balancing and overloading
protection) and get as complicated as desired.  For instance, a future
complicated algorithm could take into account transfer time in addition
to compile speed per-K and current load.  (I imagine every job that a
remote server performs coming back with the elapsed compile time as
measured on the remote system so that the sending server could figure
out how fast/slow the transfer part of the equation was.  I forget if
distcc already has this or not.)

To get status, a status tool would connect to the local server and ask
for a summary.

Just thought I'd throw that idea out there.  Shoot it full of holes, as
needed.

..wayne..