[distcc] load management

[Dan Kegel]

Martin Pool wrote:
> For the particular case of talking to distccd, we could arrange for
> distccd to keep listening on the socket after a job is complete, which
> avoids the need to run fshd.  We just need something on the client to
> hold open the fd and pass it to distcc client processes as they need
> it.  Perhaps this would be a good place for a client daemon using a
> unix socket, though I'm pretty sure it should be per-user.

Hey, I know where to put the local daemon: inside the make process.
That solves the issue of how to tell distcc where to connect
(just put the address in an environment variable, or heck,
open up an fd before forking).

The next step, of course, is to turn distcc into a library,
and build it into make as well, to get rid of one fork+exec.
And then you might want to turn the gcc front end
into a library, to get rid of one more fork+exec,
and merge argument processing with distcc's, perhaps.
And then you might want to turn the c preprocessor into a library,
to get rid of one more fork+exec.

Maybe Per Bothner's compile server
(http://per.bothner.com/papers/GccSummit03/gcc-server.pdf)
went about things the wrong way; it assumed the world consisted
of a single workstation.  If the world is really that one workstation
plus a bunch of idle workstations waiting to run compile jobs for you,
maybe an easier place to start is by sucking the toolchain
into 'make' bit by bit...

>>I'd like to go for a world record time compiling e.g. the
>>linux kernel, so I'm a bit more worried about silly little
>>performance issues than is reasonable or healthy :-)
> 
>   http://lwn.net/2002/0321/a/7.5-second-compile.php3
> 
> Hard to beat. :-)  But it might be fun to try.

Nooo, I doubt we'll ever beat that in absolute time, but
we can probably trounce it in compiles-per-second-per-dollar.

On the other hand, my dual 650 MHz workstation at home takes 86 seconds
just to do
    time find linux-2.6.5 -type f | xargs cat > /dev/null
so I suppose I'll have to find me a faster box if I actually
want any speed out of all this :-)

OK, I'll go take my medication now.
- Dan

-- 
My technical stuff: http://kegel.com
My politics: see http://www.misleader.org for examples of why I'm for regime change