[distcc] Auto discovery of volunteers

[Martin Pool]

On 17 Oct 2003, Gerard Beekmans <gerard at linuxfromscratch.org> wrote:
> Hi guys,
> 
> It's been a little while since I last checked up on distcc's status so
> this might have been discussed and implemented already. Allow me to ask
> it anyway, feel free to redirect me to a FAQ of some sort.
> 
> The one missing feature from distcc that prevents me from making optimal
> use of it is a way to keep a dynamic list of hosts that are available. I
> can't hard-code a list because I don't always have the same machines
> available (either turned on, or they're not running distccd at certain
> times for one reason or another) so there's a high chance of
> distribution failures which slows things down.

I'm inclined to address this by making it handle a failure to connect
by trying a different host, rather than immediately going back to
localhost.  Therefore if you list an superset of available machines,
it will eventually discover which ones are not available, and mark
them as down.  The price you pay will be at most one connection
attempt to down hosts every minute.  

In most but not all cases doing that will be quite cheap, as it will
fail quickly with 'host unreachable' or 'connection refused'.  The
only problem is with hosts where connecting just times out, but I
don't think that will happen too often on local networks without
firewalls or routing problems.  (Or will it?)

If you list a hundred hosts it might be a problem but if say five out
of ten are down I think it will work OK.

Zygo has convinced me that distcc should change a few things to
support multi-A records, so it may be enough to set

  DISTCC_HOSTS="ia64-linux-gcc-3.3/20"

We could also get Apple's patch for using Rendezvous.

> Last I checked distcc, it would only read from an environment variable
> which hosts to try as volunteer. I just noticed in the latest version it
> will also read files. That makes things easier, I could write a program
> that queries a list of potential volunteers every minute and checks if
> the distcc port is open. If not, remove that host name from the file
> distcc uses.

The proposal above seems to have a similar effect, but without needing
a separate program.

Generating the file from some kind of script might still be wortwhile
for particular cases.

> Are there plans to build this into distccd that so it can maintain some
> file that the client reads. This won't work if distccd doesn't run on
> localhost, but I'm sure there's something else that can be done about
> that (something like running distccd locally as a discovery only, not
> accepting compile jobs).

Can you explain this a bit more?

--
Martin