[distcc] feature request: command-line switch to avoid using localhost

[Martin Pool]

(Very old message)

On 12 Dec 2002, Alexandre Oliva <oliva at lsd.ic.unicamp.br> wrote:
> I've been working on and off on trying to make distcc useful for
> bootstrapping gcc.  The main difficulties are:
> 
> - bootstrapping gcc involves building a stage1 compiler, and then
>   using that to build stage2, and using stage2 to build stage3.  This
>   means the compiler in the build tree must be accessible to distccd.
>   I've managed to fix this for the case of a shared NFS filesystem,
>   using a wrapper script that transforms relative pathnames into
>   network-neutral ones.
> 
> - compiling with the integrated preprocessor generates different
>   debugging information from what we get with separate preprocessing
>   and compilation.  This causes the test at the end of building
>   stage3, that compares it with stage2, to fail, rendering the entire
>   bootstrap procedure useless (the whole point is to verify that the
>   compiler is capable of rebuilding itself, without random changes,
>   that always indicate bugs).
> 
>   After banging my head against the wall for a couple of days trying
>   to fix the differences in debugging output and not only failing, but
>   also getting one of the maintainers of the preprocessor to disagree
>   it was a bug in the first place :-(, the solution I came up with was
>   to arrange for the wrapper script to take out `localhost´ from
>   DISTCC_HOSTS, replacing it with the local hostname (which works as
>   long as distccd is running on localhost too).  This worked in
>   earlier versions of distcc, but it apparently no longer does ;-(
> 
> One of my main problems at the moment is that the wrapper script I
> created that network-neutralizes pathnames is somewhat slow, such that
> using it for small builds such as those of a number of gcc target
> libraries doesn't pay off.
> 
> So I came up with the idea of reducing the number of times the wrapper
> script runs: instead of transforming pathnames for every single
> compilation, I get it to print the transformed compiler command (e.g.,
> turning ./xgcc -B./ -B/usr/local/.../ into /net/`hostname`/`pwd`/xgcc
> -B/net/`hostname`/`pwd`/ -B/usr/local/.../), and then using the second
> command all over.
> 
> The problem is that, in this case, I can't override the DISTCC_HOSTS
> setting, which means I can't take localhost out of it, and then
> bootstrap fails.  So I tried to arrange for the script to print say
> DISTCC_HOSTS='the edited DISTCC_HOSTS' distcc ..../xgcc ..., but this
> failed because the build machinery is too sensitive to quotation
> issues, and it ended up setting DISTCC_HOSTS='host1, and then running
> the command host2, which didn't quite work as expected.
> 
> See, I might set DISTCC_HOSTS without localhost, but this would slow
> down the building of stage1 and all the rest of the toolchain that
> doesn't undergo bootstrap-checking, so I'm wondering if a command-line
> option for distcc that would tell it to not short-circuit localhost
> compilations, using distccd running on localhost instead, would be
> acceptable.  Failing that, perhaps a separate environment variable?
> Even using a different separator for DISTCC_HOSTS would do, as long as
> it's not a shell active character.

You can use 127.0.0.1 instead of 'localhost' to cause compilations to
be passed through a daemon running locally.  Will that do what you
want or did I miss something?

Also, in 2.1cvs you can set DISTCC_FALLBACK=0, and distcc will not
fallback to building on localhost when a remote compilation fails, but
will rather fail completely.  This is mostly meant for testing, but
perhaps it would be useful in your case (although I can't see why at
the moment. :_)
 
> I take this opportunity to plead people to consider such issues when
> proposing additional syntax for DISTCC_HOSTS: if it was possible to
> handle DISTCC_HOSTS as a single shell word (perhaps after turning
> blanks into say commas), without the risk of any shell active
> characters such as {, }, ~, $, quotes getting in the way, outputting
> distcc commands that override DISTCC_HOSTS would be far simpler.

Excellent point.  I think that is the case at the moment, and I will
try to remember to retain it.

> Failing all of this, I could use another wrapper script that would
> just override localhost in DISTCC_HOSTS, but this would certainly not
> be as fast.
> 
> I'm willing to implement one of the changes in distcc that I proposed
> above, but I'd like to avoid wasting time implementing something that
> wouldn't be acceptable, so I'm asking first.  Which, if any, would you
> prefer to have in distcc?

Is there anything else that can be done here now?

-- 
Martin