[distcc] A way to validate the distcc setup?

[Martin Pool]

On 24 May 2003, Oscar Fudd <ofudd at speed-test.net> wrote:
> Hi!
> 
> Just downloaded and installed distcc, and I like it a lot, thanks!

I'm glad you like it.  Thanks for the suggestions.
 
> Now I'm curious about testing.  What I want to do is compile my linux
> kernel, using various machines that I administer over the net.
> 
>   1. I want to see if I'm successfully connecting to the distccd
> programs.  around the world, at the end of a serial cable, down a
> well, it's going to be a veeeerrrrrrry slow connection.  If my
> compile is going to hang waiting for something from this server, I'm
> going to be annoyed (at myself).

Don't do that then. :-)

Better monitoring is on the todo list.

In the meantime you can see what's happening by either turning on
client logging, or looking at "netstat -t" on the client.  This last
one gives a simple and accurate picture of what's happening.

>   2. I want to see if the compilers on the various machines are
> working right.  like to compile the same code on all the servers and
> compare them to what my local machine produces.  This would detect
> bad cross-compiler setups too.  several different computers, and
> compares the results that come back before saving the results.  This
> guards against accidental (or malicious) introduction of bad data by
> a single source.  I'd like to do that here too, if only to guard
> against bad versions of gcc or bad memory.  beginning of a big
> compile, and periodically every 5 minutes after that.  This would
> catch misconfigurations and (some) memory errors while retaining
> most of the speed.

There's a patch to do that. 

One problem is that due to gcc quasi-bugs a remote compile may be
slightly different (mostly in debug symbols) from a local compile,
even if they're functionally identical.

Is this an actual problem for you, or just an idea?  If you have flaky
memory then you have bigger problems.  For most people once they get
the right version of the compiler installed then things don't
magically break.

Don't fool yourself that you could catch malicious servers.

>   3. Is the distccd program always necessary?
>     tar c inputfile.i | ssh user at host 'tar x; gcc inputfile.i -c
> inputfile.o; tar c inputfile.o ; rm inputfile.i inputfile.o' | tar x
>     It skips a lot of steps, at a cost of 25% of the speed, if I read
> right.

Yes, you could do that.  I thought about doing that originally and
other people have built distributed make systems on similar
techniques.

I don't expect people to use ssh in most situations.  In my experience
most places where people do development have firewalled networks that
can be trusted for insecure protocols.  I suppose the exceptions are
universities or very large offices, in either of which you might have
a few rogue users or compromised machines.  I suppose you could use
rsh there though so it's not really an argument against the general
idea.

I think installing a server program is not very much trouble,
particularly if you're going to invoke it over ssh and therefore don't
have to worry about adding another security interface.  It's one apt
or rpm command on Linux and about five commands if there's no package
for your OS.  Surely the speed benefit justifies that tiny amount of
work.  It's certainly less work than getting the right compiler
installed.

Just running a remote command might make error handling harder.  Your
command (only an example I know) will keep running on even if the tar
command fails, and it doesn't use unique filenames.  All of these can
be fixed.

Invoking helpers like the shell, tar, etc also have a performance
cost.

tar can behave quite differently on various OSs.  Your invocation by
assuming it writes to stdout, would only work on GNU tar.  Some tars
always print garbage messages to stdout; they would have to be
silenced.  Some tars use incompatible formats.

tramp-mode under emacs implements remote file access independent of
connection method by sending shell commands across and does an
excellent job at the illusion of being able to directly edit remote
files.  It discovers the capabilities of the remote host and sends
bits of perl, sh, etc to be able to reliably transfer and manipulate
binary files.  This is quite slow on high-latency links, and a bit
fragile.

The better scheduler planned for 3.0 requires some server
intelligence.

Keeping ssh connections for performance alive requires some kind of
custom program on the server, either a generic fsh-workalike or
distccd.

I'm not sure this would simplify distcc.  The intelligence about
interacting with the compiler and make would still have to be there,
and that's most of the complexity.  Building reliable and portable
remote shell commands is not necessarily simpler than the simple
protocol distcc uses.

Perhaps using rsh would have been a good idea for the first version,
but for the features coming up next it's probably not worthwhile.
  
> (I'm not on the list; could you Cc me?  Thanks!)
-- 
Martin 

I think that I can safely speak for the whole troll community when I
say "I like watching train wrecks".
		-- AC