[distcc] Benchmarking the server-side cache. How bad is 4ms latency to distcc server?

[Dan Kegel]

On 2/5/06, Jake McGuire <jake at boom.net> wrote:
> On Feb 5, 2006, at 2:44 PM, Dan Kegel wrote:
> > Examining the connection between client and server
> > using ping, I see that the round-trip time is usually 400
> > microseconds, but sometimes 4ms; the average is
> > about 1ms.
>
> Server misconfiguration?  Or overloaded network hardware?
>
> Is netstat -s showing noticeable packet drops?

Dunno.  I'll check.  But ping didn't show any packet drops at all,
just periods of 4ms latency.  I'll bet it's congestion.  Just
shows how silly it is to have a distcc server on a different
network, if you ask me.   I guess I'll have to pick closer machines
to benchmark on.

> > I guess I'll experiment with turning on compression next,

Didn't seem to help in initial experiments, but I didn't check too carefully.

> > and look at a protocol change for large files to try sending
> > just the hash of the source first, and sending the full
> > source only if the server replies with 'sorry, not in the cache'.
>
> This doesn't sound like a bad idea to me (the protocol change, not
> the compression).  I don't think that many people are going to be
> running different versions of distcc/distccd to make backwards
> compatibility a problem - the rpm builds both of them - and even if
> they are, making the hash message optional seems like an obvious choice.

Yeah.  And I think I'm going to do it without adding an extra
round trip, too.  The client can, if it wants, give the hash of the
sources just before it sends the actual source.
The server, when it recieves that, will reply *early* with
the compiled files rather than waiting for the client to finish sending
the source.   Yeah, it's evil because the client ends up sending more
than it really had to, but if the server's quick, the client won't have
sent more than a little bit of the file before it realizes it can stop.
And it will unambiguously reduce network traffic from large files without
hurting short ones.

> Given the network traffic in the current distcc protocol, especially
> if DISTCC_DIR is on networked storage and you have a bunch of servers
> (i.e. many NFS/CIFS lock attempts) this is going to be bad news.

Oh, yeah.  DISTCC_DIR on an NFS directory is a known killer.
I avoid that like the plague.  Ixnay on the nfs-ay :-)
- Dan

--
Wine for Windows ISVs: http://kegel.com/wine/isv