[distcc] SSH encryption overhead

[Aaron Lehmann]

On Thu, Sep 19, 2002 at 03:04:51PM +1000, Martin Pool wrote:
> In both these cases, it is possible that the network will be the
> limiting factor.  It would possibly be better for the client to
> schedule no local compilations and do more work on compression,
> because it will allow it to shift more of the work elsewhere.
> 
> If you consider an infinite supply of volunteer machines, then the
> challenge is for the client to balance up its own CPU usage (for cpp
> and lzo) and network bandwidth to dispatch as many jobs as it can.  By
> changing the compression level depending on cpu load, it might keep
> both close to 100% (thought actually doing that would probably be
> overoptimization.)

Even with LZO, the network will still be the limiting factor. AFAICT
the current design is very serial. First cpp is run, then the source
is shipped over the network. These are the two steps to getting a
compilation started. The faster the source is recieved on the other
side, the faster compilation can begin. So the server will always have
to wait on the network to get the source, as network transmission is
one of the stages.

A computer should pretty much always be able to compress LZO much
faster than it can send the resulting data over the network, unless
you have a slow computer on a fast network. So there will be CPU time
for cpp left over.

There is a threshold where focusing on preprocessing and compression
only becomes a more productive strategy for the client. This is
heavily dependent on the setup. For now, it might be nice to provide
guidelines about when to omit localhost from the compilation server
list for maximum performance. I think increasing the compression level
is a waste and it would be better to run a compiler on localhost if
there are CPU cycles to spare, unless you have an extremely slow
network.

> It may be that the cost of LZO is not so much the compression itself,
> but rather than it will prevent us from using sendfile(), and we'll
> therefore take more context switches while sending.  (But they're
> pretty cheap, so who can tell?)

I don't think this is significant. At least for me, LZO is not a win
in terms of CPU cycles for the sender anyway. The point of it is to
maximize network bandwidth at a minimal CPU cost. A few context
switches is nothing compared to the cost of compression (yes, if the
compression algorithm was fast enough, it could result in an overall
decrease in CPU consumption on an encrypted link, but my tests haven't
found this to be the case for LZO).

If we stop using sendfile() anyway, maybe we should read cpp output
from a pipe instead of a file. This would reduce filesystem overhead,
I think. It's worth testing.