superlifter design notes and a new proposal

[Wayne Davison]

On Mon, 5 Aug 2002, Martin Pool wrote:
> I also had a look at the byte-level documentation for rZync again.
> Can I persuade you to switch to a more XDR-like format?

I could be persuaded as long as the transferred-byte overhead is kept
under control.  The nice thing about the on-the-wire protocol is that it
is a small, self-contained sub-problem that does not affect the rest of
the app.  It's essentially a black-box in between the send_msg() and
read_msg() functions, and as long as the arg values make it from one
side to the other dependably, the rest of the code doesn't care how it
happens.

The current byte-level code in rZync is rather cryptic, but I wanted to
keep the byte-overhead near to rsync in my tests so I could see how the
single-process-per-side design was performing.  Thus, it does go to
pains to squish the bytes out of the headers wherever possible.  I'm not
wedded to this implementation, however.  I'll probably leave it alone in
rZync until I get the code dependable enough to be able to do some
really huge transfer tests (since I'm curious how big the program will
grow to be and how quickly it will run).  The code is almost to that
point now, but I haven't had much of a chance to work on it in the past
month or so.

>  - Using lzo or gzip will squish them to a similar size, so
>    micro-optimization is not necessary.

I'm somewhat cautious about adding a compression layer over the top of
the message headers.  It should be doable, but it will make things a
bit more complex.  For instance, my current code reads one message at a
time because it understands the header format and knows how long the
data will be from that understanding.  A zlib deflate/inflate that
included the message headers themselves will mean that we need to slurp
as much data as is available on the socket, decompress as much of that
as possible (which might not be all the currently-read data), and then
operate on as many messages as we find in the decompressed data.  We
also have to be sure that we flush the data on the compression side
often enough that we avoid deadlocking waiting for new message headers
that aren't coming (i.e. zlib adds a buffering layer of its own).  If
we make it flush once per message, that would probably be very byte-
inefficient for smaller messages.  If we instead only force a flush
when a pause in the action occurs, that would probably be optimal, but
might add delays to the transfer and would add overall complexity.

>  - Packing/unpacking these things will use much less CPU

I'm not sure that this will be true if the headers are then compressed
with something like zlib.

So, I think the tradeoffs between the two schemes are closer than you
might think.  My variable-width message header scheme is essentially a
very specific compression scheme placed over the top of the arg values
that come in to the send_msg() function.  I think that using your idea
of fixed-width header fields and a zlib (or whatever) compression over
the top would be able to surpass my current implementation in byte
efficiency over the wire, but will be slightly more complex to implement
(due to the extra buffering layer and the blind socket reads).

In the future, we should be able to code up a replacement for the
read_msg()/write_msg() functions and just plug it into the code and try
it out (and also have a nice test suite that exercised these functions
and checked the resulting values).  In the present my rZync code saves a
buffer copy by exploiting knowledge of how the send_msg() function
works, so I'd have to change the sig/delta code first before we could
try out a replacement (I know, I know; I'm an optimizing-fool sometimes).

..wayne..