superlifter design notes and a new proposal

[Wayne Davison]

On Sun, 4 Aug 2002, Martin Pool wrote:
> My first draft was proposing what you might call a "fine-grained" rpc
> system, with operations like "list this directory", "delete this
> file", "calculate the checksum of this file."  I think Wayne's rzync
> system was kind of like that too.

Your previous proposal sounded quite a bit more fine-grained than what
rZync is doing.  For instance, it sounded like you would have much more
primitive building-block messages and move much of the controlling
smarts into something like a python-language scripting layer.  While
rZync allows ftp-level control (such as "send this file", "send this
directory tree", "delete this file", "create this directory") it does
this with a small number of higher-level command messages.

Rsync, as you know, is a much more modal protocol.  It has a strict set
of steps that must be specified in order and nothing else.  This saves
bytes because so much of the protocol is determined by context, but is
very limiting.

My rZync protocol opens this up by using message numbers for everything
that gets sent, but it still keeps some context-oriented "smarts" when
transferring files.  There is no micro-management of a file transfer
from start to finish.  The messages cascade from side to side as the
sig, delta, patch sequence of events unfold.  The most CISC-like message
in rZync is the recursive-directory-send message.  Using this is very
much like starting an entire "rsync -r src/ dest" transfer sequence via
a single message.

> So the client will send something more or less equivalent to its whole
> command line.

I think that's a good idea.  My rZync app currently operates on each arg
independently, but I recently discovered that this makes it incompatible
with rsync when merging directories and such.  For instance, the command
"rsync -r dir1/ dir2/ dir3" merges the file list and removes duplicates
before starting the transfer to dir3.  rZync currently just transfers
the contents of dir1 to dir3 and then transfers the contents of dir2 to
dir3.  Fortunately, this is not going to be hard to fix.

> While staying with that overall approach, we may still be able to make
> some improvements in
> 
>  - documenting the protocol
> 
>  - doing one directory at a time
> 
>  - possibly, doing librsync deltas of directories
> 
>  - just one process on either end
> 
>  - getting rid of interleaved streams on top of TCP
> 
>  - sending errors as distinct packets, including a reference to the 
>    file that caused them (if any)
> 
>  - handling ACLs, EAs, and other "incidental" things
> 
>  - holding the connection open and doing more operations afterwards

This is very much in keeping with what I've been fiddling with in rZync
(which nearly implements this whole list).  I like the simplicity of one
process per side, which makes it easy to cache data that will be used
later and discard it when it is no longer needed.  I got rid of the
"multi-IO" idiom of rsync in favor of sending all data via messages and
limiting each chunk to 32K to allow other messages to be mixed into the
middle of a large file's data-stream (such as verbose output).

I think the basic idea of how rZync envisions a new protocol working is
a good one -- not so much the specifics of the bytes sent in the
message-header format, but how the messages flow, how each side handles
the messages in a single process, how all I/O is handled by a single
function, etc.  There's certainly lots of room for improvement, though.

This also reminds me that I hadn't responded to jw's question about why
I thought his pipelined approach was more conducive to a batch protocol
than an interactive protocol.  To make the pipelined protocol as
efficient as rsync will require the complexity of his backchannel
implementation, which I think will be harder to get right than a
single-process message-oriented protocol.  If every stage is a separate
process, it seems less clear how to implement something like an
interactive "mkdir" or a "delete" command. (What process handles this?
How do we signal that process?  Do we need yet another socket path for a
control stream in some circumstances?)  It also seems to me that the
extra processes/threads and socket-channels will make a less portable
interactive app than a single select-using interactive app.

..wayne..