superlifter design notes and a new proposal

[Martin Pool]

I've been thinking a bit more about Wayne and jw's ideas.

My first draft was proposing what you might call a "fine-grained" rpc
system, with operations like "list this directory", "delete this
file", "calculate the checksum of this file."  I think Wayne's rzync
system was kind of like that too.

One unusual feature of rsync compared to most protocols is that a
single request causes an enormous amount of stuff to happen: there is
only one request/response per connection at the moment, really.  It is
a very CISC-like protocol.

I wonder what we could achieve if we stay broadly within that model,
of both parties knowing about the whole job, and working in tandem,
rather than one of them controlling the other per file?  So the client
will send something more or less equivalent to its whole command line.

This would be a more conservative design in some ways, because it is
more similar to the existing system.  It also perhaps avoids some of
the issues about pipelining that have been giving me trouble at least.

While staying with that overall approach, we may still be able to make
some improvements in 

 - documenting the protocol

 - doing one directory at a time

 - possibly, doing librsync deltas of directories

 - just one process on either end

 - getting rid of interleaved streams on top of TCP

 - sending errors as distinct packets, including a reference to the 
   file that caused them (if any)

 - handling ACLs, EAs, and other "incidental" things

 - holding the connection open and doing more operations afterwards

What made me start thinking this way is the realization that the basic
idea of cooperating processes (rather than client-server) is not
really causing us any trouble at the moment.  Other things in that
list are, like the interleaved error stream, or the 3-process model.
But perhaps sending the arguments across the network and having the
remote process know what to do is not such a problem.

I will try to write up a more detailed description of this idea later
on.

-- 
Martin