rsync replacement

[Matt McCutchen]

On 7/17/07, Jamie Lokier <jamie at shareable.org> wrote:
> So am I right in thinking that using rsync in conjunction with your
> tunnelling product as the underlying transport might give better
> performance for incremental file transfers than your current client?

As I understand it, there are three performance-increasing techniques:

1. Delta transmission
2. Sustained full utilization of the connection via better congestion control
3. The ability to act immediately on any packet that reaches the
destination rather than waiting for the next one in sequence (helpful
when congestion is causing significant packet loss)

Rsync gets #1.  Any program running on MTP gets #2.  However, #3 can
only be achieved by a parallelized application protocol running on a
parallelized network protocol (MTP, UDP, etc.).  I infer that syncdat
uses a parallelized application protocol; Mike, please correct me if
this is not the case.

Thus, syncdat gets #2 and #3 but (it seems) not #1.  Rsync running on
a TCP-over-MTP tunnel would get #1 and #2 but not #3.  To get all
three benefits, we would need to make a program that has both delta
transmission like rsync and a parallelized protocol like syncdat and
run the result on a parallelized, congestion-controlling protocol like
MTP.  (Encryption, if any, would have to be done on individual packets
instead of as a stream filter; I think MTP does this already.)  I
would be very interested to see such a program and even more
interested if it had a BitTorrent-like capability to pull pieces of
the data from multiple sources.

Matt