matt at mattmccutchen.net
Tue Sep 15 23:44:58 MDT 2009
Both of your proposals have been discussed before (see below), but
neither has been taken very far because they would both involve large
changes to rsync.
On Wed, 2009-09-16 at 01:16 -0400, Lee Winter wrote:
> A simplistic approach would be to add a file-list option able to
> create-and-save a new file list or load-and-use an existing file list.
> This would permit a newly updated mirror to perform the file list
> generation once per update rather than once per client.
A first step in this direction is the maintained patch "db.diff", which
caches checksums of entire source files but does not (yet) cache the
file list itself.
> But in an severely asymmetrical environment where the goal is minimal
> load on the sender, it would be better to swap the roles by using the
> sender's newer version as the "basis file" and let each receiver chew
> through its comparisons rather than having the sender perform the
> comparisons for all of the receivers.
This is a good idea that has been discussed before and is implemented in
another tool called "zsync". Wayne noted some of its drawbacks here:
Perhaps you could respond to his concerns.
> But, while I did see some traffic about the
> conflict between delta processing and compressed files, I never
> located a resolution or summary of the issue. Does the current
> version of rsync actually provide much benefit for compressed files?
Not for typical compression formats, where a local change to the input
changes the output from that point on. If compression would preserve
locality of changes, then delta transfer would work, but locality is
somewhat at odds with effective compression. There used to be a patch
"gzip-rsyncable.diff" that modified gzip for better locality, but I
don't know if it's maintained any more.
The other approach is to delta-transfer the uncompressed content:
More information about the rsync