Rsyncing really large files

[Craig Barratt]

Lars Karlslund writes:

> Also the numbers speak for themselves, as the --whole-file option is
> *way* faster than the block-copy method on our setup.

At the risk of jumping into the middle of this thread without
remembering everything that was discussed...

Remember that by default rsync writes a new file and then renames that
file.  So a single byte change to a file requires a complete read and
write (plus the earlier read to generate the block checksums).

The --inplace option is more efficient in terms of disk IO, but the
drawback is that blocks earlier in the original file cannot be matched.
I haven't looked at the code, but I'm guessing --inplace still does
byte-by-byte matching.  An additional optimization for --inplace would
be to only try to match on multiples of the block size.

Also, the matching only proceeds byte-by-byte when there is no match.
Once a match is found then the entire block is skipped.  So on a file
with few changes, the byte-by-byte matching doesn't slow things down
very much.

Craig