Rsyncing really large files
cbarratt at users.sourceforge.net
Tue Mar 1 07:09:43 GMT 2005
Lars Karlslund writes:
> Also the numbers speak for themselves, as the --whole-file option is
> *way* faster than the block-copy method on our setup.
At the risk of jumping into the middle of this thread without
remembering everything that was discussed...
Remember that by default rsync writes a new file and then renames that
file. So a single byte change to a file requires a complete read and
write (plus the earlier read to generate the block checksums).
The --inplace option is more efficient in terms of disk IO, but the
drawback is that blocks earlier in the original file cannot be matched.
I haven't looked at the code, but I'm guessing --inplace still does
byte-by-byte matching. An additional optimization for --inplace would
be to only try to match on multiples of the block size.
Also, the matching only proceeds byte-by-byte when there is no match.
Once a match is found then the entire block is skipped. So on a file
with few changes, the byte-by-byte matching doesn't slow things down
More information about the rsync