Thought on large files

[Matt McCutchen]

On Wed, 2008-01-23 at 13:38 +0900, Brendan Grieve wrote:
> Lets say 
> the file, whatever it is, is a 10Gb file, and that some small amount of 
> data changes in it. This is efficiently sent accross by rsync, BUT the 
> rsync server side will correctly break the hard-link and create a new 
> file with the changed bits. This means, if even 1 byte of that 10Gb file 
> changes, you now have to store that whole file again.

> What my thoughts were is that if the server could transparently break a 
> large file into chunks and store them that way, then one can still make 
> use of hard-links efficiently.

This is a fine idea, but I don't think support for this should be added
to rsync.  Instead, I suggest that you use rdiff-backup
( http://www.nongnu.org/rdiff-backup/ ), a backup tool that stores an
ordinary latest snapshot of the source along with reverse deltas for
previous snapshots and redundant attribute information both in its own
format.

Matt