Thought on large files

[Brendan Grieve]

Hi There,

I've been toying around with the code of rsync on and off for a while, 
and I had a thought that I would like some comments on. Its to do with 
very large files and disk space.

One of the common uses of rsync is to use it as a backup program. A 
client connects to the rsync server, and sends over any changed files. 
If the client has very large files that have changed marginally, then 
rsync efficiently only sends the changed bits.

On the server side, one may have it set up to create 'snapshots' of the 
existing data there by hardlinking that data to another directory 
periodically. Theres plenty of documentation on the web how to do this 
so I won't go into it further.

This is very effective and uses quite little disk space since a file 
that does not change effectively doesn't take up any more disk space 
(not much more anyway), even if it exists now in many snapshots.

One place where this falls down is if the file is very large. Lets say 
the file, whatever it is, is a 10Gb file, and that some small amount of 
data changes in it. This is efficiently sent accross by rsync, BUT the 
rsync server side will correctly break the hard-link and create a new 
file with the changed bits. This means, if even 1 byte of that 10Gb file 
changes, you now have to store that whole file again.

I won't get into the whole issue of why one would have big files etc... 
I see it all the time, especially in the Microsoft world, with Outlook 
PST files, and Microsoft Exchange Database files.

What my thoughts were is that if the server could transparently break a 
large file into chunks and store them that way, then one can still make 
use of hard-links efficiently.

For example, going back to a 10Gb Exchange Database file, its likely not 
going to change too much during use. So if the server stored the huge 
clumsy 'priv1.edb' as:
  .priv1.edb._somemagicstring_.1
  .priv1.edb._somemagicstring_.2
etc...

and intelligently only broke the 'hard-links' of the bits that actually 
change, then it all works well. One could have an option to enable this 
for files bigger than a certain size, and break them into specific sized 
chunks.

One could quite rightly argue that this changes rsync from a tool that 
synchronizes data between places to a dedicated backup tool (as the two 
sides will now have physically different data), however I could see it 
being useful, especially since it wouldn't need changes on the client 
side as the server still presents it as just one file.

What are your comments? Good idea? Stupid idea? Been done before? Does 
anyone have some hints about where in the code I should look to make 
these changes so I can test it out?



Brendan Grieve