max file size
hjclaes at web.de
Mon Nov 9 10:20:17 MST 2009
Am Montag, 9. November 2009 17:48:35 schrieb Matt McCutchen:
> On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote:
> > does anybody know what's the maximum file size (terabytes?) when using
> > rsync with options --checksum and / or --inplace?
> > What file sizes have been tested in reality? Are there any experiences
> > using rsync (with --checksum and / or --inplace) for big files with
> > several / dozens or terabytes?
> I don't believe rsync has a fixed maximum size other than "what can fit
> in 64 bits", but I can't speak to any reliability issues that might come
> up with extremely large files.
I've read about a fix for overrun checksum buffers with more than some hundred
terabytes but that was just something undefined . . .
> For what purpose are you considering --checksum? In the case where the
> file's size hasn't changed (probably true for large image files), it
> will add an extra full read of the file on both sides before the
> transfer begins, which would be very expensive for multi-terabyte files.
I want to check if the following is possible:
1. transport a big block of data (several terabytes) physically from location
A to location B (very long distance) via tapes (or disks).
(Location A and B use different storage technologies.)
When the tapes arrive in location B, the block of data has changed in location
A (a program / OS is running and storing data in it).
2. shutdown application / OS in location A, rsync the delta between Location A
and B online, then restart the system in location B.
(Perhaps step 2 has to be done multiple times.)
There a lots of other aspects in this scenario, but that's another story.
More information about the rsync