max file size

[Heinz-Josef Claes]

Am Montag, 9. November 2009 17:48:35 schrieb Matt McCutchen:
> On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote:
> > does anybody know what's the maximum file size (terabytes?) when using
> > rsync with options --checksum and / or --inplace?
> >
> > What file sizes have been tested in reality? Are there any experiences
> > using rsync (with --checksum and / or --inplace) for big files with
> > several / dozens or terabytes?
> 
> I don't believe rsync has a fixed maximum size other than "what can fit
> in 64 bits", but I can't speak to any reliability issues that might come
> up with extremely large files.
> 
I've read about a fix for overrun checksum buffers with more than some hundred 
terabytes but that was just something undefined . . .

> For what purpose are you considering --checksum?  In the case where the
> file's size hasn't changed (probably true for large image files), it
> will add an extra full read of the file on both sides before the
> transfer begins, which would be very expensive for multi-terabyte files.

I want to check if the following is possible:

1. transport a big block of data (several terabytes) physically from location 
A to location B (very long distance) via tapes (or disks).
(Location A and B use different storage technologies.)

When the tapes arrive in location B, the block of data has changed in location 
A (a program / OS is running and storing data in it).

2. shutdown application / OS in location A, rsync the delta between Location A 
and B online, then restart the system in location B.

(Perhaps step 2 has to be done multiple times.)

--
There a lots of other aspects in this scenario, but that's another story.

Regards, HJC