max file size
Matt McCutchen
matt at mattmccutchen.net
Thu Nov 12 23:38:48 MST 2009
On Mon, 2009-11-09 at 18:20 +0100, Heinz-Josef Claes wrote:
> Am Montag, 9. November 2009 17:48:35 schrieb Matt McCutchen:
> > On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote:
> > > does anybody know what's the maximum file size (terabytes?) when using
> > > rsync with options --checksum and / or --inplace?
> > >
> > > What file sizes have been tested in reality? Are there any experiences
> > > using rsync (with --checksum and / or --inplace) for big files with
> > > several / dozens or terabytes?
> >
> > I don't believe rsync has a fixed maximum size other than "what can fit
> > in 64 bits", but I can't speak to any reliability issues that might come
> > up with extremely large files.
> >
> I've read about a fix for overrun checksum buffers with more than some hundred
> terabytes but that was just something undefined . . .
Indeed, I forgot about that. The delta-transfer algorithm doesn't work
for files longer than 2^31 blocks. With the default maximum block size
of 2^17, the limit is 2^48 bytes or 256 TB. You could stretch the limit
by fixing a larger block size with --block-size . See:
https://bugzilla.samba.org/show_bug.cgi?id=5459#c2
> > For what purpose are you considering --checksum? In the case where the
> > file's size hasn't changed (probably true for large image files), it
> > will add an extra full read of the file on both sides before the
> > transfer begins, which would be very expensive for multi-terabyte files.
>
> I want to check if the following is possible:
>
> 1. transport a big block of data (several terabytes) physically from location
> A to location B (very long distance) via tapes (or disks).
> (Location A and B use different storage technologies.)
>
> When the tapes arrive in location B, the block of data has changed in location
> A (a program / OS is running and storing data in it).
>
> 2. shutdown application / OS in location A, rsync the delta between Location A
> and B online, then restart the system in location B.
>
> (Perhaps step 2 has to be done multiple times.)
Since the source and destination versions are practically certain to
differ, --checksum would serve no purpose. See the man page description
of --checksum.
--
Matt
More information about the rsync
mailing list