rsync 1tb+ each day

[Donovan Baarda]

On Thu, 2003-02-06 at 10:01, Eric Whiting wrote:
> Kenny Gorman wrote:
[...]
> I tested with a small 256M datafile. rsync -av is showing me about 200kBytes of
> changes in the datafile between each snapshot. (about 1/1000th of the file has
> actually changed between the hot backups) Rsync reports the speedup as somewhere
> between 800 and 1000 just as I would expect. This speedup is a number that
> accounts only for bytes transferred (or not transferred) -- not real time. When
> I time the rsync -av runs and compare to a rsync -W the real run times are
> approximately the same. (similar results if I rm the file on the dest and rsync
> -av). block checksum and rewrite overhead right? 
[...]
> Similar speedup numbers for some 1G database files -- but the real time is 2x
> when the destination is there and mostly correct. For this type of data it is
> 'faster' from a time standpoint to remove the destination file before running
> rsync. Not what I would expect. 

You are probably seeing 2x because you are running rsync on a 1G file
without a sufficiently large block size. When you do this rsync actually
transfers the file twice; once using the rsync algorithm, and again
sending the whole file because the rsync transfer corrupted the file (as
indicated by the final checksum). There is a 99.7% chance the rsync
transfer will fail for a 1G file and the default block size.

This is because rsync doesn't use sufficiently large signature checksums
for large files, which results in corruption of the transferred file.
Fortunately rsync detects this using a final checksum, and falls back to
a whole file transfer.

To benefit (95%+ chance of success) from using rsync on a 1GB file, you
need to use a block size of 75kB or greater. For details, see;

http://www.mail-archive.com/rsync@lists.samba.org/msg05219.html

In any case, you might find that setting larger block sizes improves
performance when you have a fast network; large block sizes reduce the
CPU overhead at the expense of reduced traffic savings. You might find a
"sweet spot" somewhere that is faster than "-W" for your particular
situation. 

-- 
----------------------------------------------------------------------
ABO: finger abo at minkirri.apana.org.au for more info, including pgp key
----------------------------------------------------------------------