Rsync help needed...

[Matt McCutchen]

On Tue, 2006-02-28 at 07:43 -0800, lsk wrote:
> Matt I have strange results to report I transferred 300 oracle datafiles of
> total 30 GB in size. Using the option " rsync -zv --no-whole-file --stats"
> it took 1:15 min and using "rsync -cvz" options earlier had took 1:25 min so
> there wasn't much time savings I thought it would be a big difference since
> the files are there and only little is modified (header info) + additional
> datafiles.

Hm.  I guess the checksumming was only taking 10 minutes, which means
you must have a fast disk!  Adding --inplace should give a bigger
improvement.

> {{ Other info is I am running "rsync version 2.6.5  protocol version 29"  on
> source and "rsync  version 2.5.2  protocol version 26" on destination server
> }}

You might want to upgrade the destination server.

> Also one more thing from the stats rsync has transferred few files more than
> once "Number of files transferred: 2" Why is this so ?
> 
> Number of files: 1
> Number of files transferred: 2
> Total file size: 167780352 bytes
> Total transferred file size: 335560704 bytes
> Literal data: 67010560 bytes
> Matched data: 268550144 bytes
> File list size: 42
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 184089
> Total bytes received: 254026
> 
> sent 184089 bytes  received 254026 bytes  2131.95 bytes/sec
> total size is 167780352  speedup is 382.96

Once in a while, a block hash collision will occur in rsync's
incremental transfer algorithm, causing the receiver to falsely decide
it already has part of the file.  When this happens, the after-transfer
checksum fails, a warning is printed, and the transfer starts over; this
second transfer is being counted in the stats.

It looks like you're invoking rsync once per file.  If you create a file
listing all the filenames and invoke rsync once overall with
--files-from, you can reduce the overhead of SSH-ing and setting up the
rsync process triangle.
-- 
Matt McCutchen
hashproduct at verizon.net
http://hashproduct.metaesthetics.net/