rsyncing very large files - extremely slow at end

[Peter Schuller]

Hello,

I am trying to rsync a tree containing very large files. The first one
encountered is a file that is 127 GB on the remote (authorative) end and
around 80 GB on the receiving end.

rsync version is 2.6.2, patched to fix the bug mentioned at:

   https://bugzilla.samba.org/show_bug.cgi?id=1529

But even so transfer is extremely slow towards the end. It will sync
approximately (or exactly, not sure) as much data as is contain in the
local file (i.e, 80 GB in this case) at normal multi-MB-per-second
speeds. And then the speed suddently goes down to perhaps 3kb/sec. I
tried increasing the block size to 25k and instead of 3kb it's now
100 kb/sec or so, but still extremely slow.

I googled and found some posts indicating there are problems with extremely
large block sizes, so I haven't tried going way above that - plus I am
unsure of whether it affects the probability of false matches.

Is it supposed to be possible to syncornize files this large at normal
speeds? If not, what would help? Increasing the currently 2^16 hash table
size while retaining a small block size perhaps? Something else?

Any input would be appreciated.

Thanks!

-- 
/ Peter Schuller, InfiDyne Technologies HB

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller at infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org