Rsync performance with very large files

Eric Cron ericcron at yahoo.com
Fri Jan 8 13:20:30 MST 2010


We're having a performance issue when attempting to rsync a very large file. Transfer rate is only 1.5MB/sec.  My issue looks very similar to this one: 
 
http://www.mail-archive.com/rsync@lists.samba.org/msg17812.html  
 
In that thread, a  'dynamic_hash.diff' patch was developed to work around this issue. I applied the 'dynamic_hash' patch included in the 2.6.7 src, but it didn't help.   
 
We are trying to evaluate the possibility of using rsync as an
alternative to IBM's FlashCopy, which only works within the storage
pool controlled by our San Volume Controller.

Some details about our test environment:
 
- Sender and Receiver are both POWER6 servers running AIX 5.3 
- Fiber attached disk, DS8300  storage 
- Gigabit network (Hypervisor Virtual I/O) 
- Test file is 232GB 
- I've tried rsync version 3.0.7 (vanilla) and 2.6.7 with the dynamic_hash.diff patch, both compiled with IBM's xlc compiler.  
Same behavior with both versions.
- It takes approx 1.5 hours to 'consider' the file before transfers begin, no big deal... 
- Once the changes are being sent, the rate is only 1.5MB/sec 
- Nothing is using either the source or destination files, only rsync (these are test servers.) 
- Both servers appear healthy, no CPU or memory problems.  
 
Just hoping somebody might have some insight.  The thread I linked above didn't have any info indicating success or failure of the patch - the original poster didn't provide any feedback.   
 
Eric Cron 
ericcron at yahoo.com


      


More information about the rsync mailing list