Where is my bottleneck?
Tim Conway
conway at us.ibm.com
Thu Mar 11 20:53:00 GMT 2004
This seems obvious, but it's easy to forget. Are all the filesystems on
both ends locally-attached? If they're NFS or SMBFS, rsync is about a
third the speed of SCP, unless you use "-W" to tell it to send the whole
file if the timestamp/size don't match... in most cases, anyway. ethernet
lan and dialup wan to the remote server, I'd probably go ahead and let it
use the rsync algorithm.
Try not using "-z"... Actually, maybe you already aren't.
Here's a strange one for you to try.
I'm sure you're running wide open. Try using --bwlimit= to hold it back a
bit. If your network is chattering, backing off the throttle might get
you down the track a hair faster.
Since everything is inside, try opening up rsh for a while (don't leave it
open if you're internet-connected.)
on host1
time dd if=/dev/zero bs=1024k count=1024k |rsh host2 dd of=/dev/null
That'll give you a good idea of what your network can do.
Try tarring up whatever you're sending by rsync, sending it straight to
/dev/null.
Feed it through dd for a byte count.
time tar -cf - whatever |dd of=/dev/null bs=1024k
Some controllers don't do very good high-speed/volume I/O. Cache will
keep things seeming fast, but when you go past cache, you see the true
speed of your disk subsystem. The above test will show you some of that.
if read's good, check write
time dd if=/dev/zero bs=1024k count=1024k
of=/bigfilesystem/fileyouregoingtodelete
Tim Conway
Unix System Administration
Contractor - IBM Global Services
conway at us.ibm.com
I'm running rsync via ssh on two machines connected with 100 Mbit/s
Ethernet cards (at high speed) via a Linksys switch. It's all right
here in a single closet. I'm sending large files (initial transfer,
nothing preexists on the destination machine) and seeing transfer rates
in the 2 Mbit/s range. The CPUs on both machines are more than 90%
idle, and I upped the --block-size to 256 kilobytes. Disk activity is
very light. Fedora Core 1, i686.
I don't expect 100 Mbit/s by any means, but 2? Is this typical? What
am I missing?
More information about the rsync
mailing list