Rsync Throughput issues
ewhiting at amis.com
Wed Oct 10 01:14:00 EST 2001
Ben Ricker wrote:
> I am using Rsync between a Redhat Linux box and an AIX RS600. We have
> a about 30gb of database we need to sync to a backup server. Sounds
> good, right? The problem is that Rsync is so slow when we do the
> initial dump. We have files that are 1 - 5gb. It takes around 14-20
> hours to Rsync the file structure, which means our daily backup will
> miss that days data because we are waiting for the data to rsync.
How many files are in the tree? Are all the files 1-5G? Are the source
files on new/fast disk? For older systems, the 100Mbit network is often
faster than disk.
I have a 40G tree with 1 million files in it. I just killed the tree on
a development box and restarted a rsync to it. I think it might take 10+
hours mainly because of how many files are in the tree -- not so much
because of the total size. I'll let you know. I know it takes about 2
hours just to verify that all 1 million files are in sync even when
there are no changes to the tree.
> At first we were using ssh as the agent but we thought that might be
> slowing us down. We then tried to use rsync as a server and push the
> data to the module/zone of the server. Consistently, it takes roughly
> 2 minutes to transfer 10meg of data
That seems way too long. Sounds like a reverse dns lookup failure. Long
shot debug question -- Do you have dns/nis/files all working properly?
Sometimes reverse dns lookups can take a few minutes to fail or time out
and thus affect the overall speed of transfers. Repeated reverse dns
failures can add up fast.
> I thought it was a network problem so we tried ftp. A 10 mb file took
> 90% the time (about 20 seconds. This is on a 100bit Full Duplex
> switched network with both boxes on the SAME subnet.
I think you still have other problems. A ftp of a 10Mbyte file takes my
boxes about 1s on a 100Mbit wire.
Be sure to try ftp both get and put. Oftentimes a duplex mismatch or
speed problem will only affect traffic in one direction and not the
other. I think you might have some network issues. Do a ifconfig on the
linux box to see what sort of errors and collisions you are seeing.
> Am I missing something? Should I expect these speeds? Could there be
> something on the App switch throttling rsync for some reason?
> Any thought will help,
> Ben Ricker
> System Administrator
More information about the rsync