Rsync Throughput issues

Rusty Carruth rcarruth at tempe.tt.slb.com
Wed Oct 10 03:01:36 EST 2001


(Courtesy copy sent directly to Ben)

Eric Whiting <ewhiting at amis.com> wrote:
> Ben Ricker wrote:
> > 
> > I am using Rsync between a Redhat Linux box and an AIX RS600. We have
> > a about 30gb of database we need to sync to a backup server. Sounds
> > good, right? The problem is that Rsync is so slow when we do the
> > initial dump. We have files that are 1 - 5gb. It takes around 14-20
> > hours to Rsync the file structure, which means our daily backup will
> > miss that days data because we are waiting for the data to rsync.
> 
> ...

I've got 2 areas that I sync across a T-1, as follows:

Number of files: 8122
Number of files transferred: 626
Total file size: 2,101,157,198 bytes
Total transferred file size: 1,855,602,142 bytes
Literal data: 2,851,745 bytes
Matched data: 1,852,750,397 bytes
File list size: 197,222
Total bytes written: 10,194,719
Total bytes read: 487,766

wrote 10,194,719 bytes  read 487,766 bytes  6,841.17 bytes/sec
total size is 2,101,157,198  speedup is 196.69

Number of files: 13,502
Number of files transferred: 0
Total file size: Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 304636
Total bytes written: 2741
Total bytes read: 304,713

wrote 2,741 bytes  read 304,713 bytes  3,967.15 bytes/sec
total size is 

The above 2 things took 27 minutes 20 seconds total.  Remeber again,
this is across a T1 (full 1.5 Mbit T1), not 10Mbit or 100Mbit. Without
ssh or anything else.

> > At first we were using ssh as the agent but we thought that might be
> > slowing us down. We then tried to use rsync as a server and push the
> > data to the module/zone of the server. Consistently, it takes roughly
> > 2 minutes to transfer 10meg of data
> 
> That seems way too long. ...

> > I thought it was a network problem so we tried ftp. A 10 mb file took
> > 90% the time (about 20 seconds. This is on a 100bit Full Duplex

20 seconds for 10 MB (I'm assuming you meant MegaByte, not MegaBit ;-) is
way slow.  ESPECIALLY for ftp. A long time ago I ran an ftp test on
a 10 Mbit link between 2 Sun Sparc2 machines.  I was able to use 99.9%
of the bandwidth (i.e. I got over 1 megaByte/sec transfer rates!). Put
another way, using 2 sparcstation 2s on a slightly occupied 10Mbit network
I should be able to transfer that file in 10 seconds.

One word of warning - FTP transfers are way efficient, and can burn all
the bandwidth up.  Most other protocols are not nearly so efficient, and
so you cannot get the speed from them.  The same machines and all worked
much slower doing rcp as transfer means (but I don't remember the numbers,
this was what, 9 years ago?).

So, whats the point?  Its this:

What's your network loading?  Have you run anything (like Sun's SE performance
toolkit (which unfortunatly only works on sun) to see if your network
is working up to snuff?

Going off in another direction: Do you have enough RAM?  Programs running
out of disk instead of out of ram are DEADLY slow...  And the amount of 
data you have to worry about is non-trivial here...  ;-)

> > switched network with both boxes on the SAME subnet.
> 
> I think you still have other problems. A ftp of a 10Mbyte file takes my
> boxes about 1s on a 100Mbit wire. 

probably due as much to file transfer rates as network speeds!
What happens if you do the transfer twice?  (So you use the cache
for the file data instead of having to read it from the disk -
assuming that you're not memory limited!)

> Be sure to try ftp both get and put. Oftentimes ...
> I think you might have some network issues. 

I'm going to have to agree.  DNS is one possiblility, but that slow
ftp reported tends to make me think its something else.

> Do a ifconfig on the
> linux box to see what sort of errors and collisions you are seeing.

And on the AIX box also!

What's the loading on that AIX box like?

Can you try rsync-ing between 2 linux boxes?  Between 2 aix boxes?
(First, try ftp transfers and see what kind of performance you get,
once THAT is fixed then see how well rsync works between the 'correct'
machines, before you worry about trying between other than the machines
you plan to use in the final setup)

> > Am I missing something?

Probably (not much help, eh? ;-)

> > Should I expect these speeds? 

No, this is slow.

> > Could there be something on the App switch throttling rsync for some reason?

Umm, I think I missed something.  What is the app switch?  Unless you mean the
network switching hub - maybe.  Is is SNMP manageable?  How much bandwidth is
it currently handling?  How much CAN it handle?

One note on switching hubs - I have one at home, and I've noticed a really
weird thing - when transferring data from my laptop (10Mbit PCMCIA on a
P133 laptop) to my 'big machine' (Celeron 450MHz with 100Mbit ethernet,
lotsa ram, etc) through the switching hub I get collisions(!!!) between
the laptop and the hub.  Scratch head time, and one of these days I'm
going to have to figure out how and why...  The point is - don't assume
your hub does all its supposed to do and can never have collisions!

I think I've deleted the ssh stuff.  I had one comment - IF you are bandwidth
limited, AND the data compresses nicely, you MIGHT get an improvement using
ssh as the transport IF it compresses AND you have spare CPU power for the
compress/decompress.  (There, did I give enough disclaimers? ;-)

rc


Rusty E. Carruth       Email:     rcarruth at Tempe.tt.slb.com or rcarruth at slb.com
Voice: (480) 345-3621  SnailMail: Schlumberger ATE                   ___
FAX:   (480) 345-8793             7855 S. River Parkway, Suite 116   \e/
Ham: N7IKQ @ 146.82+,pl 162.2     Tempe, AZ 85284-1825                V
ICBM: 33 20' 44"N   111 53' 47"W  http://tuxedo.org/~esr/ecsl/index.html
"Why would anyone choose a tool that is the primary virus vector of the known universe?" - me




More information about the rsync mailing list