Rsync to remote host much faster than 10k/sec speeds on local rsync

[Will Smith]

I run several remote rsyncs off a CentOS 4 server and also back up between
two drives on the same server.

Performance is far better on the remote rsyncs and the local rsync only
manages 10k/sec despite being on modern hardware.

Here are the stats from a local backup:

Number of files: 979548
Number of files transferred: 2289
Total file size: 24703651523 bytes
Total transferred file size: 20870616 bytes
Literal data: 20870616 bytes
Matched data: 0 bytes
File list size: 43811752
File list generation time: 430.928 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 64787950
Total bytes received: 50426

sent 64787950 bytes  received 50426 bytes  10730.39 bytes/sec
total size is 24703651523  speedup is 381.00

Here are the stats from a remote backup:

Number of files: 979318
Number of files transferred: 543
Total file size: 29764063787 bytes
Total transferred file size: 8771412 bytes
Literal data: 8771412 bytes
Matched data: 0 bytes
File list size: 43796536
File list generation time: 179.610 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 52592412
Total bytes received: 10876

sent 52592412 bytes  received 10876 bytes  204284.61 bytes/sec
total size is 29764063787  speedup is 565.82
What I have found is that if I empty the destination directory on the local
backup disk, so rsync has to start from scratch and copy all files,
performance is as to be expected but this isn't feasible for a regular
backup routine as there are almost a million files.

I have also noticed that the load on the local rsync is much higher than the
remote rsyncs, with 90%+ cpu usage. I have tried --bwlimit 100 in a vague
hope it might affect disk io but it didn't help. Rsync version is 2.6.9
1.el4.rf from rpm. The directory size of the destination backup directory is
around 30gb. There is one directory where the majority of the files are and
they are jpgs between 5k-150kb.

The command I am running is simple: /usr/bin/rsync -a --bwlimit=100 --stats
/src/dir1 /src/dir2 /backup/dir
-------------- next part --------------
HTML attachment scrubbed and removed