rsync performance

[Ralf Fassel]

We're using rsync 2.6.3 to sync two DELL PowerEdge servers with both
Redhat-EL4 and otherwise nearly identical hardware (2.8/3GHz, 1GB RAM
each).  The source machine has a SCSI-RAID1, the destination a
SATA-RAID1 disk attached.

There are 5 filesystems which are rsynced via ssh.  On the smaller
filesystems with ~200.000 files/7GB, rsync takes 1-3 minutes:
    lion:/atg/ ========= Tue Aug 22 12:13:29 CEST 2006 ==================
    receiving file list ... done
  --<snip-snip>--
    Number of files: 233780
    Number of files transferred: 2
    Total file size: 6716087965 bytes
    Total transferred file size: 1470 bytes
    Literal data: 1470 bytes
    Matched data: 0 bytes
    File list size: 5015765
    Total bytes sent: 68
    Total bytes received: 5017339

    sent 68 bytes  received 5017339 bytes  56060.41 bytes/sec
    total size is 6716087965  speedup is 1338.56
    lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 ==================

233780 files/89 sec = 2626 files/sec

On the large filesystems with ~1.200.000 files/17GB, rsync takes 30
minutes, even when only few files change (and even when invoked with -n):
    lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 ==================
    receiving file list ... done
  --<snip-snip>--
    Number of files: 1232323
    Number of files transferred: 124
    Total file size: 17103825898 bytes
    Total transferred file size: 1416942923 bytes
    Literal data: 27658581 bytes
    Matched data: 1389311598 bytes
    File list size: 95325482
    Total bytes sent: 643209
    Total bytes received: 123363867

    sent 643209 bytes  received 123363867 bytes  62963.73 bytes/sec
    total size is 17103825898  speedup is 137.93
    lion:/homes/atg/ ========= Tue Aug 22 12:47:48 CEST 2006 ==================

1232323 files / 1960 sec = 629 files/sec

In comparison, on the large filesystem, 'find'ing all files on the
source system and 'stat'ing them takes 1.75 minutes:
    source# time find /homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l
    1232287
    real 1m41.987s
    user 0m7.075s
    sys	 0m33.216s
On the destination 4 minutes:
    dest# time find /backup/lion-homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l
    1233202
    real 4m3.650s
    user 0m9.895s
    sys	 0m41.047s

Receiving the 123363867 bytes should not take longer than 20sec on a
100Mbit LAN, and transferring a GB to disk should not take longer than
say a minute (we have write transfer rates of ~50MB/sec on those disks
on each machine).

So what is rsync doing in addition that it takes >30 minutes to sync
the large filesystem?

Thanks
R'