rsync performance
Tim Moore
rsync at nsr500.net
Wed Aug 23 16:01:35 GMT 2006
Try using major subdirectories one level lower. Example sequence from my
automated home machine backup:
rsync -ax --stats / /rsync_backup/
rsync -ax --stats /usr/ /rsync_backup/usr/
rsync -ax --stats /home/ /rsync_backup/home/
Compare to "rsync -a --stats / /backup/"
Ralf Fassel wrote:
> We're using rsync 2.6.3 to sync two DELL PowerEdge servers with both
> Redhat-EL4 and otherwise nearly identical hardware (2.8/3GHz, 1GB RAM
> each). The source machine has a SCSI-RAID1, the destination a
> SATA-RAID1 disk attached.
>
> There are 5 filesystems which are rsynced via ssh. On the smaller
> filesystems with ~200.000 files/7GB, rsync takes 1-3 minutes:
> lion:/atg/ ========= Tue Aug 22 12:13:29 CEST 2006 ==================
> receiving file list ... done
> --<snip-snip>--
> Number of files: 233780
> Number of files transferred: 2
> Total file size: 6716087965 bytes
> Total transferred file size: 1470 bytes
> Literal data: 1470 bytes
> Matched data: 0 bytes
> File list size: 5015765
> Total bytes sent: 68
> Total bytes received: 5017339
>
> sent 68 bytes received 5017339 bytes 56060.41 bytes/sec
> total size is 6716087965 speedup is 1338.56
> lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 ==================
>
> 233780 files/89 sec = 2626 files/sec
>
> On the large filesystems with ~1.200.000 files/17GB, rsync takes 30
> minutes, even when only few files change (and even when invoked with -n):
> lion:/homes/ ========= Tue Aug 22 12:14:58 CEST 2006 ==================
> receiving file list ... done
> --<snip-snip>--
> Number of files: 1232323
> Number of files transferred: 124
> Total file size: 17103825898 bytes
> Total transferred file size: 1416942923 bytes
> Literal data: 27658581 bytes
> Matched data: 1389311598 bytes
> File list size: 95325482
> Total bytes sent: 643209
> Total bytes received: 123363867
>
> sent 643209 bytes received 123363867 bytes 62963.73 bytes/sec
> total size is 17103825898 speedup is 137.93
> lion:/homes/atg/ ========= Tue Aug 22 12:47:48 CEST 2006 ==================
>
> 1232323 files / 1960 sec = 629 files/sec
>
> In comparison, on the large filesystem, 'find'ing all files on the
> source system and 'stat'ing them takes 1.75 minutes:
> source# time find /homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l
> 1232287
> real 1m41.987s
> user 0m7.075s
> sys 0m33.216s
> On the destination 4 minutes:
> dest# time find /backup/lion-homes -xdev -print0 | xargs -0 stat -c "%s %Y" | wc -l
> 1233202
> real 4m3.650s
> user 0m9.895s
> sys 0m41.047s
>
> Receiving the 123363867 bytes should not take longer than 20sec on a
> 100Mbit LAN, and transferring a GB to disk should not take longer than
> say a minute (we have write transfer rates of ~50MB/sec on those disks
> on each machine).
>
> So what is rsync doing in addition that it takes >30 minutes to sync
> the large filesystem?
>
> Thanks
> R'
--
More information about the rsync
mailing list