rsync and many files

[Steven Levine]

In <F992406D6E81B54DBB33210217FE7AFD07F80C84 at exchange1.mtb.netclusive.de>,
on 06/06/11
   at 12:04 PM, Cliff Simon <cliff.simon at netclusive.com> said:

Hi,

>We are using rsync via rsnapshot, but this is not elementary. It is used
>to backup many (above 100 servers) and works very well. Now there is one
>server with many (several millions) files. The files are not very big, so
>the complete backup is about 500 GB.

>Now my problem is, that the backup needs about 14 hours - the most time
>is to generate the filelist and check whether the files are new/changed
>or not.

>My rsync-command is:
>/usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative
>--delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh
>--link-dest=/dest.path/daily.1/ root at 192.x.x.x:/path.to.backup/

>Do you have an idea to reduce the backup time?

A bit of math says 2*10^6 / 14 hours is about 40 files/second.  How fast
do you think rsync should be and how does this compare to backups on your
other servers?

Are you sure the it is not the hardware that is limiting the rsync's
performance?

Based on my knowledge of the rsync sources, I believe the file list
generation algorithms are pretty efficient.  There is quite a bit of code
in the code path, but it's hard to avoid this given the number of options
available to control the sync process.

Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53 at earthlink.net>  eCS/Warp/DIY etc.
www.scoug.com www.ecomstation.com
----------------------------------------------------------------------