How to make rsync faster?

[Maxim Veksler]

On Nov 15, 2007 9:08 PM, Tang, Clayton (Yiqi) <yiqi.tang at lehman.com> wrote:
>
> I manage 250+ redhat linux boxes. The boxes are all setup the same way.
> On a daily basis, we sync the app directory which is about 30gb out to
> all hosts. The daily delta is actually less than 1gb, but since I can't
> be sure if any individual box was tempered during the day, I always do a
> full sync. On a monthly basis, we run with "--delete" to clean out the
> stale files on the hosts.
>
> The command I use daily is: "/usr/bin/rsync -a -e ssh", with a ksh for
> loop on the 250+ host names
> The version is: "rsync  version 2.5.7  protocol version 26"
>
> Since rsync must do a chksum on the local and remote box on all files,
> the whole sync process takes over 2hrs even if nothing was changed.
>
> My questions are:
>
> 1) I know I have an old version, are there performance improvements in
> the later versions? I am not the SA, the process to request a new
> install is lengthy.
>
> 2) Is there a "parallel rsync" program? Looping 250 times to invoke
> causes rsync to checksum the local files 250 times, which is a waste of
> resource. Can "parallel rsync" be considered for a future version?
>
> 3) Are there better ways to achieve what I need to do with rsync or
> another tool?
>
> Thank you,
> Clayton
>


Hello Tang,

First, for such operation you should RTFM about rsync "batch mode" [1].

Second, If I were you I would look for other solutions. Perhaps a
shared NFS storage or a copied FS based on drbd. Using rsync sounds
like a quick hack to me when you had 2 servers and 0 time to market.

I would love to hear other suggestions people have on this list for your issue.

[1] http://samba.anu.edu.au/ftp/rsync/rsync.html

-- 
Cheers,
Maxim Veksler

"Free as in Freedom" - Do u GNU ?