How to make rsync faster?
maxim.veksler at gmail.com
Thu Nov 15 19:36:11 GMT 2007
On Nov 15, 2007 9:08 PM, Tang, Clayton (Yiqi) <yiqi.tang at lehman.com> wrote:
> I manage 250+ redhat linux boxes. The boxes are all setup the same way.
> On a daily basis, we sync the app directory which is about 30gb out to
> all hosts. The daily delta is actually less than 1gb, but since I can't
> be sure if any individual box was tempered during the day, I always do a
> full sync. On a monthly basis, we run with "--delete" to clean out the
> stale files on the hosts.
> The command I use daily is: "/usr/bin/rsync -a -e ssh", with a ksh for
> loop on the 250+ host names
> The version is: "rsync version 2.5.7 protocol version 26"
> Since rsync must do a chksum on the local and remote box on all files,
> the whole sync process takes over 2hrs even if nothing was changed.
> My questions are:
> 1) I know I have an old version, are there performance improvements in
> the later versions? I am not the SA, the process to request a new
> install is lengthy.
> 2) Is there a "parallel rsync" program? Looping 250 times to invoke
> causes rsync to checksum the local files 250 times, which is a waste of
> resource. Can "parallel rsync" be considered for a future version?
> 3) Are there better ways to achieve what I need to do with rsync or
> another tool?
> Thank you,
First, for such operation you should RTFM about rsync "batch mode" .
Second, If I were you I would look for other solutions. Perhaps a
shared NFS storage or a copied FS based on drbd. Using rsync sounds
like a quick hack to me when you had 2 servers and 0 time to market.
I would love to hear other suggestions people have on this list for your issue.
"Free as in Freedom" - Do u GNU ?
More information about the rsync