How to make rsync faster?

[Craig Hammond]

-t is good enough for me. It checks size and time and would be heaps
faster than chksum.
It has never caused me a problem as yet.

Whether size & time over chksum is good enough for you is up to you.

Craig...

-----Original Message-----
From: Tang, Clayton (Yiqi) [mailto:yiqi.tang at lehman.com] 
Sent: Saturday, 17 November 2007 1:16 AM
To: Craig Hammond; rsync at lists.samba.org
Subject: RE: How to make rsync faster?


Thanks for the reply. How safe is it NOT to chksum? Does rsync use size
instead or date instead or both together? 

Actually splitting is what I just did. I split 250 hosts into 4 lists
and running 4 rsync jobs from the master in parallel. This causes
80%-90% total CPU usage, still runs for about 50min...


Regards,
Clayton
--
Clayton (Yiqi) Tang, LMX / Autotrader Production Management
212-526-7493, 745-7th Ave, New York, NY 10019

-----Original Message-----
From: Craig Hammond [mailto:craig at sbisolutions.com.au] 
Sent: Thursday, November 15, 2007 5:57 PM
To: Tang, Clayton (Yiqi); rsync at lists.samba.org
Subject: RE: How to make rsync faster?

I'm no rsync guru my any means, but two things spring to mind.

Use the -t option to stop all the spurious check summing.

Split your script into multiple scripts, each with a share of host
names.
Run each in parallel. Multiple rsyncs can run on the one box
concurrently.

Craig....


-----Original Message-----
From: rsync-bounces+craig=sbisolutions.com.au at lists.samba.org
[mailto:rsync-bounces+craig=sbisolutions.com.au at lists.samba.org] On
Behalf Of Tang, Clayton (Yiqi)
Sent: Friday, 16 November 2007 6:09 AM
To: rsync at lists.samba.org
Subject: How to make rsync faster?


I manage 250+ redhat linux boxes. The boxes are all setup the same way.
On a daily basis, we sync the app directory which is about 30gb out to
all hosts. The daily delta is actually less than 1gb, but since I can't
be sure if any individual box was tempered during the day, I always do a
full sync. On a monthly basis, we run with "--delete" to clean out the
stale files on the hosts.

The command I use daily is: "/usr/bin/rsync -a -e ssh", with a ksh for
loop on the 250+ host names The version is: "rsync  version 2.5.7
protocol version 26"

Since rsync must do a chksum on the local and remote box on all files,
the whole sync process takes over 2hrs even if nothing was changed.

My questions are:

1) I know I have an old version, are there performance improvements in
the later versions? I am not the SA, the process to request a new
install is lengthy.

2) Is there a "parallel rsync" program? Looping 250 times to invoke
causes rsync to checksum the local files 250 times, which is a waste of
resource. Can "parallel rsync" be considered for a future version?

3) Are there better ways to achieve what I need to do with rsync or
another tool?

Thank you,
Clayton

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - -

This message is intended only for the personal and confidential use of
the designated recipient(s) named above.  If you are not the intended
recipient of this message you are hereby notified that any review,
dissemination, distribution or copying of this message is strictly
prohibited.  This communication is for information purposes only and
should not be regarded as an offer to sell or as a solicitation of an
offer to buy any financial product, an official confirmation of any
transaction, or as an official statement of Lehman Brothers.  Email
transmission cannot be guaranteed to be secure or error-free.
Therefore, we do not represent that this information is complete or
accurate and it should not be relied upon as such.  All information is
subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained
within this communication (including any attachments) is not intended or
written to be used and cannot be used for the purpose of (i) avoiding
U.S. tax related penalties or (ii) promoting, marketing or recommending
to another party any transaction or matter addressed herein.


--
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - -

This message is intended only for the personal and confidential use of
the designated recipient(s) named above.  If you are not the intended
recipient of this message you are hereby notified that any review,
dissemination, distribution or copying of this message is strictly
prohibited.  This communication is for information purposes only and
should not be regarded as an offer to sell or as a solicitation of an
offer to buy any financial product, an official confirmation of any
transaction, or as an official statement of Lehman Brothers.  Email
transmission cannot be guaranteed to be secure or error-free.
Therefore, we do not represent that this information is complete or
accurate and it should not be relied upon as such.  All information is
subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained
within this communication (including any attachments) is not intended or
written to be used and cannot be used for the purpose of (i) avoiding
U.S. tax related penalties or (ii) promoting, marketing or recommending
to another party any transaction or matter addressed herein.