rsync takes long pauses in xfer ?

Tue Jun 12 10:00:03 MDT 2012

Kevin,

        I'm not saying the stat() call is slow  or trying to reduce 
syscalls.  If the quick check fails due to differing timestamps the entire 
transfer will be significantly slower because rsync has to defer to the 
full checksum algorithm just to find out the files are identical which 
means the entire file has to be read on both ends causing significantly 
more work (IO) and hence a slowdown.

        Bad timestamps is the only case where ignoring the timestamps will 
significantly increase performance. The root problem is your timestamps 
are off between systems.

        I believe we are saying the same thing.

Eric Bambach | Discover
Senior Assoc. Programmer, Warehouse Infrastructure and Tools
2500 Lake Cook Road, Riverwoods IL 60015
P: 224.405.2896 ericbambach1 at discover.com

From:   Kevin Korb <kmk at sanitarium.net>
To:     <rsync at lists.samba.org>
Date:   06/12/2012 10:02 AM
Subject:        Re: rsync takes long pauses in xfer ?
Sent by:        <rsync-bounces at lists.samba.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The time stamp and file size are both returned by a single call to
stat().  Not looking at the time stamp value doesn't make it any
faster.  The option is there for cases where you know the time stamps
don't match but you don't want to copy the file because of that.

On 06/12/12 10:21, ericbambach1 at discover.com wrote:
> Marc,
> 
> Well you can remove the timestamp checks by adding  "--size-only".
>  If that speeds things up or removes the long "pauses" the
> timestamps were to blame. If the behavior is the same then I'm
> wrong. If I'm right then you might want to synchronize your clocks
> or use the " --modify-window" argument to rsync.
> 
> Eric Bambach | Discover Senior Assoc. Programmer, Warehouse
> Infrastructure and Tools 2500 Lake Cook Road, Riverwoods IL 60015 
> P: 224.405.2896 ericbambach1 at discover.com
> 
> 
> 
> 
> From:   Marc Rechté <marc3 at rechte.fr> To:
> <rsync at lists.samba.org> Date:   06/12/2012 09:12 AM Subject:
> Re: rsync takes long pauses in xfer ? Sent by:
> <rsync-bounces at lists.samba.org>
> 
> 
> 
> Eric,
> 
> How would you know (debug) that size+time compare fails in a rsync
> session ?
> 
> Thanks Marc
> 
> Le 12/06/2012 15:39, ericbambach1 at discover.com a écrit :
>> Alan,
>> 
>> What is the rsync command used? If it hits a large group of
> files
>> with no differences (easy to believe across 12TB) but has to
>> compare the checksums anyways (forced by you or failing the
>> size+time compare)
> you'll
>> be I/O bound with almost no corresponding network traffic as it
>> just has to send the checksums.
>> 
>> 
>> Eric Bambach | Discover Senior Assoc. Programmer, Warehouse
>> Infrastructure and Tools 2500 Lake Cook Road, Riverwoods IL
>> 60015 P: 224.405.2896 ericbambach1 at discover.com
>> 
>> 
>> 
>> 
>> From:   Alan McKay<alan.mckay+rsync at gmail.com> 
>> To:<rsync at lists.samba.org> Date:   06/12/2012 08:30 AM Subject:
>> rsync takes long pauses in xfer ? Sent
>> by:<rsync-bounces at lists.samba.org>
>> 
>> 
>> 
>> Hey folks,
>> 
>> I did some googling on this but did not come up with much.  I'm
>> using rsnapshot which uses rsync, and I notice some pretty long
>> pauses in the xfers as you can see on this graph from "munin".
>> THe machine in question right at the moment is doing nothing but
>> rsyncing ( rsnapshoting ) some 12T of NAS storage to local disk,
>> so there is nothing else going on at all.
>> 
>> 
> 
https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250

>
> 
>> 
>> 
>> You can see a couple of spots in the graph where it just drops
>> to nothing ( checking more closely you do see a very small bit
>> of activity )
>> 
>> Here is another graph from MRTG which records the traffic from
>> the switch port side.  You can see around 2am it drops off pretty
>> low, then from about 4am til about 7am it drops even lower.
>> 
>> 
> 
https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753140504922328322

>
> 
>> 
>> 
>> Is that normal?  Or is this something I should ask on the
>> rsnapshot list (if there is one)?
>> 
>> I did find something in google about depending on the types of
>> files it can change xfer rate but not much detail.
>> 
>> thanks, -Alan
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
                 Kevin Korb Phone:    (407) 252-6853
                 Systems Administrator                           Internet:
                 FutureQuest, Inc. Kevin at FutureQuest.net  (work)
                 Orlando, Florida kmk at sanitarium.net (personal)
                 Web page:                                               
http://www.sanitarium.net/
                 PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/XWhgACgkQVKC1jlbQAQfAqwCgwgEv4V8n0E/mCVTCJ0yd/yPp
y5cAniy4LrBTSALpGAZ71s1kQRkSezOO
=bo0x
-----END PGP SIGNATURE-----
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: 
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Please consider the environment before printing this email.