--stats and performance issue

Tue Jun 12 08:42:11 MDT 2012

Thanks Eric,

I have been struggling for a few days (see my former message --stats and 
performance issue) with a similar problem to Alan's. They are very long 
pauses and very few files / byte transferred.

For instance one run gave:

Mon Jun 11 18:01:34 CEST 2012

Number of files: 1187751
Number of files transferred: 1494
Total file size: 203.90G bytes
Total transferred file size: 11.72G bytes
Literal data: 139.17M bytes
Matched data: 11.58G bytes
File list size: 32.74M
File list generation time: 0.059 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.09M
Total bytes received: 175.26M

sent 5.09M bytes  received 175.26M bytes  246.20K bytes/sec
total size is 203.90G  speedup is 1130.61

Mon Jun 11 18:13:46 CEST 2012

And another one with half the number of files (by exluding a few big directories) taking the same amount of time !

Tue Jun 12 16:01:52 CEST 2012

Number of files: 570880
Number of files transferred: 258
Total file size: 163.37G bytes
Total transferred file size: 5.16G bytes
Literal data: 76.84M bytes
Matched data: 5.09G bytes
File list size: 15.25M
File list generation time: 0.014 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.35M
Total bytes received: 93.78M

sent 2.35M bytes  received 93.78M bytes  129.47K bytes/sec
total size is 163.37G  speedup is 1699.43

Tue Jun 12 16:14:14 CEST 2012

I really don't know what test to undertake to farther investigate the problem.

Thanks

Le 12/06/2012 16:21, ericbambach1 at discover.com a écrit :
> Marc,
>
>          Well you can remove the timestamp checks by adding  "--size-only".
> If that speeds things up or removes the long "pauses" the timestamps were
> to blame. If the behavior is the same then I'm wrong. If I'm right then
> you might want to synchronize your clocks or use the " --modify-window"
> argument to rsync.
>
> Eric Bambach | Discover
> Senior Assoc. Programmer, Warehouse Infrastructure and Tools
> 2500 Lake Cook Road, Riverwoods IL 60015
> P: 224.405.2896 ericbambach1 at discover.com
>
>
>
>
> From:   Marc Rechté<marc3 at rechte.fr>
> To:<rsync at lists.samba.org>
> Date:   06/12/2012 09:12 AM
> Subject:        Re: rsync takes long pauses in xfer ?
> Sent by:<rsync-bounces at lists.samba.org>
>
>
>
> Eric,
>
> How would you know (debug) that size+time compare fails in a rsync session
> ?
>
> Thanks
> Marc
>
> Le 12/06/2012 15:39, ericbambach1 at discover.com a écrit :
>> Alan,
>>
>>           What is the rsync command used? If it hits a large group of
> files
>> with no differences (easy to believe across 12TB) but has to compare the
>> checksums anyways (forced by you or failing the size+time compare)
> you'll
>> be I/O bound with almost no corresponding network traffic as it just has
>> to send the checksums.
>>
>>
>> Eric Bambach | Discover
>> Senior Assoc. Programmer, Warehouse Infrastructure and Tools
>> 2500 Lake Cook Road, Riverwoods IL 60015
>> P: 224.405.2896 ericbambach1 at discover.com
>>
>>
>>
>>
>> From:   Alan McKay<alan.mckay+rsync at gmail.com>
>> To:<rsync at lists.samba.org>
>> Date:   06/12/2012 08:30 AM
>> Subject:        rsync takes long pauses in xfer ?
>> Sent by:<rsync-bounces at lists.samba.org>
>>
>>
>>
>> Hey folks,
>>
>> I did some googling on this but did not come up with much.  I'm using
>> rsnapshot which uses rsync, and I notice some pretty long pauses in
>> the xfers as you can see on this graph from "munin".   THe machine in
>> question right at the moment is doing nothing but rsyncing (
>> rsnapshoting ) some 12T of NAS storage to local disk, so there is
>> nothing else going on at all.
>>
>>
> https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250
>
>>
>> You can see a couple of spots in the graph where it just drops to
>> nothing ( checking more closely you do see a very small bit of
>> activity )
>>
>> Here is another graph from MRTG which records the traffic from the
>> switch port side.  You can see around 2am it drops off pretty low,
>> then from about 4am til about 7am it drops even lower.
>>
>>
> https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753140504922328322
>
>>
>> Is that normal?  Or is this something I should ask on the rsnapshot
>> list (if there is one)?
>>
>> I did find something in google about depending on the types of files
>> it can change xfer rate but not much detail.
>>
>> thanks,
>> -Alan

-- 