--stats and performance issue
marc3 at rechte.fr
Tue Jun 12 08:42:11 MDT 2012
I have been struggling for a few days (see my former message --stats and
performance issue) with a similar problem to Alan's. They are very long
pauses and very few files / byte transferred.
For instance one run gave:
Mon Jun 11 18:01:34 CEST 2012
Number of files: 1187751
Number of files transferred: 1494
Total file size: 203.90G bytes
Total transferred file size: 11.72G bytes
Literal data: 139.17M bytes
Matched data: 11.58G bytes
File list size: 32.74M
File list generation time: 0.059 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 5.09M
Total bytes received: 175.26M
sent 5.09M bytes received 175.26M bytes 246.20K bytes/sec
total size is 203.90G speedup is 1130.61
Mon Jun 11 18:13:46 CEST 2012
And another one with half the number of files (by exluding a few big directories) taking the same amount of time !
Tue Jun 12 16:01:52 CEST 2012
Number of files: 570880
Number of files transferred: 258
Total file size: 163.37G bytes
Total transferred file size: 5.16G bytes
Literal data: 76.84M bytes
Matched data: 5.09G bytes
File list size: 15.25M
File list generation time: 0.014 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.35M
Total bytes received: 93.78M
sent 2.35M bytes received 93.78M bytes 129.47K bytes/sec
total size is 163.37G speedup is 1699.43
Tue Jun 12 16:14:14 CEST 2012
I really don't know what test to undertake to farther investigate the problem.
Le 12/06/2012 16:21, ericbambach1 at discover.com a écrit :
> Well you can remove the timestamp checks by adding "--size-only".
> If that speeds things up or removes the long "pauses" the timestamps were
> to blame. If the behavior is the same then I'm wrong. If I'm right then
> you might want to synchronize your clocks or use the " --modify-window"
> argument to rsync.
> Eric Bambach | Discover
> Senior Assoc. Programmer, Warehouse Infrastructure and Tools
> 2500 Lake Cook Road, Riverwoods IL 60015
> P: 224.405.2896 ericbambach1 at discover.com
> From: Marc Rechté<marc3 at rechte.fr>
> To:<rsync at lists.samba.org>
> Date: 06/12/2012 09:12 AM
> Subject: Re: rsync takes long pauses in xfer ?
> Sent by:<rsync-bounces at lists.samba.org>
> How would you know (debug) that size+time compare fails in a rsync session
> Le 12/06/2012 15:39, ericbambach1 at discover.com a écrit :
>> What is the rsync command used? If it hits a large group of
>> with no differences (easy to believe across 12TB) but has to compare the
>> checksums anyways (forced by you or failing the size+time compare)
>> be I/O bound with almost no corresponding network traffic as it just has
>> to send the checksums.
>> Eric Bambach | Discover
>> Senior Assoc. Programmer, Warehouse Infrastructure and Tools
>> 2500 Lake Cook Road, Riverwoods IL 60015
>> P: 224.405.2896 ericbambach1 at discover.com
>> From: Alan McKay<alan.mckay+rsync at gmail.com>
>> To:<rsync at lists.samba.org>
>> Date: 06/12/2012 08:30 AM
>> Subject: rsync takes long pauses in xfer ?
>> Sent by:<rsync-bounces at lists.samba.org>
>> Hey folks,
>> I did some googling on this but did not come up with much. I'm using
>> rsnapshot which uses rsync, and I notice some pretty long pauses in
>> the xfers as you can see on this graph from "munin". THe machine in
>> question right at the moment is doing nothing but rsyncing (
>> rsnapshoting ) some 12T of NAS storage to local disk, so there is
>> nothing else going on at all.
>> You can see a couple of spots in the graph where it just drops to
>> nothing ( checking more closely you do see a very small bit of
>> activity )
>> Here is another graph from MRTG which records the traffic from the
>> switch port side. You can see around 2am it drops off pretty low,
>> then from about 4am til about 7am it drops even lower.
>> Is that normal? Or is this something I should ask on the rsnapshot
>> list (if there is one)?
>> I did find something in google about depending on the types of files
>> it can change xfer rate but not much detail.
More information about the rsync