Duration

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The rsync is doing a whole lot more than transferring 600MB of data.  It
is transferring 600MB of differences.  It is going through 27.4 million
files on one end and possibly that many on the other.  That is a 402MB
listing of files plus as many as 54.8 million calls to stat().

Rsync is not about minimizing transfer time.  It is about minimizing
network IO at the expense of CPU and disk IO.  Rsync reduced your
network IO from 185.39GB to 600MB only 200 of which was actually
differences in your data.

IOW rsync did its job.

The unfortunate truth is that no file based tool will perform anywhere
near wire speeds when it has millions of files to deal with.  If you
want something wire speed then you should look for a block level tool.

On 09/12/11 21:54, David Somers-Harris wrote:
> No, I don't have checksum based checking enabled.
> 
> Sorry for neglecting to mention the options I'm using, here they are:
> 
> *rsync -vah --exclude=".*" --delete --stats*
> 
> Would anything in there cause it to be unnecessarily slow? It's taking 3
> hours but shouldn't take more than 5 minutes...
> 
> 
> On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness <sandon at van-ness.com
> <mailto:sandon at van-ness.com>> wrote:
> 
>     It could definitely be normal if say you have checksum based
>     checking enabled...
> 
> 
>     On 09/12/2011 01:56 AM, David Somers-Harris wrote:
> 
> 
>         Is it normal for rsync to take 3 hours on this transfer?
> 
>         Number of files: 27419348
>         Number of files transferred: 19501
>         Total file size: 185.39G bytes
>         Total transferred file size: 195.92M bytes
>         Literal data: 195.68M bytes
>         Matched data: 241.09K bytes
>         File list size: 402.01M
>         File list generation time: 0.561 seconds
>         File list transfer time: 0.000 seconds
>         Total bytes sent: 600.61M
>         Total bytes received: 2.44M
> 
>         sent 600.61M bytes  received 2.44M bytes  54.42K bytes/sec
>         total size is 185.39G  speedup is 307.42
> 
> 
>         A normal transfer of 600 MB for me between these two servers
>         only takes a few seconds.... Is there some way I can speed this up?
> 
> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5uuhYACgkQVKC1jlbQAQcV1wCeLLN+f45ydJre5UHmrdIj77FK
OakAoOMdRTPLvdnSFSEX9407LiWuEMH3
=O2vB
-----END PGP SIGNATURE-----