Duration

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Depends on what block level tool you use.  Which will often depend on
what kind of block device, filesystem, and operating system is involved.

On 09/13/11 02:14, David Somers-Harris wrote:
> A block level tool won't do a differential back up like rsync does will it?
> 
> On Tue, Sep 13, 2011 at 11:04, Kevin Korb <kmk at sanitarium.net
> <mailto:kmk at sanitarium.net>> wrote:
> 
> The rsync is doing a whole lot more than transferring 600MB of data.  It
> is transferring 600MB of differences.  It is going through 27.4 million
> files on one end and possibly that many on the other.  That is a 402MB
> listing of files plus as many as 54.8 million calls to stat().
> 
> Rsync is not about minimizing transfer time.  It is about minimizing
> network IO at the expense of CPU and disk IO.  Rsync reduced your
> network IO from 185.39GB to 600MB only 200 of which was actually
> differences in your data.
> 
> IOW rsync did its job.
> 
> The unfortunate truth is that no file based tool will perform anywhere
> near wire speeds when it has millions of files to deal with.  If you
> want something wire speed then you should look for a block level tool.
> 
> On 09/12/11 21:54, David Somers-Harris wrote:
>> No, I don't have checksum based checking enabled.
> 
>> Sorry for neglecting to mention the options I'm using, here they are:
> 
>> *rsync -vah --exclude=".*" --delete --stats*
> 
>> Would anything in there cause it to be unnecessarily slow? It's
> taking 3
>> hours but shouldn't take more than 5 minutes...
> 
> 
>> On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness
> <sandon at van-ness.com <mailto:sandon at van-ness.com>
>> <mailto:sandon at van-ness.com <mailto:sandon at van-ness.com>>> wrote:
> 
>>     It could definitely be normal if say you have checksum based
>>     checking enabled...
> 
> 
>>     On 09/12/2011 01:56 AM, David Somers-Harris wrote:
> 
> 
>>         Is it normal for rsync to take 3 hours on this transfer?
> 
>>         Number of files: 27419348
>>         Number of files transferred: 19501
>>         Total file size: 185.39G bytes
>>         Total transferred file size: 195.92M bytes
>>         Literal data: 195.68M bytes
>>         Matched data: 241.09K bytes
>>         File list size: 402.01M
>>         File list generation time: 0.561 seconds
>>         File list transfer time: 0.000 seconds
>>         Total bytes sent: 600.61M
>>         Total bytes received: 2.44M
> 
>>         sent 600.61M bytes  received 2.44M bytes  54.42K bytes/sec
>>         total size is 185.39G  speedup is 307.42
> 
> 
>>         A normal transfer of 600 MB for me between these two servers
>>         only takes a few seconds.... Is there some way I can speed
> this up?
> 
> 
> 
> 
- --
Please use reply-all for most replies to avoid omitting the mailing
list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
<http://www.catb.org/%7Eesr/faqs/smart-questions.html>

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5vSjcACgkQVKC1jlbQAQfy8QCg2KipjXb6vMGc4F090eF12RRC
FGIAnA2BHTdw70o1M61kRGJovqi30plw
=tppS
-----END PGP SIGNATURE-----