Duration
Kevin Korb
kmk at sanitarium.net
Tue Sep 13 06:19:03 MDT 2011
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Depends on what block level tool you use. Which will often depend on
what kind of block device, filesystem, and operating system is involved.
On 09/13/11 02:14, David Somers-Harris wrote:
> A block level tool won't do a differential back up like rsync does will it?
>
> On Tue, Sep 13, 2011 at 11:04, Kevin Korb <kmk at sanitarium.net
> <mailto:kmk at sanitarium.net>> wrote:
>
> The rsync is doing a whole lot more than transferring 600MB of data. It
> is transferring 600MB of differences. It is going through 27.4 million
> files on one end and possibly that many on the other. That is a 402MB
> listing of files plus as many as 54.8 million calls to stat().
>
> Rsync is not about minimizing transfer time. It is about minimizing
> network IO at the expense of CPU and disk IO. Rsync reduced your
> network IO from 185.39GB to 600MB only 200 of which was actually
> differences in your data.
>
> IOW rsync did its job.
>
> The unfortunate truth is that no file based tool will perform anywhere
> near wire speeds when it has millions of files to deal with. If you
> want something wire speed then you should look for a block level tool.
>
> On 09/12/11 21:54, David Somers-Harris wrote:
>> No, I don't have checksum based checking enabled.
>
>> Sorry for neglecting to mention the options I'm using, here they are:
>
>> *rsync -vah --exclude=".*" --delete --stats*
>
>> Would anything in there cause it to be unnecessarily slow? It's
> taking 3
>> hours but shouldn't take more than 5 minutes...
>
>
>> On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness
> <sandon at van-ness.com <mailto:sandon at van-ness.com>
>> <mailto:sandon at van-ness.com <mailto:sandon at van-ness.com>>> wrote:
>
>> It could definitely be normal if say you have checksum based
>> checking enabled...
>
>
>> On 09/12/2011 01:56 AM, David Somers-Harris wrote:
>
>
>> Is it normal for rsync to take 3 hours on this transfer?
>
>> Number of files: 27419348
>> Number of files transferred: 19501
>> Total file size: 185.39G bytes
>> Total transferred file size: 195.92M bytes
>> Literal data: 195.68M bytes
>> Matched data: 241.09K bytes
>> File list size: 402.01M
>> File list generation time: 0.561 seconds
>> File list transfer time: 0.000 seconds
>> Total bytes sent: 600.61M
>> Total bytes received: 2.44M
>
>> sent 600.61M bytes received 2.44M bytes 54.42K bytes/sec
>> total size is 185.39G speedup is 307.42
>
>
>> A normal transfer of 600 MB for me between these two servers
>> only takes a few seconds.... Is there some way I can speed
> this up?
>
>
>
>
- --
Please use reply-all for most replies to avoid omitting the mailing
list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
<http://www.catb.org/%7Eesr/faqs/smart-questions.html>
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk5vSjcACgkQVKC1jlbQAQfy8QCg2KipjXb6vMGc4F090eF12RRC
FGIAnA2BHTdw70o1M61kRGJovqi30plw
=tppS
-----END PGP SIGNATURE-----
More information about the rsync
mailing list