Duration

[David Somers-Harris]

A block level tool won't do a differential back up like rsync does will it?

On Tue, Sep 13, 2011 at 11:04, Kevin Korb <kmk at sanitarium.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> The rsync is doing a whole lot more than transferring 600MB of data.  It
> is transferring 600MB of differences.  It is going through 27.4 million
> files on one end and possibly that many on the other.  That is a 402MB
> listing of files plus as many as 54.8 million calls to stat().
>
> Rsync is not about minimizing transfer time.  It is about minimizing
> network IO at the expense of CPU and disk IO.  Rsync reduced your
> network IO from 185.39GB to 600MB only 200 of which was actually
> differences in your data.
>
> IOW rsync did its job.
>
> The unfortunate truth is that no file based tool will perform anywhere
> near wire speeds when it has millions of files to deal with.  If you
> want something wire speed then you should look for a block level tool.
>
> On 09/12/11 21:54, David Somers-Harris wrote:
> > No, I don't have checksum based checking enabled.
> >
> > Sorry for neglecting to mention the options I'm using, here they are:
> >
> > *rsync -vah --exclude=".*" --delete --stats*
> >
> > Would anything in there cause it to be unnecessarily slow? It's taking 3
> > hours but shouldn't take more than 5 minutes...
> >
> >
> > On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness <sandon at van-ness.com
> > <mailto:sandon at van-ness.com>> wrote:
> >
> >     It could definitely be normal if say you have checksum based
> >     checking enabled...
> >
> >
> >     On 09/12/2011 01:56 AM, David Somers-Harris wrote:
> >
> >
> >         Is it normal for rsync to take 3 hours on this transfer?
> >
> >         Number of files: 27419348
> >         Number of files transferred: 19501
> >         Total file size: 185.39G bytes
> >         Total transferred file size: 195.92M bytes
> >         Literal data: 195.68M bytes
> >         Matched data: 241.09K bytes
> >         File list size: 402.01M
> >         File list generation time: 0.561 seconds
> >         File list transfer time: 0.000 seconds
> >         Total bytes sent: 600.61M
> >         Total bytes received: 2.44M
> >
> >         sent 600.61M bytes  received 2.44M bytes  54.42K bytes/sec
> >         total size is 185.39G  speedup is 307.42
> >
> >
> >         A normal transfer of 600 MB for me between these two servers
> >         only takes a few seconds.... Is there some way I can speed this
> up?
> >
> >
> >
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>        Kevin Korb                      Phone:    (407) 252-6853
>        Systems Administrator           Internet:
>        FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
>        Orlando, Florida                kmk at sanitarium.net (personal)
>        Web page:                       http://www.sanitarium.net/
>        PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk5uuhYACgkQVKC1jlbQAQcV1wCeLLN+f45ydJre5UHmrdIj77FK
> OakAoOMdRTPLvdnSFSEX9407LiWuEMH3
> =O2vB
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20110913/d8601e58/attachment.html>