Duration

[David Somers-Harris]

Well what if it was FreeBSD on ext3 on a hard disk? What block level tools
are available for that setup which can do differential back ups?

On Tue, Sep 13, 2011 at 21:19, Kevin Korb <kmk at sanitarium.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Depends on what block level tool you use.  Which will often depend on
> what kind of block device, filesystem, and operating system is involved.
>
> On 09/13/11 02:14, David Somers-Harris wrote:
> > A block level tool won't do a differential back up like rsync does will
> it?
> >
> > On Tue, Sep 13, 2011 at 11:04, Kevin Korb <kmk at sanitarium.net
> > <mailto:kmk at sanitarium.net>> wrote:
> >
> > The rsync is doing a whole lot more than transferring 600MB of data.  It
> > is transferring 600MB of differences.  It is going through 27.4 million
> > files on one end and possibly that many on the other.  That is a 402MB
> > listing of files plus as many as 54.8 million calls to stat().
> >
> > Rsync is not about minimizing transfer time.  It is about minimizing
> > network IO at the expense of CPU and disk IO.  Rsync reduced your
> > network IO from 185.39GB to 600MB only 200 of which was actually
> > differences in your data.
> >
> > IOW rsync did its job.
> >
> > The unfortunate truth is that no file based tool will perform anywhere
> > near wire speeds when it has millions of files to deal with.  If you
> > want something wire speed then you should look for a block level tool.
> >
> > On 09/12/11 21:54, David Somers-Harris wrote:
> >> No, I don't have checksum based checking enabled.
> >
> >> Sorry for neglecting to mention the options I'm using, here they are:
> >
> >> *rsync -vah --exclude=".*" --delete --stats*
> >
> >> Would anything in there cause it to be unnecessarily slow? It's
> > taking 3
> >> hours but shouldn't take more than 5 minutes...
> >
> >
> >> On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness
> > <sandon at van-ness.com <mailto:sandon at van-ness.com>
> >> <mailto:sandon at van-ness.com <mailto:sandon at van-ness.com>>> wrote:
> >
> >>     It could definitely be normal if say you have checksum based
> >>     checking enabled...
> >
> >
> >>     On 09/12/2011 01:56 AM, David Somers-Harris wrote:
> >
> >
> >>         Is it normal for rsync to take 3 hours on this transfer?
> >
> >>         Number of files: 27419348
> >>         Number of files transferred: 19501
> >>         Total file size: 185.39G bytes
> >>         Total transferred file size: 195.92M bytes
> >>         Literal data: 195.68M bytes
> >>         Matched data: 241.09K bytes
> >>         File list size: 402.01M
> >>         File list generation time: 0.561 seconds
> >>         File list transfer time: 0.000 seconds
> >>         Total bytes sent: 600.61M
> >>         Total bytes received: 2.44M
> >
> >>         sent 600.61M bytes  received 2.44M bytes  54.42K bytes/sec
> >>         total size is 185.39G  speedup is 307.42
> >
> >
> >>         A normal transfer of 600 MB for me between these two servers
> >>         only takes a few seconds.... Is there some way I can speed
> > this up?
> >
> >
> >
> >
> - --
> Please use reply-all for most replies to avoid omitting the mailing
> list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read:
> http://www.catb.org/~esr/faqs/smart-questions.html
> <http://www.catb.org/%7Eesr/faqs/smart-questions.html>
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>        Kevin Korb                      Phone:    (407) 252-6853
>        Systems Administrator           Internet:
>        FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
>        Orlando, Florida                kmk at sanitarium.net (personal)
>        Web page:                       http://www.sanitarium.net/
>        PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk5vSjcACgkQVKC1jlbQAQfy8QCg2KipjXb6vMGc4F090eF12RRC
> FGIAnA2BHTdw70o1M61kRGJovqi30plw
> =tppS
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20110914/cbd9ad87/attachment.html>