<span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 13px; background-color: rgb(255, 255, 255); ">Well what if it was FreeBSD on ext3 on a hard disk? What block level tools are available for that setup which can do differential back ups?</span><br>
<br><div class="gmail_quote">On Tue, Sep 13, 2011 at 21:19, Kevin Korb <span dir="ltr"><<a href="mailto:kmk@sanitarium.net">kmk@sanitarium.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
</div>Depends on what block level tool you use. Which will often depend on<br>
what kind of block device, filesystem, and operating system is involved.<br>
<div class="im"><br>
On 09/13/11 02:14, David Somers-Harris wrote:<br>
> A block level tool won't do a differential back up like rsync does will it?<br>
><br>
> On Tue, Sep 13, 2011 at 11:04, Kevin Korb <<a href="mailto:kmk@sanitarium.net">kmk@sanitarium.net</a><br>
</div><div class="im">> <mailto:<a href="mailto:kmk@sanitarium.net">kmk@sanitarium.net</a>>> wrote:<br>
><br>
> The rsync is doing a whole lot more than transferring 600MB of data. It<br>
> is transferring 600MB of differences. It is going through 27.4 million<br>
> files on one end and possibly that many on the other. That is a 402MB<br>
> listing of files plus as many as 54.8 million calls to stat().<br>
><br>
> Rsync is not about minimizing transfer time. It is about minimizing<br>
> network IO at the expense of CPU and disk IO. Rsync reduced your<br>
> network IO from 185.39GB to 600MB only 200 of which was actually<br>
> differences in your data.<br>
><br>
> IOW rsync did its job.<br>
><br>
> The unfortunate truth is that no file based tool will perform anywhere<br>
> near wire speeds when it has millions of files to deal with. If you<br>
> want something wire speed then you should look for a block level tool.<br>
><br>
> On 09/12/11 21:54, David Somers-Harris wrote:<br>
>> No, I don't have checksum based checking enabled.<br>
><br>
>> Sorry for neglecting to mention the options I'm using, here they are:<br>
><br>
>> *rsync -vah --exclude=".*" --delete --stats*<br>
><br>
>> Would anything in there cause it to be unnecessarily slow? It's<br>
> taking 3<br>
>> hours but shouldn't take more than 5 minutes...<br>
><br>
><br>
>> On Tue, Sep 13, 2011 at 00:02, Sandon Van Ness<br>
> <<a href="mailto:sandon@van-ness.com">sandon@van-ness.com</a> <mailto:<a href="mailto:sandon@van-ness.com">sandon@van-ness.com</a>><br>
</div><div class="im">>> <mailto:<a href="mailto:sandon@van-ness.com">sandon@van-ness.com</a> <mailto:<a href="mailto:sandon@van-ness.com">sandon@van-ness.com</a>>>> wrote:<br>
><br>
>> It could definitely be normal if say you have checksum based<br>
>> checking enabled...<br>
><br>
><br>
>> On 09/12/2011 01:56 AM, David Somers-Harris wrote:<br>
><br>
><br>
>> Is it normal for rsync to take 3 hours on this transfer?<br>
><br>
>> Number of files: 27419348<br>
>> Number of files transferred: 19501<br>
>> Total file size: 185.39G bytes<br>
>> Total transferred file size: 195.92M bytes<br>
>> Literal data: 195.68M bytes<br>
>> Matched data: 241.09K bytes<br>
>> File list size: 402.01M<br>
>> File list generation time: 0.561 seconds<br>
>> File list transfer time: 0.000 seconds<br>
>> Total bytes sent: 600.61M<br>
>> Total bytes received: 2.44M<br>
><br>
>> sent 600.61M bytes received 2.44M bytes 54.42K bytes/sec<br>
>> total size is 185.39G speedup is 307.42<br>
><br>
><br>
>> A normal transfer of 600 MB for me between these two servers<br>
>> only takes a few seconds.... Is there some way I can speed<br>
> this up?<br>
><br>
><br>
><br>
><br>
- --<br>
</div><div class="im">Please use reply-all for most replies to avoid omitting the mailing<br>
list.<br>
To unsubscribe or change options:<br>
<a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a><br>
Before posting, read:<br>
<a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br>
</div><<a href="http://www.catb.org/%7Eesr/faqs/smart-questions.html" target="_blank">http://www.catb.org/%7Eesr/faqs/smart-questions.html</a>><br>
<br>
- --<br>
<div class="im">~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<br>
Kevin Korb Phone: (407) 252-6853<br>
Systems Administrator Internet:<br>
FutureQuest, Inc. Kevin@FutureQuest.net (work)<br>
Orlando, Florida <a href="mailto:kmk@sanitarium.net">kmk@sanitarium.net</a> (personal)<br>
Web page: <a href="http://www.sanitarium.net/" target="_blank">http://www.sanitarium.net/</a><br>
PGP public key available on web site.<br>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2.0.17 (GNU/Linux)<br>
Comment: Using GnuPG with Mozilla - <a href="http://enigmail.mozdev.org/" target="_blank">http://enigmail.mozdev.org/</a><br>
<br>
</div>iEYEARECAAYFAk5vSjcACgkQVKC1jlbQAQfy8QCg2KipjXb6vMGc4F090eF12RRC<br>
FGIAnA2BHTdw70o1M61kRGJovqi30plw<br>
=tppS<br>
<div><div></div><div class="h5">-----END PGP SIGNATURE-----<br>
--<br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br>
</div></div></blockquote></div><br>