performance with huge files

[daniel wilson]

Hi Kevin,

it's ssh between two LAN-connected hosts.  some examples might help -


1) scp of the 500GB file might run at 16MB/s

2) rsync initially will also transfer the file at about 16MB/s

3) subsequent rsyncs after minor changes to the take about the same
amount of time as initial transfer, but with very heavy CPU.  The file
transfer shown when using --progress will vary from ~40MB/s to 500kB/s
wildly.

So I'm guessing the checksumming of the file for changes is the
processing that's going on in 3), but I was under the impression this
might be improved by changing the block size, or hashing algorithm or
some other factor ?

Thanks,

Dan



On 30 September 2011 14:44, Kevin Korb <kmk at sanitarium.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> What transport are you using?  ssh, rsyncd, local copy, or something else?
>
> On 09/30/11 09:41, daniel wilson wrote:
>> Hi all,
>>
>> I'm working with some very large files, 500GB or thereabouts, using
>> rsync 3.0.7.   I'm not familiar with the internal operation of the
>> rsync protocol, but I've read a few old posts that suggest there may
>> be ways to improve performance.
>>
>> Can anyone give me pointers to config changes or patches that might
>> help reduce the CPU burden when working with such large files ?
>>
>> Thanks in advance,
>>
>> Dan
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>        Kevin Korb                      Phone:    (407) 252-6853
>        Systems Administrator           Internet:
>        FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
>        Orlando, Florida                kmk at sanitarium.net (personal)
>        Web page:                       http://www.sanitarium.net/
>        PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk6Fx7gACgkQVKC1jlbQAQdzwQCg2I0G+dG3zw+JPiTb9NWgA0nr
> bj8AoL+ErywMynAoKpFNWg/X1UsBezLE
> =/wNa
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>