performance with huge files
Kevin Korb
kmk at sanitarium.net
Fri Sep 30 10:02:51 MDT 2011
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
You might get better disk IO performance with --inplace. It would
probably eliminate those parts where it seems to almost stop. Read the
warning in the man page about it first though.
You can probably go beyond 16MB/sec if you configure ssh to use a weaker
cipher. Something like rsync -e 'ssh -c arcfour'
If you are using compression with ssh (-C) or with rsync (-z or
- --compress) don't do that on a fast LAN. It just wastes CPU.
If none of that helps tell us your command line.
On 09/30/11 11:59, daniel wilson wrote:
> Hi Kevin,
>
> it's ssh between two LAN-connected hosts. some examples might help -
>
>
> 1) scp of the 500GB file might run at 16MB/s
>
> 2) rsync initially will also transfer the file at about 16MB/s
>
> 3) subsequent rsyncs after minor changes to the take about the same
> amount of time as initial transfer, but with very heavy CPU. The file
> transfer shown when using --progress will vary from ~40MB/s to 500kB/s
> wildly.
>
> So I'm guessing the checksumming of the file for changes is the
> processing that's going on in 3), but I was under the impression this
> might be improved by changing the block size, or hashing algorithm or
> some other factor ?
>
> Thanks,
>
> Dan
>
>
>
> On 30 September 2011 14:44, Kevin Korb <kmk at sanitarium.net> wrote:
> What transport are you using? ssh, rsyncd, local copy, or something else?
>
> On 09/30/11 09:41, daniel wilson wrote:
>>>> Hi all,
>>>>
>>>> I'm working with some very large files, 500GB or thereabouts, using
>>>> rsync 3.0.7. I'm not familiar with the internal operation of the
>>>> rsync protocol, but I've read a few old posts that suggest there may
>>>> be ways to improve performance.
>>>>
>>>> Can anyone give me pointers to config changes or patches that might
>>>> help reduce the CPU burden when working with such large files ?
>>>>
>>>> Thanks in advance,
>>>>
>>>> Dan
>
- --
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk6F6CsACgkQVKC1jlbQAQc4vACfXt4WmIW6BPUz7p8b7u5Vl/MJ
WxUAmwZV4QiTr7uST5j6xumrwkdlCDdr
=YbIq
-----END PGP SIGNATURE-----
More information about the rsync
mailing list