performance with huge files

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You might get better disk IO performance with --inplace.  It would
probably eliminate those parts where it seems to almost stop.  Read the
warning in the man page about it first though.

You can probably go beyond 16MB/sec if you configure ssh to use a weaker
cipher.  Something like rsync -e 'ssh -c arcfour'

If you are using compression with ssh (-C) or with rsync (-z or
- --compress) don't do that on a fast LAN.  It just wastes CPU.

If none of that helps tell us your command line.

On 09/30/11 11:59, daniel wilson wrote:
> Hi Kevin,
> 
> it's ssh between two LAN-connected hosts.  some examples might help -
> 
> 
> 1) scp of the 500GB file might run at 16MB/s
> 
> 2) rsync initially will also transfer the file at about 16MB/s
> 
> 3) subsequent rsyncs after minor changes to the take about the same
> amount of time as initial transfer, but with very heavy CPU.  The file
> transfer shown when using --progress will vary from ~40MB/s to 500kB/s
> wildly.
> 
> So I'm guessing the checksumming of the file for changes is the
> processing that's going on in 3), but I was under the impression this
> might be improved by changing the block size, or hashing algorithm or
> some other factor ?
> 
> Thanks,
> 
> Dan
> 
> 
> 
> On 30 September 2011 14:44, Kevin Korb <kmk at sanitarium.net> wrote:
> What transport are you using?  ssh, rsyncd, local copy, or something else?
> 
> On 09/30/11 09:41, daniel wilson wrote:
>>>> Hi all,
>>>>
>>>> I'm working with some very large files, 500GB or thereabouts, using
>>>> rsync 3.0.7.   I'm not familiar with the internal operation of the
>>>> rsync protocol, but I've read a few old posts that suggest there may
>>>> be ways to improve performance.
>>>>
>>>> Can anyone give me pointers to config changes or patches that might
>>>> help reduce the CPU burden when working with such large files ?
>>>>
>>>> Thanks in advance,
>>>>
>>>> Dan
> 
- --
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6F6CsACgkQVKC1jlbQAQc4vACfXt4WmIW6BPUz7p8b7u5Vl/MJ
WxUAmwZV4QiTr7uST5j6xumrwkdlCDdr
=YbIq
-----END PGP SIGNATURE-----