User controlled i/o block size?

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You didn't say if you were networking or what features of rsync you
are using but if you aren't networking and aren't doing anything fancy
you are probably better off with cp -au which is essentially the same
as rsync -au except faster.

Anyways, smaller reads and writes are usually better handled by the
OS's caches than really big ones.

On 04/11/2016 07:00 PM, Greg Freemyer wrote:
> All,
> 
> One big thing I failed to mention is I was running rsync inside a 
> cygwin windows 8.1 setup.
> 
> I moved it to a linux box and the behavior is much better.  I get
> a nice smooth 85-90 MB/sec.  That might be the max speed of the
> source drive.
> 
> I'd still like a way to improve rsync's performance in cygwin, but
> I can understand it is a low priority.
> 
> Thanks Greg -- Greg Freemyer www.IntelligentAvatar.net
> 
> 
> On Mon, Apr 11, 2016 at 4:08 PM, Greg Freemyer
> <greg.freemyer at gmail.com> wrote:
>> I hope this isn't a FAQ.
>> 
>> Per the man page I see ways to control the blocksize for hash 
>> comparison reasons, but no way to control it for i/o performance 
>> reasons.
>> 
>> I'm using rsync to copy folder trees full of large files and I'd
>> like to have control of how much data is read / written at a
>> time.  Maybe read 10 MB, write 10 MB, etc.
>> 
>> Is there an existing way to do that?
>> 
>> == details ==
>> 
>> When copying a bunch of 1.5 GB files with rsync, I'm only seeing
>> 50% of the throughput I hope to see.
>> 
>> I haven't looked at the code, or even run strace, but it seems
>> like the code is doing something like:
>> 
>> while (files)  { read 1.5 GB file to ram write 1.5 GB file from
>> ram fsync()  ensure 1.5 GB file is on disk } endwhile
>> 
>> I say that because I see several seconds of high-speed reading,
>> then no reads.
>> 
>> When the reads stop, I see writes kick in, then they stop and
>> reads start up again.
>> 
>> The end result is I'm only using 50% of the available bandwidth.
>> 
>> Not that I'm copying my source folder tree to a newly created
>> folder tree, so there is not any reading of the destination
>> needed. My ultimate would be something like:
>> 
>> while (files) { while (data_in_file) { read
>> user_defined_blocksize to ram from file write
>> user_defined_blocksize from ram to file } fsync()  ensure 1.5 GB
>> file is on disk } endwhile
>> 
>> Thanks Greg -- Greg Freemyer www.IntelligentAvatar.net
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlcMLaIACgkQVKC1jlbQAQcRfACgkhNohkizfd1zm502bXjX0cN9
BnwAn1sMsWRg3er1aiynU4koDEYEiI91
=/1Va
-----END PGP SIGNATURE-----