User controlled i/o block size?

[Greg Freemyer]

All,

One big thing I failed to mention is I was running rsync inside a
cygwin windows 8.1 setup.

I moved it to a linux box and the behavior is much better.  I get a
nice smooth 85-90 MB/sec.  That might be the max speed of the source
drive.

I'd still like a way to improve rsync's performance in cygwin, but I
can understand it is a low priority.

Thanks
Greg
--
Greg Freemyer
www.IntelligentAvatar.net


On Mon, Apr 11, 2016 at 4:08 PM, Greg Freemyer <greg.freemyer at gmail.com> wrote:
> I hope this isn't a FAQ.
>
> Per the man page I see ways to control the blocksize for hash
> comparison reasons, but no way to control it for i/o performance
> reasons.
>
> I'm using rsync to copy folder trees full of large files and I'd like
> to have control of how much data is read / written at a time.  Maybe
> read 10 MB, write 10 MB, etc.
>
> Is there an existing way to do that?
>
> == details ==
>
> When copying a bunch of 1.5 GB files with rsync, I'm only seeing 50%
> of the throughput I hope to see.
>
> I haven't looked at the code, or even run strace, but it seems like
> the code is doing something like:
>
> while (files)  {
>     read 1.5 GB file to ram
>     write 1.5 GB file from ram
>     fsync()  ensure 1.5 GB file is on disk
> } endwhile
>
> I say that because I see several seconds of high-speed reading, then no reads.
>
> When the reads stop, I see writes kick in, then they stop and reads
> start up again.
>
> The end result is I'm only using 50% of the available bandwidth.
>
> Not that I'm copying my source folder tree to a newly created folder
> tree, so there is not any reading of the destination needed.
> My ultimate would be something like:
>
> while (files) {
>     while (data_in_file) {
>         read user_defined_blocksize to ram from file
>         write user_defined_blocksize from ram to file
>     }
>     fsync()  ensure 1.5 GB file is on disk
> } endwhile
>
> Thanks
> Greg
> --
> Greg Freemyer
> www.IntelligentAvatar.net