increasing the write block size for high latency

[Jonathan Aquilina]

One thing you that im not seeing factored in is rpm speed of the drives.
On 26 Jul 2014 15:05, "L. A. Walsh" <rsync at tlinx.org> wrote:

> Adam Edgar wrote:
>
>> It seems the issue is indeed in the ssh layer. scp has the same issue and
>> some work has been done in “fixing” that:
>>
>> http://www.psc.edu/index.php/hpn-ssh
>>
>> From the papers abstract:
>> Status: O
>>
>>         SCP and the underlying SSH2 protocol implementation in OpenSSH is
>> network performance limited by statically defined internal flow control
>> buffers. These buffers often end up acting as a bottleneck for network
>> throughput of SCP, especially on long and high bandwith network links.
>>
>>
> ----
>    It is *A* bottle neck over networks.  look for  extensions to ssh to
> ship unencrypted data streams.
>    There's a patch for this @ http://www.psc.edu/index.php/hpn-ssh.
>
>    However, rsync is dog slow locally as well for exactly the reasons you
> mention.
>
> An extract from another note on this topic (came up on suse list this
> week).
>
> Someone suggested compression for a speed up... I responded to that:
>
>
> On a local copy or local network, that usually slows down transfers.
>
> [**** 1000:1 speed ratio with large vs. small io sizes):]
>
> One might ask why rsync is so slow --
> copying 800G from 1 partition to another via xfsdump/restore takes a bit
> under 2 hours,
> or about 170MB/s, but with rsync, on the same partition with rsync
> transfering
> less than 1/1000th as much (700MB), it took ~70-80 minutes... or about
> 163kB/s.
>
> That's on the same system (local drive -> another local drive)
>
> Transfer speeds depend on many factors.  One of the largest is transfer
> size (how much
> transfered with 1 write /read.
> Transfer 1GB, 1-meg at a time, took 2.08s read, and 1.56s to write (using
> direct io).
>
> Transfer it at 4K: 37.28s, to read, and 43.02s to write.
>
> So 20-40x can be accounted for just on R/W size (1k buffers were 4x
> slower).
>
> Many desktop apps still think 4k is a good "read size"
>
> Over a network, causes drop from 500MB/s down to less than 200KB/s
> (as seen in FF and TB) -- 2500X.
>
> Optimal i/o size on my sys is  between 16M-256M.
>
> So -- to answer your question, MANY things can affect speed, but I'd look
> at the
> R/W size first.
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/
> mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20140726/03e389d0/attachment.html>