increasing the write block size for high latency
eagles051387 at gmail.com
Sat Jul 26 07:47:52 MDT 2014
One thing you that im not seeing factored in is rpm speed of the drives.
On 26 Jul 2014 15:05, "L. A. Walsh" <rsync at tlinx.org> wrote:
> Adam Edgar wrote:
>> It seems the issue is indeed in the ssh layer. scp has the same issue and
>> some work has been done in “fixing” that:
>> From the papers abstract:
>> Status: O
>> SCP and the underlying SSH2 protocol implementation in OpenSSH is
>> network performance limited by statically defined internal flow control
>> buffers. These buffers often end up acting as a bottleneck for network
>> throughput of SCP, especially on long and high bandwith network links.
> It is *A* bottle neck over networks. look for extensions to ssh to
> ship unencrypted data streams.
> There's a patch for this @ http://www.psc.edu/index.php/hpn-ssh.
> However, rsync is dog slow locally as well for exactly the reasons you
> An extract from another note on this topic (came up on suse list this
> Someone suggested compression for a speed up... I responded to that:
> On a local copy or local network, that usually slows down transfers.
> [**** 1000:1 speed ratio with large vs. small io sizes):]
> One might ask why rsync is so slow --
> copying 800G from 1 partition to another via xfsdump/restore takes a bit
> under 2 hours,
> or about 170MB/s, but with rsync, on the same partition with rsync
> less than 1/1000th as much (700MB), it took ~70-80 minutes... or about
> That's on the same system (local drive -> another local drive)
> Transfer speeds depend on many factors. One of the largest is transfer
> size (how much
> transfered with 1 write /read.
> Transfer 1GB, 1-meg at a time, took 2.08s read, and 1.56s to write (using
> direct io).
> Transfer it at 4K: 37.28s, to read, and 43.02s to write.
> So 20-40x can be accounted for just on R/W size (1k buffers were 4x
> Many desktop apps still think 4k is a good "read size"
> Over a network, causes drop from 500MB/s down to less than 200KB/s
> (as seen in FF and TB) -- 2500X.
> Optimal i/o size on my sys is between 16M-256M.
> So -- to answer your question, MANY things can affect speed, but I'd look
> at the
> R/W size first.
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rsync