<div dir="ltr"><div><div><div>The biggest problem we've found with modern day gateways is that they are dramatically overbuffered and this messes with tcp's congestion avoidance mode, a lot.<br><br></div><div>Innumerable resources on "bufferbloat" are on <a href="http://bufferbloat.net">bufferbloat.net</a> to this effect:<br>
<br><a href="https://www.bufferbloat.net/projects/bloat/wiki/TechnicalIntro">https://www.bufferbloat.net/projects/bloat/wiki/TechnicalIntro</a><br><br>and for various rants and data:<br><br><a href="http://gettys.wordpress.com/">http://gettys.wordpress.com/</a><br>
<br>You can mitigate your problem with an fq_codel based shaper on the gateway with something like CeroWrt's "SQM" system, (or openwrt's or dd-wrts, etc)<br><br></div><a href="http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310">http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_SQM_for_CeroWrt_310</a><br>
<br>and further make rsync invisible by marking it with the diffserv marking of CS1 in that system.<br><br></div>I submitted patches long ago to add classification and support for other tcps' congestion control algorithms like lp and westwood to rsync's patch db.<br>
<br></div>Info on codel and fq_codel:<br><br><a href="https://datatracker.ietf.org/doc/draft-nichols-tsvwg-codel/">https://datatracker.ietf.org/doc/draft-nichols-tsvwg-codel/</a><br><a href="https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00">https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00</a><br>
<div><div><div><br><br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Apr 4, 2014 at 10:28 AM, Satish Shukla <span dir="ltr"><<a href="mailto:satish@cadence.com" target="_blank">satish@cadence.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>With multiple rsync (I run around 10)
streams the bwlimit becomes complicated w.r.t. optimizing total
bandwidth, its no way near close to what I want to achieve. I want
it to be dynamically scale up and down within maximum threshold
depending on overall load. trickle
(<a href="http://monkey.org/~marius/pages/?page=trickle" target="_blank">http://monkey.org/~marius/pages/?page=trickle</a>) seems to be
promising but I haven't tried it yet.<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
~satish</font></span><div><div class="h5"><br>
<br>
On 4/4/2014 1:03 PM, Frank Terhaar-Yonkers wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<br>
I needed to back up some of my NAS over the WAN to another
(friends) NAS (3/4TB-total). A lot of expensive hi-def music
files, mostly very large. He backs up to mine, vice versa for
disaster recovery.<br>
<br>
There were two issues: 1) sucking up all my UL bandwidth, only
5mbit in this case, 2) having the option to do a "delayed" kill of
the rsync when it *finished* the current file rather than an
immediate stop so as not to waste the bandwidth already used to
move a portion of a large file.<br>
<br>
I implemented both with signals. A signal toggles the
enable/disable of the --bwlimit=XXX param so you can do a simple
throttle back or run wide open. The second signal is a delayed
quit - when current file is finished.<br>
<br>
This seemed to work pretty well. During the day when I needed the
BW for work, etc. I'd throttle it, then let it run wide open at
night. This could be expanded with a BW list, so instead of a
simple toggle, the next BW setting in the list could be used,
round-robin. The delayed quit was nice if new content was added
that I felt I wanted backed up first. Quit (delayed), then
restart, easily scripted.<br>
<br>
Let me know if there is any interest in putting this in the code
base and I'll create some diffs for review.<br>
<br>
Cheers,<br>
Frank<br>
<br>
<div>On 4/3/14, 10:47 AM, Marian Marinov
wrote:<br>
</div>
<blockquote type="cite">On
04/03/2014 03:35 PM, Christoph Biedl wrote: <br>
<blockquote type="cite">Joe wrote... <br>
<br>
<blockquote type="cite">This is way beyond my level of
expertise, but wouldn't something like <br>
ionice help with that? <br>
</blockquote>
<br>
Although I'm not Marian, probably not. The ionice program does
a <br>
reasonable good job when it's about prioritizing read
operation. The <br>
context makes me guess it's rather about writing. <br>
</blockquote>
<br>
We were using ionice and it did not help. The problem is that if
for some reason all of your concurrent rsyncs are running with
the same ionice level and class, even thou they are ioniced,
effectively there is no change. <br>
So at that point we wrote a simple perl daemon that monitored
rsyncs and changed ionice levels based on the amount of time
each rsync spent in each level. <br>
We pushed all new rsyncs onto the lowest level and dynamically
moved them to the top. It simply does not work as well as I
imagined. <br>
<br>
The best solution was to use the bklio control-group, BUT, with
it the backups were still slower then with the slow-down option.
<br>
<br>
Marian <br>
<br>
<blockquote type="cite"> <br>
<blockquote type="cite">Also, check out: <br>
<br>
2 more pipe utilities <br>
<br>
Viewer & throttle <br>
<a href="http://www.ivarch.com/programs/pv.shtml" target="_blank">http://www.ivarch.com/programs/pv.shtml</a>
<br>
<br>
Throttle - limits bandwidth of a pipe - for use with network
transfers <br>
<a href="http://linux.die.net/man/1/throttle" target="_blank">http://linux.die.net/man/1/throttle</a>
<br>
</blockquote>
<br>
The pv utility is way to little known and served me well in
many <br>
situations, and throttle seems to do quite the same. Both
however seem <br>
to do quite the same thing rsync's --bwlimit option does,
while the <br>
latter is more sophisticated. <br>
<br>
It the issue is the one I have in mind (which is one I
constantly <br>
suffer from), the actual problem is the dirty buffer writeback
<br>
strategy, deep in the Linux kernel. <br>
<br>
Christoph <br>
<br>
</blockquote>
<br>
</blockquote>
<br>
<div>-- <br>
<hr>
<pre> Frank Terhaar-Yonkers W4FTY
Cisco Systems, Inc.
7025 Kit Creek Road PO Box 14987
Research Triangle Park, North Carolina 27709
<a href="mailto:fty@cisco.com" target="_blank">fty@cisco.com</a> voice<a href="tel:%28919%29392-2101" value="+19193922101" target="_blank">(919)392-2101</a>
</pre>
<br>
<table>
<tbody>
<tr>
<td style="background-color:#888888;font-family:Calibri,Sans-Serif;font-size:11pt;width:95px;height:18px;text-align:center"> <a href="https://sjc-jabberc-ext.cisco.com/call/83922101@cisco.com?name=RTP%20Phone" style="text-decoration:none;color:white" target="_blank"> <b>JabberCall me</b> </a> </td>
<td style="color:gray;font-size:9pt;font-family:Calibri,Sans-Serif;font-style:italic;height:18px;width:375px">browser-based video chat</td>
</tr>
</tbody>
</table>
</div>
</blockquote>
<br>
</div></div></div>
<br>--<br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br></blockquote></div><br><br clear="all"><br>-- <br>Dave Täht<br>
<br>NSFW: <a href="https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article" target="_blank">https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article</a>
</div>