RFC: slow-down option
satish at cadence.com
Fri Apr 4 11:28:06 MDT 2014
With multiple rsync (I run around 10) streams the bwlimit becomes
complicated w.r.t. optimizing total bandwidth, its no way near close to
what I want to achieve. I want it to be dynamically scale up and down
within maximum threshold depending on overall load. trickle
(http://monkey.org/~marius/pages/?page=trickle) seems to be promising
but I haven't tried it yet.
On 4/4/2014 1:03 PM, Frank Terhaar-Yonkers wrote:
> I needed to back up some of my NAS over the WAN to another (friends)
> NAS (3/4TB-total). A lot of expensive hi-def music files, mostly very
> large. He backs up to mine, vice versa for disaster recovery.
> There were two issues: 1) sucking up all my UL bandwidth, only 5mbit
> in this case, 2) having the option to do a "delayed" kill of the rsync
> when it *finished* the current file rather than an immediate stop so
> as not to waste the bandwidth already used to move a portion of a
> large file.
> I implemented both with signals. A signal toggles the enable/disable
> of the --bwlimit=XXX param so you can do a simple throttle back or run
> wide open. The second signal is a delayed quit - when current file is
> This seemed to work pretty well. During the day when I needed the BW
> for work, etc. I'd throttle it, then let it run wide open at night.
> This could be expanded with a BW list, so instead of a simple toggle,
> the next BW setting in the list could be used, round-robin. The
> delayed quit was nice if new content was added that I felt I wanted
> backed up first. Quit (delayed), then restart, easily scripted.
> Let me know if there is any interest in putting this in the code base
> and I'll create some diffs for review.
> On 4/3/14, 10:47 AM, Marian Marinov wrote:
>> On 04/03/2014 03:35 PM, Christoph Biedl wrote:
>>> Joe wrote...
>>>> This is way beyond my level of expertise, but wouldn't something like
>>>> ionice help with that?
>>> Although I'm not Marian, probably not. The ionice program does a
>>> reasonable good job when it's about prioritizing read operation. The
>>> context makes me guess it's rather about writing.
>> We were using ionice and it did not help. The problem is that if for
>> some reason all of your concurrent rsyncs are running with the same
>> ionice level and class, even thou they are ioniced, effectively there
>> is no change.
>> So at that point we wrote a simple perl daemon that monitored rsyncs
>> and changed ionice levels based on the amount of time each rsync
>> spent in each level.
>> We pushed all new rsyncs onto the lowest level and dynamically moved
>> them to the top. It simply does not work as well as I imagined.
>> The best solution was to use the bklio control-group, BUT, with it
>> the backups were still slower then with the slow-down option.
>>>> Also, check out:
>>>> 2 more pipe utilities
>>>> Viewer & throttle
>>>> Throttle - limits bandwidth of a pipe - for use with network transfers
>>> The pv utility is way to little known and served me well in many
>>> situations, and throttle seems to do quite the same. Both however seem
>>> to do quite the same thing rsync's --bwlimit option does, while the
>>> latter is more sophisticated.
>>> It the issue is the one I have in mind (which is one I constantly
>>> suffer from), the actual problem is the dirty buffer writeback
>>> strategy, deep in the Linux kernel.
> Frank Terhaar-Yonkers W4FTY
> Cisco Systems, Inc.
> 7025 Kit Creek Road PO Box 14987
> Research Triangle Park, North Carolina 27709
> fty at cisco.com voice(919)392-2101
> *JabberCall me*
> browser-based video chat
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rsync