RFC: slow-down option

[Satish Shukla]

With multiple rsync (I run around 10) streams the bwlimit becomes 
complicated w.r.t. optimizing total bandwidth, its no way near close to 
what I want to achieve. I want it to be dynamically scale up and down 
within maximum threshold depending on overall load. trickle 
(http://monkey.org/~marius/pages/?page=trickle)  seems to be promising 
but I haven't tried it yet.


~satish

On 4/4/2014 1:03 PM, Frank Terhaar-Yonkers wrote:
>
> I needed to back up some of my NAS over the WAN to another (friends) 
> NAS (3/4TB-total).  A lot of expensive hi-def music files, mostly very 
> large.  He backs up to mine, vice versa for disaster recovery.
>
> There were two issues:  1) sucking up all my UL bandwidth, only 5mbit 
> in this case, 2) having the option to do a "delayed" kill of the rsync 
> when it *finished* the current file rather than an immediate stop so 
> as not to waste the bandwidth already used to move a portion of a 
> large file.
>
> I implemented both with signals.  A signal toggles the enable/disable 
> of the --bwlimit=XXX param so you can do a simple throttle back or run 
> wide open.  The second signal is a delayed quit - when current file is 
> finished.
>
> This seemed to work pretty well.  During the day when I needed the BW 
> for work, etc.  I'd throttle it, then let it run wide open at night.  
> This could be expanded with a BW list, so instead of a simple toggle, 
> the next BW setting in the list could be used, round-robin.  The 
> delayed quit was nice if new content was added that I felt I wanted 
> backed up first.  Quit (delayed), then restart, easily scripted.
>
> Let me know if there is any interest in putting this in the code base 
> and I'll create some diffs for review.
>
> Cheers,
> Frank
>
> On 4/3/14, 10:47 AM, Marian Marinov wrote:
>> On 04/03/2014 03:35 PM, Christoph Biedl wrote:
>>> Joe wrote...
>>>
>>>> This is way beyond my level of expertise, but wouldn't something like
>>>> ionice help with that?
>>>
>>> Although I'm not Marian, probably not. The ionice program does a
>>> reasonable good job when it's about prioritizing read operation. The
>>> context makes me guess it's rather about writing.
>>
>> We were using ionice and it did not help. The problem is that if for 
>> some reason all of your concurrent rsyncs are running with the same 
>> ionice level and class, even thou they are ioniced, effectively there 
>> is no change.
>> So at that point we wrote a simple perl daemon that monitored rsyncs 
>> and changed ionice levels based on the amount of time each rsync 
>> spent in each level.
>> We pushed all new rsyncs onto the lowest level and dynamically moved 
>> them to the top. It simply does not work as well as I imagined.
>>
>> The best solution was to use the bklio control-group, BUT, with it 
>> the backups were still slower then with the slow-down option.
>>
>> Marian
>>
>>>
>>>> Also, check out:
>>>>
>>>> 2 more pipe utilities
>>>>
>>>> Viewer & throttle
>>>> http://www.ivarch.com/programs/pv.shtml
>>>>
>>>> Throttle - limits bandwidth of a pipe - for use with network transfers
>>>> http://linux.die.net/man/1/throttle
>>>
>>> The pv utility is way to little known and served me well in many
>>> situations, and throttle seems to do quite the same. Both however seem
>>> to do quite the same thing rsync's --bwlimit option does, while the
>>> latter is more sophisticated.
>>>
>>> It the issue is the one I have in mind (which is one I constantly
>>> suffer from), the actual problem is the dirty buffer writeback
>>> strategy, deep in the Linux kernel.
>>>
>>>      Christoph
>>>
>>
>
> -- 
> ------------------------------------------------------------------------
>   Frank Terhaar-Yonkers         W4FTY
>   Cisco Systems, Inc.
>   7025 Kit Creek Road  PO Box 14987
>   Research Triangle Park,  North Carolina  27709
>   fty at cisco.com    voice(919)392-2101
>
> *JabberCall me* 
> <https://sjc-jabberc-ext.cisco.com/call/83922101@cisco.com?name=RTP%20Phone> 
> 	browser-based video chat
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20140404/d230e5d2/attachment.html>