time variable throttling rsync traffic
Eric S. Johansson
esj at harvee.org
Thu Sep 17 19:39:05 MDT 2009
On 9/16/2009 11:11 AM, Andrew Gideon wrote:
> On Tue, 15 Sep 2009 22:01:04 +0000, Andrew Gideon wrote:
>> It can also potentially be extended in other directions. For one crazy
>> example, the utility (or some other utility that modifies the first
>> utilities configuration) could listen on a port for messages from -
>> presumably - the receiving server. That would be a way for the
>> receiving server to tell the sender what bandwidth to consume.
> Having given this a little more thought, I see issues. The largest is
> that it requires that the receiving device be able to connect to a port
> on the sending device. In general, I'd expect firewalls to be a problem.
> There's a work around that I think started with the gaming community.
> The sending device needs to send a message to the receiving device which
> causes the sending device's firewall(s) to create an opening through
> which the receiver can then send messages.
> All this is just a work around, avoiding the "real" solution: some way
> for the two intermediary processes - the one invoked via --rsh on the
> client and the one presumably invoked via --rsync-path on the server - to
> communicate over the same SSH stream used by the two rsync processes.
> Hmm...perhaps a middle ground is for the SSH process on the sending side
> to permit the receiving device to "talk back" to the sending side via use
> of SSH's -R port forwarding option?
Lee Winter and I had a conversation last night by phone about this very topic.
Turns out that a fairly small set of changes enables all whole bunch of control.
For example, if you have a single file (one to an rsync process) in which you
place the bandwidth limit of that rsync process and you sample that file on a
regular basis for change, you can modulate the data rate for that given process
by changing the contents of that file.
The next level of control requires an independent demon which allocates
bandwidth on a per process basis according to some set maximum. As rsync
processes come or go, the bandwidth allocation would shift among the remaining
If you allow the control processes to talk to each other and perform "speed
tests" you can allocate according to the link and/or limits on both sides.
If you let an rsync process fill a second say this file with how much is left to
be transferred, you could "finish up" almost on processes at a higher bandwidth
allocation than those that will be running much longer. Yet, seems
counterintuitive but it lets you complete short running tasks faster and let
them get out of the way which may (or may not) provide benefits.
by adding relatively simple signals, bandwidth can be modulated by various
mechanisms of varying complexity ranging from the very simple to the very complex.
More information about the rsync