time variable throttling rsync traffic
Eric S. Johansson
esj at harvee.org
Tue Sep 15 10:11:03 MDT 2009
On 9/14/2009 3:55 PM, Andrew Gideon wrote:
>> If there is one or more bottleneck link in the network (places where
>> traffic feeds from one or more links with aggregate larger capacity
>> into a link with smaller capacity) then it makes sense that this work
>> has to be done on the router that is on the sending side of the
>> bottleneck link(s), not on the receiving side.
> I've been assuming that this isn't possible for a couple of reasons.
> Since this is a remote backup application, I'm assuming that senders are
> "all over". That is, there is no one "sending router"; there are
> instead many sending routers. The only routers common to all these data
> streams would be those close to the receiving server.
> Next, most random users have no control over their routing
In theory, you could have graph of rsync traffic between multiple hosts. but
for right now, let's just consider two hosts. The total bandwidth available
between any two machines is the lesser of the two bwlimits. It would need to be
negotiated periodically as bandwidth limits change over time. For example, at 8
a.m. local time, my bandwidth limit my drop to 100 kbytes a second whereas the
other end may still be available to run full tilt. I need to drop that link at
8 a.m. without necessarily restarting the transfer.
It's not a very good model in that you can't take into account and with
limitations at various chokepoints but, it's better than nothing.
When you expand it to a graph network, you'll need some form of a rendezvous
point and allocation scheme to allocate bandwidth to all the others. It would
get messy because a client to or three hops away could influence the bandwidth
allocated to a local client.
Unfortunately, I have an idea for experimentation. Tell me how gross is this?
run rsync till a given time deadline, killing off the original program instance
and then restart with a new bandwidth limit. I would probably use a small
program invoking rsync and then sending a signal when "it's Time" then starting
a new rsync run.
This could only be client-side as server-side couldn't trigger proper restarts I
don't think. I'd need to look into that. My main question would be is there
any way to tell if rsync has been shut down abruptly by external signal or has
terminated on its own (with or without errors) ?
I know, this is probably ugly as all get out but, it gives me a chance to
More information about the rsync