bwlimit=

[Wallace Matthews]

Wayne replied to my original note which said that in a special situation that I was using to probe rsync to build a behavioral model that bwlimit= resulted in bimodal behavior around a 4000 kbyte/sec value.

He responded with a patch that I have tested in a limited way. I have a push scenario from a local site to a remote site. I use a file that is 6.3 Megabyte in size whose checksums (when written to batch) are 60k bytes and whose deltas (when written to batch) are 40k. I touch the local file (but do no modifications to the file other than changing its modification time) and then rsync the file with the command string

"time rsync -ar --rsh=rsh --bwlimit=<x> foo.tar remote://test" where foo.tar is in the local directory /test. I have recorded the "real" , "user", and "sys" values to get an idea of the shape of the curve as I vary <x>. I have values for <x> of 8000, 4000, 2000, 1000, 500, 250, 175, 100, 50, 25, 12, 6, 3, 2, 1. The resulting curve for lower values of <x> is a near linear line. At the upper end the "real" value is a constant. There is a knee to the curve between 500 and 100 for this particular file size.

I have done nothing to validate that the peak transmission rate is accurate. It is not important to my original goal. 

I would like to express my appreciation to Wayne for the patch. It allows me to do my work.

I have tested the patch against a large file (9.2 Gigabytes) that results in a checksum file of 3.5Meg and a delta file of 2.3Meg and used a bwlimit=1 and tested the "real" time against an expected "real" time and got a comparable result. My expected completion time was 96 seconds based upon 5.8 Meg total traffic and the measured time was 114 seconds. There is only 20 seconds unaccounted for, so the bw throttling may not be 100% on the money but I dont think it is off that much. My intuition is that a large part of the 20 seconds is idle time in the communications pipe caused by processing and scheduling delays, etc. 

Wayne's patch is better than the previous implementation. 

Wayne's patch insures that the "average" transmission rate will not exceed the bwlimit and my partial testing seems to validate that. 
It does not enforce that the instantaneous transmission rate will not exceed the bwlimit. 

In my case, I am more interested in what the total network between the local and the remote system will allow so Wayne's patch meets my needs. It may not be exactly accurate but it is close enough for me to create my models.

bwlimit is not main line functionality. It is useful for doing things like I am doing, but how much effort should we put into it?? When rsync is doing what it was designed to do, we need it to run as fast as the network will allow it to run?? Or, is there value in throttling it during normal operation? How accurate does the throttle have to be?

wally

PS - If there is interest in the data I am collecting for my model, I am willing to share that data when I have completed the testing. It will be targeted for my needs (thus incomplete for some others) and it will be based upon the hardware and OS environment of my servers and not necessarily of any value on others.