Rsync Optimization for Long Haul Links

[Lawrence D. Dunn]

David,
   What OS is on each end?
   This sounds like (maybe) classic "TCP buffers too small by default",
   which limits the amount of data in-flight for an individual
   TCP flow to min(senders_TCP_send_buf, receivers_TCP_recv_buf),
   assuming a clean network path.

   (If you're running current Linux, it should autotune those buffers
   up to a max-configured-value. But many out-of-box max's are too 
small, anyway).

   In the old days, rsync clients had no way to ask for a specific TCP 
buffer size.
   So you were stuck with whatever is default for the OS.

   Thankfully, rsync developers (Wayne) added a CLI option awhile ago, like:
  rsync -av --sockopts=SO_SNDBUF=2000000,SO_RCVBUF=2000000 host::module

   This requests ~2MB for send and receive buffer from the OS.
   (If the OS has a smaller upper bound, it might silently not give you
   what you ask for).
   Need to do this sort of ask-for-large-buffers on both ends,
   whether client or server/daemon.

   If working properly, and if the OS is configured to allow what you
   ask for, and if the network is reasonably clean (few losses),
   and the on-path routers have reasonable (bw*delay) buffers
   to help when there's an occasional loss,
   you should be able to get very high single-flow rsync performance.

   It's worked for me, anyway.

   Not sure of the impact of in-line h/w-VPN's...

   Let me know if I can help (perhaps unicast?).
   Once it's resolved, it's be nice to post a note back to the
   list on what the resolution is.
   But it sounds suspiciously like TCP-buffer-limited, from what 
you've said so far...
   (although CA->germany 440/500 doesn't sound bad...)

   Another diagnostic approach would be to verify that you can
   get good single-flow performance with iperf (specifying reasonable buffers)
   or similar.
   If you can't get good performance with iperf, then you won't
   be able to with single-flow rsync either...
   If iperf is tripping over some loss-rate on the path, then you *could*
   look at more aggressive TCP stacks - but let's save that for later.

Larry
--

At 5:01 PM -0700 8/10/06, David Halliday wrote:
>Rsync speed seems to be effected more by latency than bandwidth, why is
>this?
>
>I have been happily rsyncing all over the world for some time, but today I
>had cause to sync 40GB of data to all our remote sites causing me to look
>closer at the statistics.
>
>We effectively pull data using.
>
>rsync -az --delete --force hqserver::filesystem /export/sync/filesystem
>
>Our network is a 20Meg+ T3 at HQ and a T1/E1 at each remote site. All
>running Hardware VPN over internet.
>
>When rsync is the only network user I am only getting a fraction of the
>available bandwith utilized. With the above command running at each site I
>used 'show int' on the VPN router to get a breakdown of bandwidth use on a
>site to site basis.
>
>Total Outbound Usage 5354000 bits/sec (out of 20Meg) at California
>>From CA to Arizona 1071000bits/sec (Out of 1.5Meg)
>>From CA to Texas    838000bits/sec (Out of 1.5Meg)
>>From CA to Boston   672000bits/sec (Out of 1.5Meg)
>>From CA to France   429000bits/dec (Out of 2.0Meg)
>>From CA to UK       312000bits/sec (Out of 2.0Meg)
>>From CA to Germany  444000bits/dec (Out of 500K)
>>From CA to Korea    452000bits/dec (Out of 1.5Meg)
>>From CA to Taiwan   395000bits/dec (Out of 1.5Meg)
>>From CA to Singapore 340000bits/dec (Out of unknown)
>
>Finally, when I run several bandwidth using apps (such as ftp, scp etc)
>concurrently I can get close to line speed at any site all sites.
>
>>From this I conclude a single rsync at a site can not utilize all available
>bandwidth. It is also interesting to note that the bandwidth the rsync is
>able to utilize is directly proportional to the distance (latency) to the
>remote site.
>
>Now the question, Is there any way to optimize these transfers so that a
>single rsync can use the entire bandwidth available?
>
>I considered trying different block sizes but most references I found
>suggested that I leave it to rsync to find the optimum size.
>
>Dave.
>
>--
>To unsubscribe or change options: 
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html