rsync replacement

[Jamie Lokier]

Jason Haar wrote:
> Jamie Lokier wrote:
> > Check out the "TCP: advanced congestion control" option in a 2.6 Linux
> > kernel, and there is plenty of research on the topic.  See SCTP and
> > DSCP (among others) for the more transaction oriented side.
> >   
> Hi there Jamie
> 
> Like yourself, our WAN (VPN over Internet) suffers majorly from "large
> pipe, but large latency" issues.

The jargon is "large bandwidth-delay product".

> e.g. we have a 10Mbs Internet link, but
> can only get ~2Mbs for a single TCP stream (e.g. HTTP, Rsync).

If you aren't suffering from congestion (in the form of significant
latency jitter, or dropped packets), then the limit is due to the TCP
window size, which you can increase.

It is easy to tell if window size is the problem by looking at the
packet stream using Ethereal: every transmit will keep the window full.

Individual applications can increase it using the SO_SNDBUF and
SO_RCVBUF options.  (Use the appropriate option at the sending and
receiving sides :-)

The default (for applications which don't use those options) is
increased in an OS-specific way.  On Linux, you'd write different
values to /proc/sys/net/ipv4/{tcp_wmem,tcp_rmem,tcp_window_scaling}
and possibly other window-related settings.  Googling for "TCP window
size" and configuring Apache, Samba etc. will probably yield plenty of
suggestions.

If the slowdown is due to congestion or latency jitter, there's a lot
of tuning knobs on Linux and BSD, and the best choice depends on the
types of link you are running over.

You may also be limited by your router _intentionally_ limiting a
single TCP stream.  Or, if your 10Mbs link is implemented by an
aggregation of slower links, the aggregating routers may be mapping
individual TCP streams on to only one of those links (I've seen this,
I think), limiting the speed to one link.

If you have those router limits, all you can do is used multiple
streams.  If you switched to another protocol over UDP, the routers
would probably have the same limitation.  Assuming so, you can work
around it by bonding multiple UDP-based VPNs and run a single TCP stream
over that.  Don't try that until you're sure it's the problem :-)

> Are protocols like SCTP and DSCP really capable of helping in such
> situations? i.e would they improve the throughput of single streamed
> transactions?

No, SCTP and DSCP are not designed for that; they are more aimed at
having multiple concurrent transactions or streams multiplexed into a
single stream of larger packets, without individual delays affecting
every transaction, and sharing congestion control.  (However, you
could implement any of the things which run over UDP, using DSCP
instead and get OS-level congestion management, I think).

-- Jamie