[Bug 5124] Parallelize the rsync run using multiple threads and/or connections

[L.A. Walsh]

samba-bugs at samba.org wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=5124
>
> --- Comment #5 from Andrew J. Kroll <forge at dr.ea.ms> 2014-01-19 03:10:35 UTC ---
>  Another proven case is your typical modern web browser.
> There is a very good reason why multiple connections are used to load in those
> pretty pictures you see. It is all about getting around the latency by using
> TCP as a double buffer
---
But multiple TCP connections are not used to load a single picture.  
They are used
for separate items on the page.  A single TCP stream CAN be very fast 
and rsync
isn't close to hitting that limit. 

The proof?   Using 1Gb connections, smb/cifs could get 125MB/s writes and
119MB/s reads -- the writes were at "theoretical speeds" and were 
faster, because
the sender doesn't have to wait for the ACKs with a large window size.

Using a 10Gb connection (OR 2 of them using link aggregation to deliver up
to 20Gb), I'm limited to 400-550MB/s -- I don't get close to theoretical
maximums -- the bottleneck -- the *single* TCP connection that is allowed
per user between client and server.  1 TCP connection is limited by
how fast the CPU's can process and move the data in memory.  Depending on
the transfer size, and direction I can move the CPU bounding between client
and server, but no matter what, it hits the cpu limit for processing 1 
TCP connection
on one end or the other.

If rsync already hitting over 200MB (even over 100MB)/s I'd agree that using
multiple TCP connections would help as they could be processed by different
cpu's on both ends.  But since it doesn't even hit 100MB/s locally, the
limit *isn't* TCP connections.

Note -- I could probably get faster speeds if CIFS wasn't limited to 
<64K packet
sizes, but that's another limitation.

1 TCP connection could go faster if I had faster CPU's, but that's not 
likely to happen
soon.  TCP isn't the limit.