[Bug 5124] Parallelize the rsync run using multiple threads and/or connections

Jason Haar Jason_Haar at trimble.com
Mon Feb 10 17:01:52 MST 2014


On 26/01/14 18:03, L.A. Walsh wrote:
> But multiple TCP connections are not used to load a single picture. 
> They are used
> for separate items on the page.  A single TCP stream CAN be very fast
> and rsync
> isn't close to hitting that limit.
> The proof?   Using 1Gb connections, smb/cifs could get 125MB/s writes and
> 119MB/s reads -- the writes were at "theoretical speeds" and were
> faster, because
> the sender doesn't have to wait for the ACKs with a large window size.
>

A bit late but I'll add my 2c worth.

bbcp - multi-tcp/threaded application. Completely nails rsync when
transferring over high-bandwidth/high-latency links

http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html

Like bittorrent, it establishes multiple TCP channels between a bbcp
client and server, and I guess has a parent process that tells each
child what part of the directory structure/data stream it is responsible
for, and joins it all up at the other end

I have tested rsync over a 100Mbs continental link and am lucky to get
10Mbs. Using bbcp with 4-6 channels, I can get 40-50Mbs (that's on a
link with other real traffic on it - so it may have actually got
80-90Mbs byt itself for all I know)

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1



More information about the rsync mailing list