Native Parallelization in rsync

[ameirh at gmail.com]

It's great you brought up bbcp.  I didn't factor this into my initial email, but if we could further split up large files into N chunks and transport them concurrently, that would provide massive benefits as well.  


It's clear there are multiple areas for improvement that would let us do away with having to use other tools for various scenarios.  If we could put some planning/development time in even the easiest-to-implement cases, it would be highly-beneficial to a large number of users.


-AmeirFrom: Jason_Haar at trimble.com<Jason_Haar at trimble.com>
To: <rsync at lists.samba.org>
Sent: Sunday, September 8, 2013
Subject: Re: Native Parallelization in rsync

On 08/09/13 18:28, Francis.Montagnac at inria.fr wrote:
> There is another rather common case I think where this would help:
> when the destination directory is located on a NAS over NFS.

I think the biggest use-case is high bandwidth, high latency links. eg
you have a 50Mbs WAN link to a site in another country, and can never
get more than (say) 5Mbs. Parallelizing the data transfer could easily
push that up to 20-30Mbs

...and there is a "competitor" to rsync that does this - bbcp. Mirror a
directory from hostA to hostB using 'N' tcp streams. Runs like the
clappers :-)


-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20130909/5af2da83/attachment.html>