Native Parallelization in rsync

[Jason Haar]

On 10/09/13 03:38, ameirh at gmail.com wrote:
> It's great you brought up bbcp. I didn't factor this into my initial
> email, but if we could further split up large files into N chunks and
> transport them concurrently, that would provide massive benefits as
> well.  
>
> It's clear there are multiple areas for improvement that would let us
> do away with having to use other tools for various scenarios.  If we
> could put some planning/development time in even the
> easiest-to-implement cases, it would be highly-beneficial to a large
> number of users.
>

Indeed. I actually hacked together a shell script wrapper around rsync
which would take the directory you were wanting to copy, tar it up and
then split the resultant large file into 'N' chunks - then transfer the
chunks in parallel. Using tricks with "post xfer" at the other end, I
would automagically unpack back to the original directory structure.
Major improvement in throughput - but only good for "new" file transfers
- not for replicating deltas/etc. And before you ask, no you can't have
it :-) It's part of a rather complex file replication environment we
have and isn't a module that could easily be detached for redistribution

To finish with, even though we can make rsync run much "faster" over
WANs, we don't use the feature I just mentioned! Just because you can
saturate your WAN pipe doesn't mean you should - typically there's other
traffic that also needs to co-exist and such a hammering would have
consequences - things have to be thought through. It would be nice to
have the ability, but that doesn't mean everyone would use it all the time

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20130910/47abcf93/attachment.html>