Native Parallelization in rsync
Jason_Haar at trimble.com
Mon Sep 9 22:58:07 CEST 2013
On 10/09/13 03:38, ameirh at gmail.com wrote:
> It's great you brought up bbcp. I didn't factor this into my initial
> email, but if we could further split up large files into N chunks and
> transport them concurrently, that would provide massive benefits as
> It's clear there are multiple areas for improvement that would let us
> do away with having to use other tools for various scenarios. If we
> could put some planning/development time in even the
> easiest-to-implement cases, it would be highly-beneficial to a large
> number of users.
Indeed. I actually hacked together a shell script wrapper around rsync
which would take the directory you were wanting to copy, tar it up and
then split the resultant large file into 'N' chunks - then transfer the
chunks in parallel. Using tricks with "post xfer" at the other end, I
would automagically unpack back to the original directory structure.
Major improvement in throughput - but only good for "new" file transfers
- not for replicating deltas/etc. And before you ask, no you can't have
it :-) It's part of a rather complex file replication environment we
have and isn't a module that could easily be detached for redistribution
To finish with, even though we can make rsync run much "faster" over
WANs, we don't use the feature I just mentioned! Just because you can
saturate your WAN pipe doesn't mean you should - typically there's other
traffic that also needs to co-exist and such a hammering would have
consequences - things have to be thought through. It would be nice to
have the ability, but that doesn't mean everyone would use it all the time
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rsync