Rsync & parallelizing it for files in same directory

[Matt McCutchen]

On Fri, 2008-11-28 at 12:40 +0100, Hendrik Visage wrote:
> Now the question I'm wondering about:
> 
>  - Ease of doing a threading type setup inside rsync.
>  - Use threading architecture, to generate the checksums in parallel
> (we have >24 cores per system around here....)
>  - rsync then opens multiple connections to the remote site (Bandwidth
> optimization with the latencies involved)
>  - rsync then typically sends the files in a round robin fashion over
> the "links"
>  - or rsync does it directory per link for small directories.

There is a similar enhancement request:

https://bugzilla.samba.org/show_bug.cgi?id=5124

This would take a major redesign of rsync which I don't think is going
to happen anytime soon.  It may make more sense to write a new, better
designed tool (possibly adopting some of Wayne and Martin Pool's ideas
for "superlifter") that is capable of parallelism.  I would be
interested in helping if I could find the time.

Matt