[Bug 5124] Parallelize the rsync run using multiple threads and/or connections

[L.A. Walsh]

Matthias Schniedermeyer wrote:
> On 25.01.2014 21:03, L.A. Walsh wrote:
>> If rsync already hitting over 200MB (even over 100MB)/s I'd agree that using
>> multiple TCP connections would help as they could be processed by different
>> cpu's on both ends.  But since it doesn't even hit 100MB/s locally, the
>> limit *isn't* TCP connections.
> 
> Just FYI.
> Rsync 3.1 got much better in that regard.
> When i rsync a file locally from tmpfs to tmpfs rsync does that with 
> about 560MB/s. Rsync 3.0.X managed to do less than half of that.
---

> rsync --version
rsync  version 3.1.0  protocol version 31

To run an rsync and compare SRC with a DST specified by --compare-dest,
to an empty partition (i.e. just copy differences between SRC + cmp-dest
to the empty partition, on a 1TB partition takes 45-90 minutes.
Usually thats:
  Home-2014.01.12-03.07.03 HnS     -wi-ao---   1.04g
   Home-2014.01.14-03.07.07 HnS     -wi-ao---   2.05g
   Home-2014.01.16-03.07.02 HnS     -wi-ao---   1.42g
   Home-2014.01.18-03.07.03 HnS     -wi-ao---   1.26g
   Home-2014.01.20-03.07.03 HnS     -wi-ao---   2.30g
   Home-2014.01.21-03.07.03 HnS     -wi-ao---   2.96g
   Home-2014.01.22-03.07.03 HnS     -wi-ao---   1.57g
   Home-2014.01.23-03.07.03 HnS     -wi-ao---   1.80g


1-3g in length.

3g/45minutes = 3g/2700s => 1.38MB/s -- not even close to 100MB.

It has to read time & date stamps of quite a few files, but my
best local speed has been under 100MB.

What size files are you transferring and how many?  (average).

My times are for my home partition, (in case that wasn't
obvious from the partition names above... ;-))...

It has 4,986,955 files in 716132824K (~683M).
Lots of seeks, very little full speed reads (up to ~1TB/s raid)


So.. what size files and how much info are you transferring and
to/from what type of disks?