[Bug 5124] Parallelize the rsync run using multiple threads and/or connections
L.A. Walsh
rsync at tlinx.org
Tue Jan 28 05:26:01 MST 2014
Matthias Schniedermeyer wrote:
> On 25.01.2014 21:03, L.A. Walsh wrote:
>> If rsync already hitting over 200MB (even over 100MB)/s I'd agree that using
>> multiple TCP connections would help as they could be processed by different
>> cpu's on both ends. But since it doesn't even hit 100MB/s locally, the
>> limit *isn't* TCP connections.
>
> Just FYI.
> Rsync 3.1 got much better in that regard.
> When i rsync a file locally from tmpfs to tmpfs rsync does that with
> about 560MB/s. Rsync 3.0.X managed to do less than half of that.
---
> rsync --version
rsync version 3.1.0 protocol version 31
To run an rsync and compare SRC with a DST specified by --compare-dest,
to an empty partition (i.e. just copy differences between SRC + cmp-dest
to the empty partition, on a 1TB partition takes 45-90 minutes.
Usually thats:
Home-2014.01.12-03.07.03 HnS -wi-ao--- 1.04g
Home-2014.01.14-03.07.07 HnS -wi-ao--- 2.05g
Home-2014.01.16-03.07.02 HnS -wi-ao--- 1.42g
Home-2014.01.18-03.07.03 HnS -wi-ao--- 1.26g
Home-2014.01.20-03.07.03 HnS -wi-ao--- 2.30g
Home-2014.01.21-03.07.03 HnS -wi-ao--- 2.96g
Home-2014.01.22-03.07.03 HnS -wi-ao--- 1.57g
Home-2014.01.23-03.07.03 HnS -wi-ao--- 1.80g
1-3g in length.
3g/45minutes = 3g/2700s => 1.38MB/s -- not even close to 100MB.
It has to read time & date stamps of quite a few files, but my
best local speed has been under 100MB.
What size files are you transferring and how many? (average).
My times are for my home partition, (in case that wasn't
obvious from the partition names above... ;-))...
It has 4,986,955 files in 716132824K (~683M).
Lots of seeks, very little full speed reads (up to ~1TB/s raid)
So.. what size files and how much info are you transferring and
to/from what type of disks?
More information about the rsync
mailing list