rsync speed limited by read chunk size?

[CM Fields]

Is rsync speed limited to 160MB/sec read speeds due to the chunk read
size? Or, are we seeing just a coincidence and the limitation is
somewhere else?


Rsync (single process) is used to locally copy files from one raid
partition to another and we noticed rsync's read speed is limited.
Whether I setup an rsync between two raid disks or two ram drives,
rsync reads top out at around 160MB/sec. Writes are fine as ZFS easily
sustains 2GB/sec, we just see the rsync read speeds are limited. All
rsyncs are on the machine itself and do not cross the 10Gb network.

We have five(5) raid boxes in the following configuration. Raid card
caching is disabled which allows direct disk access.

Intel E5-2630
64GB RAM DDR3 2400MHz
LSI 9271-8i (caching disabled)
24x WD 4TB WD2001FYYG (2x 12 drive raid 6 arrays)
Ubuntu 12.04.2 LTS (patched as of today)
ZFS filesystem (v28 pool v5000, compression off, checksums on)


The following tests were run on all five boxes to rule out hardware
issues and tests were started when the machine was at a load of
zero(0). There are no files on the destination drive when the rsync is
started. While copies are running the machine's CPU usage never rises
about 10% (90% idle).


Test1: ram drive to ram drive rsync:

setup the ram disks:
 sudo mount -t tmpfs -o size=1024M tmpfs /tmp/ram1/
 sudo mount -t tmpfs -o size=1024M tmpfs /tmp/ram2/

rsync -avP /tmp/ram1/ /tmp/ram2/
 Fedora-Live-Desktop.iso
   997195776 100%  158.44MB/s    0:00:05 (xfer#1, to-check=0/2)


Test 2: rsync the iso from one raid disk to another raid disk:

rsync -avP /data1/ /data2/
 Fedora-Live-Desktop.iso
   997195776 100%  166.46MB/s    0:00:05 (xfer#1, to-check=0/2)


Test 3: dd with an increasing chuck size was able to read from ram
faster then rsync:

dd if=/tmp/ram1/Fedora-Live-Desktop.iso of=/dev/null bs=<value>

bs=512 1.2 GB/s
bs=1K  2.2 GB/s
bs=2K  3.5 GB/s
bs=4K  5.2 GB/s
bs=8K  6.8 GB/s
bs=64K 7.0 GB/s

Test 4: The reads from the raid with increasing read chunk size
resulted in better speeds. Notice the block size of 512 bytes is
164MB/sec which is similar to what we are seeing with rsync.

dd if=/data1/Fedora-Live-Desktop.iso of=/dev/null bs=<value>

bs=128   42 MB/s
bs=256   84 MB/s
bs=512  164 MB/s (dd's default bs size)
bs=1k   276 MB/s
bs=2k   567 MB/s
bs=4k   717 MB/s
bs=8k   938 MB/s
bs=16k  988 MB/s
bs=32k  1.1 GB/s
bs=64k  1.1 GB/s
bs=128k 1.2 GB/s <-- we want to be here
bs=256K 1.2 GB/s
bs=512K 1.2 GB/s


Test 5: Finally, a dd write of the iso to the raid partiitons from a
ram drive to show the raids can write at gigabyte speeds:

dd if=/tmp/ram1/Fedora-Live-Desktop.iso of=/data1/iso bs=<value>

bs=512  70.6 MB/s
bs=1K   137 MB/s
bs=2K   271 MB/s
bs=4K   472 MB/s
bs=8K   804 MB/s
bs=16K  1.2 GB/s
bs=32K  1.7 GB/s
bs=64K  2.0 GB/s
bs=128k 2.9 GB/s
bs=256K 3.0 GB/s
bs=512K 3.0 GB/s

What we are seeing is a speed limitation and _not_ a bug. Rsync works
perfectly fine otherwise. We would be happy to test out any patches.

Thanks for your time.