Rsync with direct I/O
Linda Walsh
rsync at tlinx.org
Sat Sep 10 20:30:24 MDT 2011
But in copying real files, there's alot more going on that file I/O...
and that's where rysnc spends most of its time.
If file-I/O dominates, and not much data to xfer, rsync is a win, but if
data to
xfer is 'full' .. rsync will always be notably slower.
(as you note below...though it's worse than the figures below indicate
if you
subtract out diskio)... (see another data sample below)
Dan Stromberg wrote:
>
>
> In my tests I saw the highest CPU usage with rsync and lowest
> throughput:
>
> *cp (no direct I/O):*
>
> real 1m10.767s
> user 0m0.692s
> sys 0m47.401s
>
> *Rsync (no direct I/O):*
>
> real 2m40.552s
> user 3m36.020s
> sys 1m8.181s
>
> *dd (direct I/O)=*
>
> real 0m46.869s
> user 0m0.031s
> sys 0m6.043s
>
>
> *dd (no direct I/O):*
>
> real 1m23.530s
> user 0m0.048s
> sys 0m54.304s
>
> (so pretty close to the speed of cp).
>
-----
Ok, but you forgot to include 'sync' time ...
Here's another data point with a "real dir" (oooOOOooo)...
Test dir is an old python dir:
With 6357 files in 230 dirs w/88meg by 'du -s'.
(75 meg in data, 13meg slack).
(warm up cache.. note "cp -a copies acls, xattrs on systems that have them)
> time cp -a python python2 #cold memory, so won't count this.
9.54sec 0.06usr 2.10sys (22.70% cpu)
> rm -fr python2;sync # run between every test..but won't
include for brev.
cp (2 runs)
> time cp -a python python2
1.76sec 0.04usr 1.49sys (87.41% cpu)
1.52sec 0.03usr 1.30sys (87.90% cpu)
rsync (2 runs)
> time rsync -aHAX python python2
5.04sec 0.91usr 4.52sys (107.73% cpu)
5.03sec 1.21usr 4.57sys (114.93% cpu)
'star' (with -acl -xattr)' (needs target dir pre-created as it only
copies
a dir into a target dir... (2 runs...) (prog output only shown in
first run)
Ishtar:law> mkdir python2 && time star_dircopy python python2
star: fifo had 26114 puts 26114 gets.
star: fifo was 1 times empty and 0 times full.
star: fifo held 85532672 bytes max, size was 268435456 bytes
star: 0 blocks + 85532672 bytes (total of 85532672 bytes = 83528.00k).
star: Total time 2.018sec (41391 kBytes/sec)
2.04sec 0.18usr 1.67sys (90.99% cpu)
2.03sec 0.15usr 1.72sys (92.79% cpu)
rsync is the slowest in the batch by over 100%
(note -- they ALL were copying everything.)...this wasn't a test of raw
disk-io...
the disk speed (w/dd) (w/o sync, w/sync w/direct i/o)
'dd' copying '88M' to disk (warming up the cache...)
> dd if=/dev/zero of=88m bs=88M count=1
92274688 bytes (92 MB) copied, 0.188274 s, 490 MB/s
### hot cache to ...to pre-existing file (no sync)
> time dd if=/dev/zero of=88m bs=88M count=1 conv=notrunc,nocreat
92274688 bytes (92 MB) copied, 0.12344 s, 748 MB/s
0.13sec 0.00usr 0.13sys (97.95% cpu)
## hot cache to new file...
> time dd if=88m of=88m-2 bs=88M count=1 conv=notrunc,nocreat
92274688 bytes (92 MB) copied, 0.134528 s, 686 MB/s
0.14sec 0.00usr 0.14sys (98.23% cpu)
---
Buffered w/sync:
> time dd if=/dev/zero of=88m bs=88M count=1 conv=notrunc oflag=sync
92274688 bytes (92 MB) copied, 0.372705 s, 248 MB/s
0.38sec 0.00usr 0.37sys (97.58% cpu)
92274688 bytes (92 MB) copied, 0.408106 s, 226 MB/s
0.42sec 0.00usr 0.38sys (90.81% cpu)
92274688 bytes (92 MB) copied, 0.403897 s, 228 MB/s
0.41sec 0.00usr 0.37sys (90.70% cpu)
Direct I/O:
> time dd if=/dev/zero of=88m bs=88M count=1 conv=notrunc,nocreat
oflag=direct
92274688 bytes (92 MB) copied, 0.332803 s, 277 MB/s
0.34sec 0.00usr 0.32sys (95.32% cpu)
92274688 bytes (92 MB) copied, 0.306348 s, 301 MB/s
0.31sec 0.00usr 0.29sys (94.13% cpu)
92274688 bytes (92 MB) copied, 0.296805 s, 311 MB/s
0.30sec 0.00usr 0.30sys (98.67% cpu)
---
cp will show similar rates to dd...depends on variables...
The 'data' is really a small part of the xfer cost -- especially w/rsync,
so adding direct I/O to rsync wouldn't really buy you much...
(i.e. if 5 seconds dropped to 4.9 seconds, would you care?)
More information about the rsync
mailing list