Rsync with direct I/O

[Linda Walsh]

But in copying real files, there's alot more going on that file I/O...
and that's where rysnc spends most of its time.

If file-I/O dominates, and not much data to xfer, rsync is a win, but if 
data to
xfer is 'full' .. rsync will always be notably slower.

(as you note below...though it's worse than the figures below indicate 
if you
subtract out diskio)...  (see another data sample below)

Dan Stromberg wrote:
>
>
>     In my tests I saw the highest CPU usage with rsync and lowest
>     throughput:
>
>     *cp (no direct I/O):*
>
>     real    1m10.767s
>     user    0m0.692s
>     sys     0m47.401s
>
>     *Rsync (no direct I/O):*
>
>     real    2m40.552s
>     user    3m36.020s
>     sys     1m8.181s
>
>     *dd (direct I/O)=*
>
>     real    0m46.869s
>     user    0m0.031s
>     sys     0m6.043s
>
>
>     *dd (no direct I/O):*
>
>     real    1m23.530s
>     user    0m0.048s
>     sys     0m54.304s
>
>     (so pretty close to the speed of cp).
>
-----
Ok, but you forgot to include 'sync' time ...


Here's another data point with a "real dir" (oooOOOooo)...


Test dir is an old python dir:
With 6357 files in 230 dirs w/88meg by 'du -s'.
(75 meg in data, 13meg slack).

(warm up cache.. note "cp -a copies acls, xattrs on systems that have them)

>  time cp -a python python2   #cold memory, so won't count this.
9.54sec 0.06usr 2.10sys (22.70% cpu)
>  rm -fr python2;sync             # run between every test..but won't 
include for brev.

cp (2 runs)

    >  time cp -a python python2

        1.76sec 0.04usr 1.49sys (87.41% cpu)
        1.52sec 0.03usr 1.30sys (87.90% cpu)


rsync (2 runs)

    >  time rsync -aHAX python python2

        5.04sec 0.91usr 4.52sys (107.73% cpu)
        5.03sec 1.21usr 4.57sys (114.93% cpu)


'star'   (with -acl -xattr)'  (needs target dir pre-created as it only 
copies
a dir into a target dir...  (2 runs...)  (prog output only shown in 
first run)

    Ishtar:law> mkdir python2 && time star_dircopy python python2
    star: fifo had 26114 puts 26114 gets.
    star: fifo was 1 times empty and 0 times full.
    star: fifo held 85532672 bytes max, size was 268435456 bytes
    star: 0 blocks + 85532672 bytes (total of 85532672 bytes = 83528.00k).
    star: Total time 2.018sec (41391 kBytes/sec)

        2.04sec 0.18usr 1.67sys (90.99% cpu)
        2.03sec 0.15usr 1.72sys (92.79% cpu)


rsync is the slowest in the batch by over 100%
(note -- they ALL were copying everything.)...this wasn't a test of raw 
disk-io...

the disk speed (w/dd)  (w/o sync, w/sync w/direct i/o)

'dd' copying '88M' to disk  (warming up the cache...)
>  dd if=/dev/zero of=88m bs=88M count=1
92274688 bytes (92 MB) copied, 0.188274 s, 490 MB/s

### hot cache to ...to pre-existing file (no sync)
>  time dd if=/dev/zero of=88m bs=88M count=1 conv=notrunc,nocreat
92274688 bytes (92 MB) copied, 0.12344 s, 748 MB/s

    0.13sec 0.00usr 0.13sys (97.95% cpu)


## hot cache to new file...

>  time dd if=88m of=88m-2 bs=88M count=1 conv=notrunc,nocreat   
92274688 bytes (92 MB) copied, 0.134528 s, 686 MB/s

    0.14sec 0.00usr 0.14sys (98.23% cpu)

---


Buffered w/sync:

>  time dd if=/dev/zero of=88m bs=88M count=1 conv=notrunc oflag=sync
92274688 bytes (92 MB) copied, 0.372705 s, 248 MB/s

    0.38sec 0.00usr 0.37sys (97.58% cpu)

92274688 bytes (92 MB) copied, 0.408106 s, 226 MB/s

    0.42sec 0.00usr 0.38sys (90.81% cpu)

92274688 bytes (92 MB) copied, 0.403897 s, 228 MB/s

    0.41sec 0.00usr 0.37sys (90.70% cpu)


Direct I/O:

>  time dd if=/dev/zero of=88m bs=88M count=1 conv=notrunc,nocreat 
oflag=direct
92274688 bytes (92 MB) copied, 0.332803 s, 277 MB/s

    0.34sec 0.00usr 0.32sys (95.32% cpu)

92274688 bytes (92 MB) copied, 0.306348 s, 301 MB/s

    0.31sec 0.00usr 0.29sys (94.13% cpu)

92274688 bytes (92 MB) copied, 0.296805 s, 311 MB/s

    0.30sec 0.00usr 0.30sys (98.67% cpu)



---
cp will show similar rates to dd...depends on variables...


The 'data' is really a small part of the xfer cost -- especially w/rsync,
so adding direct I/O to rsync wouldn't really buy you much...


(i.e. if 5 seconds dropped to 4.9 seconds, would you care?)