Rsync with direct I/O

[Dan Stromberg]

Direct I/O (assuming you mean O_DIRECT on open) can be a bit fiddly, but I
doubt it's out of reach.  The main difficulty is allocating a buffer with
appropriate alignment.

I put together a library to facilitate O_DIRECT I/O a while back:
http://stromberg.dnsalias.org/~dstromberg/odirect/

odirect is callable from C, C++ and CPython.  I hope to add support for
calling it via Python "ctypes" at some point in the future.

On Sun, Sep 4, 2011 at 6:09 AM, Sandon Van Ness <sandon at van-ness.com> wrote:

>  As far as I can tell rsync doesn't support this. How hard would it be to
> implement this? Is it trivial enough to just change the calls in the code
> with sed? I think this can significantly reduce CPU usage and increase I/O
> speed when dealing with fast storage solutions. It can make a huge
> difference when say transferring 30TB of data.
>
> Here are some tests I did. So far the only thing I know of that can use
> direct I/O is dd which showed a huge improvement. CPU usage went from very
> high down to under 15% and the speed almost doubled.
>
> Is there anything else I can do to tune rsync to give me better speed
> and/or less CPU usage as I seem to be hitting a CPU bottleneck in the case
> of rsync.
>
> In my tests I saw the highest CPU usage with rsync and lowest throughput:
>
> *cp (no direct I/O):*
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 23356 root      20   0 10740  676  584 D   74  0.0   0:16.48 cp
> 23344 root      20   0     0    0    0 D    9  0.0   0:04.80 flush-8:64
>  639 root      20   0     0    0    0 R    9  0.0   0:12.28 kswapd0
>
>
> root at dekabutsu: 05:54 AM :/data2# time cp -avf /data/winxp.img /data2/
> `/data/winxp.img' -> `/data2/winxp.img'
>
> real    1m10.767s
> user    0m0.692s
> sys     0m47.401s
>
> *Rsync (no direct I/O):*
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 23397 root      20   0 11572  616  256 R   97  0.0   0:08.48 rsync
> 23395 root      20   0 11864 1292  848 R   84  0.0   0:07.35 rsync
>
>
> root at dekabutsu: 05:56 AM :/data2# time rsync --progress -av
> /data/winxp.img /data2/
> building file list ...
> 1 file to consider
> winxp.img
>  37589497344 100%  223.43MB/s    0:02:40 (xfer#1, to-check=0/1)
>
> sent 37594086010 bytes  received 42 bytes  234231065.74 bytes/sec
> total size is 37589497344  speedup is 1.00
>
> real    2m40.552s
> user    3m36.020s
> sys     1m8.181s
>
> *dd (direct I/O)=*
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 23452 root      20   0 14960 1664  580 D   13  0.0   0:00.76 dd
>
>
> root at dekabutsu: 06:00 AM :/data2# time dd bs=1M iflag=direct oflag=direct
> if=/data/winxp.img of=/data2/winxp.img
> 35848+1 records in
> 35848+1 records out
> 37589497344 bytes (38 GB) copied, 46.8669 s, 802 MB/s
>
> real    0m46.869s
> user    0m0.031s
> sys     0m6.043s
>
>
> *dd (no direct I/O):*
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 23455 root      20   0 14960 1664  576 R   65  0.0   0:06.59 dd
> 23344 root      20   0     0    0    0 S   20  0.0   0:17.88 flush-8:64
>
>
> root at dekabutsu: 06:01 AM :/data2# time dd bs=1M if=/data/winxp.img
> of=/data2/winxp.img
> 35848+1 records in
> 35848+1 records out
> 37589497344 bytes (38 GB) copied, 83.5282 s, 450 MB/s
>
> real    1m23.530s
> user    0m0.048s
> sys     0m54.304s
>
> (so pretty close to the speed of cp).
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/**
> mailman/listinfo/rsync <https://lists.samba.org/mailman/listinfo/rsync>
> Before posting, read: http://www.catb.org/~esr/faqs/**smart-questions.html<http://www.catb.org/%7Eesr/faqs/smart-questions.html>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20110904/0e4ebffb/attachment.html>