<br>Direct I/O (assuming you mean O_DIRECT on open) can be a bit fiddly, but I doubt it's out of reach. The main difficulty is allocating a buffer with appropriate alignment.<br><br>I put together a library to facilitate O_DIRECT I/O a while back:<br>
<a href="http://stromberg.dnsalias.org/~dstromberg/odirect/">http://stromberg.dnsalias.org/~dstromberg/odirect/</a><br><br>odirect is callable from C, C++ and CPython. I hope to add support for calling it via Python "ctypes" at some point in the future.<br>
<br><div class="gmail_quote">On Sun, Sep 4, 2011 at 6:09 AM, Sandon Van Ness <span dir="ltr"><<a href="mailto:sandon@van-ness.com">sandon@van-ness.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
As far as I can tell rsync doesn't support this. How hard would it be to implement this? Is it trivial enough to just change the calls in the code with sed? I think this can significantly reduce CPU usage and increase I/O speed when dealing with fast storage solutions. It can make a huge difference when say transferring 30TB of data.<br>
<br>
Here are some tests I did. So far the only thing I know of that can use direct I/O is dd which showed a huge improvement. CPU usage went from very high down to under 15% and the speed almost doubled.<br>
<br>
Is there anything else I can do to tune rsync to give me better speed and/or less CPU usage as I seem to be hitting a CPU bottleneck in the case of rsync.<br>
<br>
In my tests I saw the highest CPU usage with rsync and lowest throughput:<br>
<br>
*cp (no direct I/O):*<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
23356 root 20 0 10740 676 584 D 74 0.0 0:16.48 cp<br>
23344 root 20 0 0 0 0 D 9 0.0 0:04.80 flush-8:64<br>
639 root 20 0 0 0 0 R 9 0.0 0:12.28 kswapd0<br>
<br>
<br>
root@dekabutsu: 05:54 AM :/data2# time cp -avf /data/winxp.img /data2/<br>
`/data/winxp.img' -> `/data2/winxp.img'<br>
<br>
real 1m10.767s<br>
user 0m0.692s<br>
sys 0m47.401s<br>
<br>
*Rsync (no direct I/O):*<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
23397 root 20 0 11572 616 256 R 97 0.0 0:08.48 rsync<br>
23395 root 20 0 11864 1292 848 R 84 0.0 0:07.35 rsync<br>
<br>
<br>
root@dekabutsu: 05:56 AM :/data2# time rsync --progress -av /data/winxp.img /data2/<br>
building file list ...<br>
1 file to consider<br>
winxp.img<br>
37589497344 100% 223.43MB/s 0:02:40 (xfer#1, to-check=0/1)<br>
<br>
sent 37594086010 bytes received 42 bytes 234231065.74 bytes/sec<br>
total size is 37589497344 speedup is 1.00<br>
<br>
real 2m40.552s<br>
user 3m36.020s<br>
sys 1m8.181s<br>
<br>
*dd (direct I/O)=*<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
23452 root 20 0 14960 1664 580 D 13 0.0 0:00.76 dd<br>
<br>
<br>
root@dekabutsu: 06:00 AM :/data2# time dd bs=1M iflag=direct oflag=direct if=/data/winxp.img of=/data2/winxp.img<br>
35848+1 records in<br>
35848+1 records out<br>
37589497344 bytes (38 GB) copied, 46.8669 s, 802 MB/s<br>
<br>
real 0m46.869s<br>
user 0m0.031s<br>
sys 0m6.043s<br>
<br>
<br>
*dd (no direct I/O):*<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
23455 root 20 0 14960 1664 576 R 65 0.0 0:06.59 dd<br>
23344 root 20 0 0 0 0 S 20 0.0 0:17.88 flush-8:64<br>
<br>
<br>
root@dekabutsu: 06:01 AM :/data2# time dd bs=1M if=/data/winxp.img of=/data2/winxp.img<br>
35848+1 records in<br>
35848+1 records out<br>
37589497344 bytes (38 GB) copied, 83.5282 s, 450 MB/s<br>
<br>
real 1m23.530s<br>
user 0m0.048s<br>
sys 0m54.304s<br>
<br>
(so pretty close to the speed of cp).<br><font color="#888888">
<br>
<br>
<br>
-- <br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/<u></u>mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/%7Eesr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/<u></u>smart-questions.html</a><br>
</font></blockquote></div><br>