Rsync with direct I/O

[Sandon Van Ness]

  As far as I can tell rsync doesn't support this. How hard would it be 
to implement this? Is it trivial enough to just change the calls in the 
code with sed? I think this can significantly reduce CPU usage and 
increase I/O speed when dealing with fast storage solutions. It can make 
a huge difference when say transferring 30TB of data.

Here are some tests I did. So far the only thing I know of that can use 
direct I/O is dd which showed a huge improvement. CPU usage went from 
very high down to under 15% and the speed almost doubled.

Is there anything else I can do to tune rsync to give me better speed 
and/or less CPU usage as I seem to be hitting a CPU bottleneck in the 
case of rsync.

In my tests I saw the highest CPU usage with rsync and lowest throughput:

*cp (no direct I/O):*

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23356 root      20   0 10740  676  584 D   74  0.0   0:16.48 cp
23344 root      20   0     0    0    0 D    9  0.0   0:04.80 flush-8:64
   639 root      20   0     0    0    0 R    9  0.0   0:12.28 kswapd0


root at dekabutsu: 05:54 AM :/data2# time cp -avf /data/winxp.img /data2/
`/data/winxp.img' -> `/data2/winxp.img'

real    1m10.767s
user    0m0.692s
sys     0m47.401s

*Rsync (no direct I/O):*

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23397 root      20   0 11572  616  256 R   97  0.0   0:08.48 rsync
23395 root      20   0 11864 1292  848 R   84  0.0   0:07.35 rsync


root at dekabutsu: 05:56 AM :/data2# time rsync --progress -av 
/data/winxp.img /data2/
building file list ...
1 file to consider
winxp.img
  37589497344 100%  223.43MB/s    0:02:40 (xfer#1, to-check=0/1)

sent 37594086010 bytes  received 42 bytes  234231065.74 bytes/sec
total size is 37589497344  speedup is 1.00

real    2m40.552s
user    3m36.020s
sys     1m8.181s

*dd (direct I/O)=*

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23452 root      20   0 14960 1664  580 D   13  0.0   0:00.76 dd


root at dekabutsu: 06:00 AM :/data2# time dd bs=1M iflag=direct 
oflag=direct if=/data/winxp.img of=/data2/winxp.img
35848+1 records in
35848+1 records out
37589497344 bytes (38 GB) copied, 46.8669 s, 802 MB/s

real    0m46.869s
user    0m0.031s
sys     0m6.043s


*dd (no direct I/O):*

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23455 root      20   0 14960 1664  576 R   65  0.0   0:06.59 dd
23344 root      20   0     0    0    0 S   20  0.0   0:17.88 flush-8:64


root at dekabutsu: 06:01 AM :/data2# time dd bs=1M if=/data/winxp.img 
of=/data2/winxp.img
35848+1 records in
35848+1 records out
37589497344 bytes (38 GB) copied, 83.5282 s, 450 MB/s

real    1m23.530s
user    0m0.048s
sys     0m54.304s

(so pretty close to the speed of cp).