Direct I/O support (patches included)

[Dag Wieers]

On Mon, 18 Feb 2013, Linda Walsh wrote:

> Hi dag, I really appreciate your working on this,
> but it is really annoying hard and tedious.
>
> _I_ am not certain about all the requirements of Direct I/O,
>
> I.e. would have to research (goog/kernel source...etc).
>
> It may be different on different platforms, I _vaguely_ remember
> 'talking'(email) with someone working on 'dd', and they were telling
> me how they had to compensate for a change in the kernel which used
> to handle the buffering of partial sector reads/writes for those
> who did directio on a device.  Then they decided that much hand-holding
> was wrong because, IMO, they basically wanted people to use the buffer
> cache -- since for most people, and most things it's a good thing.

It is different on some platforms, iirc the iozone source code has 
provisions for three (VX_DIRECT, O_DIRECT or O_DIRECTIO), albeit 
manageable. I would be already happy providing the functionality for 
systems supporting O_DIRECT (can't test the others anyway).


> Sorry about my example below -- I'd already replaced the --directio in
> the shell script with --drop-cache -- which I'd forgotten I already
> had in the script (memory for these things is completely gone!)..
>
> But really, it did have the direct-io in the statement before I
> changed it into a drop cache.

Ok, the "Invalid argument" error could be because the filesystem did not 
support direct I/O (e.g. NTFS) or there was a misalignment. On the system 
I wrote this patch for, it simply works out of the box on NFS. (RHEL5 64bit)


> Anyway, the first thing I'd want to find out is why it is writing to a socket 
> for a local file copy?

Because even when running rsync locally, it works as client-server over a 
socket ?


> It's going to be hard for direct I/O to make a difference (if it is workable,
> the fact that they move a 'window' over the source emulating a memory-mapped
> file isn't real helpful lining up memory with the sectors, but the minimums
> we need to go for a minimum read size of 4096 (am pretty sure that we have to 
> do that even on short files, and the kernel will just tell us we got less).
>
> 2nd thing -- need to make sure is to have a source of memory aligned 
> boundaries.

I'd look at the implementation in fio, it's well documented.

First improvement to the patch would be to bail out when we get EINVAL so 
it's clear that direct-io as-is does not work/alignment is not 
implemented.


-- 
-- dag wieers, dag at wieers.com, http://dag.wieers.com/
-- dagit linux solutions, info at dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]