Direct I/O support (patches included)
dag at wieers.com
Mon Feb 18 05:53:41 MST 2013
On Mon, 18 Feb 2013, Linda Walsh wrote:
> Hi dag, I really appreciate your working on this,
> but it is really annoying hard and tedious.
> _I_ am not certain about all the requirements of Direct I/O,
> I.e. would have to research (goog/kernel source...etc).
> It may be different on different platforms, I _vaguely_ remember
> 'talking'(email) with someone working on 'dd', and they were telling
> me how they had to compensate for a change in the kernel which used
> to handle the buffering of partial sector reads/writes for those
> who did directio on a device. Then they decided that much hand-holding
> was wrong because, IMO, they basically wanted people to use the buffer
> cache -- since for most people, and most things it's a good thing.
It is different on some platforms, iirc the iozone source code has
provisions for three (VX_DIRECT, O_DIRECT or O_DIRECTIO), albeit
manageable. I would be already happy providing the functionality for
systems supporting O_DIRECT (can't test the others anyway).
> Sorry about my example below -- I'd already replaced the --directio in
> the shell script with --drop-cache -- which I'd forgotten I already
> had in the script (memory for these things is completely gone!)..
> But really, it did have the direct-io in the statement before I
> changed it into a drop cache.
Ok, the "Invalid argument" error could be because the filesystem did not
support direct I/O (e.g. NTFS) or there was a misalignment. On the system
I wrote this patch for, it simply works out of the box on NFS. (RHEL5 64bit)
> Anyway, the first thing I'd want to find out is why it is writing to a socket
> for a local file copy?
Because even when running rsync locally, it works as client-server over a
> It's going to be hard for direct I/O to make a difference (if it is workable,
> the fact that they move a 'window' over the source emulating a memory-mapped
> file isn't real helpful lining up memory with the sectors, but the minimums
> we need to go for a minimum read size of 4096 (am pretty sure that we have to
> do that even on short files, and the kernel will just tell us we got less).
> 2nd thing -- need to make sure is to have a source of memory aligned
I'd look at the implementation in fio, it's well documented.
First improvement to the patch would be to bail out when we get EINVAL so
it's clear that direct-io as-is does not work/alignment is not
-- dag wieers, dag at wieers.com, http://dag.wieers.com/
-- dagit linux solutions, info at dagit.net, http://dagit.net/
[Any errors in spelling, tact or fact are transmission errors]
More information about the rsync