AIO on Samba 3.6 doesn't result in parallel IO being issued to the underlying file system

[Jeremy Allison]

On Fri, Feb 06, 2015 at 08:56:14AM -0800, Jeremy Allison wrote:
> On Fri, Feb 06, 2015 at 04:11:39PM +0000, Kevin Osborn wrote:
> > We are using Samba 3.6.24 on top of a custom VFS file system and we are trying to improve read and write throughput. We compiled Samba with aio enabled and I have verified from the debug output that aio calls are being made. I have also verified from a network trace that the client is sending parallel IO requests to Samba. But I still see the IO requests being serialized as they come into our VFS layer. I never see simultaneous IO requests pending come into the VFS layer.
> > 
> > While researching this problem I saw a presentation on line by
> > Jeremy: "JeremyAllison_The_Evolution_of_Samba-ion.pdf" and it had 
> > a rather depressing comment: "glibc forces multiple outstanding I/O
> > requests on the same file descriptor to be synchronous."
> 
> Yep - this is a horrible bug in glibc. I sent a fix for
> this to Ulrich (glibc maintainer at the time) but it was
> ignored.
> 
> > Perhaps this is just denial on my part, but it was a little hard
> > to discern which version of Samba corresponded with each stage of
> > the evolution in the presentation. The presentation didn't mention
> > the version of glibc at fault.
> 
> All of them (i.e. every one that has aio_read/aio_write etc.). And it's
> still not fixed.
> 
> > So my question is this: Is it possible for us to get AIO requests on
> > Samba 3.6.24? I am hoping that I have overlooked a setting somewhere.
> 
> Yes. On 3.6.x use vfs_aio_pthread or vfs_aio_fork.
> 
> > If there is no immediate solution for Samba 3.6, will upgrading to Samba
> > 4.0 or 4.1 resolve the issue? (We plan to upgrade eventually but it is a
> > bigger task than our schedule can bear at the moment.)
> 
> In 4.0.x and above the important parts of vfs_aio_pthread are
> already built into the code, so configuring with threads and
> setting the aio settings in smb.conf will do the trick.
> 
> Let me know if you need more help !

Ah - I missed the fact that you have a custom
VFS layer.

In that case, if your VFS layer can be
used safely via threads, I'd suggest that
you look closely at vfs_aio_pthread and
see if you can insert your own pread/pwrite
layer into that code pattern.

Should be easier in 4.x, as there are
explicit vfs_pread_send/vfs_write_send
async VFS calls.