AIO on Samba 3.6 doesn't result in parallel IO being issued to the underlying file system

[Jeremy Allison]

On Fri, Feb 06, 2015 at 04:11:39PM +0000, Kevin Osborn wrote:
> We are using Samba 3.6.24 on top of a custom VFS file system and we are trying to improve read and write throughput. We compiled Samba with aio enabled and I have verified from the debug output that aio calls are being made. I have also verified from a network trace that the client is sending parallel IO requests to Samba. But I still see the IO requests being serialized as they come into our VFS layer. I never see simultaneous IO requests pending come into the VFS layer.
> 
> While researching this problem I saw a presentation on line by
> Jeremy: "JeremyAllison_The_Evolution_of_Samba-ion.pdf" and it had 
> a rather depressing comment: "glibc forces multiple outstanding I/O
> requests on the same file descriptor to be synchronous."

Yep - this is a horrible bug in glibc. I sent a fix for
this to Ulrich (glibc maintainer at the time) but it was
ignored.

> Perhaps this is just denial on my part, but it was a little hard
> to discern which version of Samba corresponded with each stage of
> the evolution in the presentation. The presentation didn't mention
> the version of glibc at fault.

All of them (i.e. every one that has aio_read/aio_write etc.). And it's
still not fixed.

> So my question is this: Is it possible for us to get AIO requests on
> Samba 3.6.24? I am hoping that I have overlooked a setting somewhere.

Yes. On 3.6.x use vfs_aio_pthread or vfs_aio_fork.

> If there is no immediate solution for Samba 3.6, will upgrading to Samba
> 4.0 or 4.1 resolve the issue? (We plan to upgrade eventually but it is a
> bigger task than our schedule can bear at the moment.)

In 4.0.x and above the important parts of vfs_aio_pthread are
already built into the code, so configuring with threads and
setting the aio settings in smb.conf will do the trick.

Let me know if you need more help !

Jeremy.