I wonder whether pushing the complexity of tevent into the VFS was a good idea?

[Volker Lendecke]

On Sat, Nov 10, 2012 at 04:23:08PM -0800, Richard Sharpe wrote:
> Yes, I have been getting my head around it ... one of the difficulties
> is all these functions all over the place that are called
> wb_simple_trans_send, which of course, does no such thing. It queues
> an event that will call the function wb_simple_trans_send, which
> queues an event that will call the function wb_req_write_send which
> actually queues an event to call writev_send which queues an event to
> call an underlying writev_send function ... which just might call
> writev or something like it ... and then there are all these call back
> routines merrily calling callback routines and calling routines
> misleadingly called xxx_recv ...

As confusing as those layers might be, they are one of the
main reasons we did tevent_req the way we did it: It makes
async sequences of events composable and stackable. The
previous posix API was not stackable at all. No for example
you can pretty easily have a virus scanner and a HSM engine
doing the pread in an asynchronous fashion one after the
other. Also, the S3 module could easily defer the pread and
later have a virus scanner scan the module, all in an async
manner.

> (and the name struct tevent_req *tdgram_recvfrom_send is confusing
> enough but tdgram_recvfrom_recv is worse)

I would also argue that the _send and _recv style is really
regular. Sure dgram_recv_recv might be confusing, but it is
the finishing end of an async recv operation. So far we have
not been able to come up with a better naming scheme, and
for this level of complexity in operations I find naming
things consistently pretty important.

> Still, I have to get my head around it so I will press on.

Feel free to ask specific questions in case you have any.
Maybe you also want to post your own module here for review.
Maybe some iterations of review can clean up some
irritations.

> > In short, while I understand where you are coming from, I'm hesitant to
> > have another situation like ldb, which has an async modal that isn't
> > tevent, and which (due to this and other issues, like a primary use case
> > that is pure sync in any case) has therefore essentially been unable to
> > be used async in Samba.
> 
> Yes. I can understand that. However, up until now the contract that
> the VFS has provided, I think, is that all calls have the feeling of
> standard function calls.

The problem with AIO for me is that the Posix AIO using
signals severely sucks. See the composability issue above.

> Now, I happen to think that the thread-pool idea that AIO has been
> implemented on recently is a good idea and wonder if VFS module
> writers can take advantage of that when writing their own VFS modules.

Feel free to use pthreadpool.h on your own. Please make 100%
sure to only call thread-safe functions from within the job
workers. This rules out pretty much anything from the Samba
internal API, so far we only use direct syscalls.

With best regards,

Volker Lendecke

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de