tevent in multithreaded programs

[Jeff Layton]

On Fri, 30 Jan 2015 20:49:25 +0100
Volker Lendecke <Volker.Lendecke at SerNet.DE> wrote:

> On Fri, Jan 30, 2015 at 12:30:27PM -0500, Jeff Layton wrote:
> > Hi All!
> > 
> > I'm looking at using tevent in a multithreaded program, primarily to
> > handle socket descriptors. The basic idea is to have a tevent_wait loop
> > running in one thread that will handle events by dispatching them to a
> > workqueue (queue of jobs and a pile of threads to handle them).
> > 
> > So basically we'll open a listener socket, and set it up to be watched
> > by tevent and run that in one thread. When a connect request comes in,
> > I want to dispatch the accept to a workqueue, which will then add the
> > accept()ed fd to the event context.
> > 
> > Questions:
> > 
> > - is this legit? Can I add events in one thread while the event loop is
> >   running in another?
> > 
> > - if so, do I need to serialize that somehow (with pthread mutexes or
> >   something)? I assume so, since tevent is based on talloc and you need
> >   to synchronize between threads when dealing with talloc contexts...
> 
> You pretty much nailed it I guess. talloc does not play well
> with threads if you want to share contexts. If you have a
> talloc root per thread it should be okay, but the NULL
> context is always a problem.
> 
> Who will handle the tevent_loop_once?

A single dedicated thread for now. Eventually, I'd like to have
multiple threads running event loops, but they'd have different tevent
contexts.
 
> If it's only one
> thread, this might work. However, doing an tevent_add_fd
> from a thread where another thread loops in
> tevent_loop_once this will cause problems I guess. I'd
> rather use other mechanisms to make the tevent_loop_once
> thread add the fd.
> 

Thanks Volker,

That's basically what Simo said too on IRC. His suggestion was:

- create a pipe, a list and a lock
- dispatch the accept to a workqueue thread
- once the accept is done, acquire the lock and add the new fd to the list
- drop the lock and then write to the pipe
- pipe event handler reads and discards all of the pipe data then
  acquires the lock and adds all of the fds on the list to the tevent
  context

...I think that sounds pretty reasonable, and it's a lot simpler than
locking across multiple threads. It also means that we can use
tevent_wait too and avoid any locking in the event loop unless
we're pulling entries off the list.

Thanks for the insight!
-- 
Jeff Layton <jlayton at samba.org>