[RFC] [WIP] tevent/glib integration

[Stefan Metzmacher]

Hi Noel,

very interesting work! It would be really good to have this
problem solved cleanly in future.

However while reading the patches the first time, I have the feeling
that the approach adds a lot of complexity and yet another layer
of abstraction.

We used to have integration into a glib/gtk mainloop in the other direction
See the following link to the historic source:

https://git.samba.org/?p=samba-gtk.git;a=blob;f=common/gtk_events.c;hb=4129587eda424b80225db06c12815414f1c17888

That approach used the glib mainloop as master and was designed for gtk
applications
using our client libraries.

I think doing it in the other direction (where tevent is the master)
is also possible even without modifying the core tevent code!

Instead we can use a tracecallback and hook into the
TEVENT_TRACE_BEFORE_WAIT (and maybe TEVENT_TRACE_AFTER_WAIT)
events.

For samba this could be done within smbd_tevent_trace_callback().

In order to make this more generic we could add a helper function
which can be called in a TEVENT_TRACE_BEFORE_WAIT hook.

This function would maintain some metadata, it needs an array
of tevent_fd pointers and old array returned by g_main_context_query()
and have a 2nd array to get the new g_main_context_query() result.
Then it uses this information to work out which tevent_fd pointers need
to be added/modified/removed.

Instead of using g_main_context_default(), we should make use of
g_main_context_new(), g_main_loop_new() and
g_main_context_{push,pop}_thread_default()

See the current spotlight code.
https://git.samba.org/?p=samba.git;a=blob;f=source3/rpc_server/mdssvc/mdssvc.c;hb=HEAD#l1807

Does that make sense to you?

metze

Am 16.12.2015 um 13:44 schrieb Noel Power:
> Hi
> 
> I started experimenting with trying to integrate glib with tevent,
> specifically my goal is to be able to use tracker-sparql asynchronously
> without the need for funky polling or iterating mechanisms. Generally
> it's not a good idea to run multiple event loops simultaneously, it's
> best if one loop is 'master' and the other event loop's events are
> somehow integrated into the 'master' loop. Also I don't really want (if
> possible) to have a solution specifically tailored for tracker-sparql
> but rather have something that would allow other GLib/GObject library
> based functionality to be used in a similar way. GLib provides the
> following functions for interation with an external event loops
>     g_main_context_prepare(), g_main_context_query(),
> g_main_context_check() and g_main_context_dispatch()
> To effectively use these functions you need to be able to access the
> underlying 'poll/select/epoll' functionality to pass foreign fds and
> timeouts, Why stop at glib, there are other event loops out there too so
> I tried to work on a generic solution.
> 
> attached and available at
> http://cgit.freedesktop.org/~noelp/noelp-samba/log/?h=glib-integration-wip&qt=range&q=17bc0fab767e59976c6360739d31aa9ad2febc5b...eae9982000eaf1bd4345900e561fd09de64d96b0
> are a set of patches showing what I am doing (well trying anyway)
> 
> The idea is quite simple, allow 2 callback functions to be registered
> 'prepare_poll' & 'check_poll'
>  + 'prepare_poll' allows a set of file descriptors and associated
> timeout to be provided to the underlying poll/epoll/select function used
> by tevent
>  + 'check_poll' tells the results of the poll
> 
> Note: the 'foreign' file descriptors although passed to the same 'poll'
> function call as any existing tevent file descriptors are processed
> separately
> 
> prepare_poll (ask for fds to monitor and timeout for poll)
>       |
> poll/select/epoll
>       |
> process foriegn fds (examine results for monitored events etc.)
>       |
> check_poll (tell the polll is complete passing some result)
>       |
> normal tevent fd processing (continue normal tevent processing)
> 
> Each tevent backend needs to be modified to call the 'prepare_poll' &
> 'check_poll' functions
> 
> All the required glue code needed to provide the specific event loop
> integration should be performed in 'prepare-poll' & 'check_poll'
> 
> e.g for glib;
>  + 'prepare_poll'  calls ' g_main_context_prepare()' &
> 'g_main_context_query()'
>  + 'check_poll'  calls 'g_main_context_check()' and
> 'g_main_context_dispatch()'
> 
> The set of patches provides
> 
>  +  2 modified tevent backends
>         a) the 'poll' backend in the tevent library, I only modified one
> for demonstration and testing, without discussion seemed no point in
> doing the others at this point
>         b) the 's3' backend in samba
> 
>  + Standalone example 'source3/utils/async-tracker.c it produces a test
> binary ./bin/default/source3/tevent_glib
> 
>      ./bin/default/source3/tevent_glib -h
>      tevent_glib [-t|-h|-g]
>           -t use tevent event loop
>           -g use glib event loop (default)
>           -h help
>   + a single function 'tevent_integrate_glib' which can be used with
> processes that use the 's3' tevent backend to enable glib integration
> 
> if you want to try it out apply the patches (they should apply to
> current master) you need to a configure with  '--enable-spotlight' in
> order to be able to run the example above *and* enable the 's3' glib
> event integration, it should build without it (but will only provide the
> foreign file descriptor 'poll' integration and no glib event integration)
> 
> If you have got this far, well done
> 
> some things to consider
> 
> a) this is experimental and *very* rough and ready still
> b) I only discovered the 's3' backend after doing the work in the tevent
> library, so it would be possible to not expose any of this in the tevent
> library at all but instead just implement glib integration (in the same
> way) internally in samba (would that be preferred ?)
> 
> thanks and looking forward to some comments
> 
> Noel
> 
> p.s. Ralph I *really* hope you are interested in this and would be
> willing to test this against your spotlight stuff
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20151216/e5194069/signature.sig>