[PATCH] restructure messaging

[Andrew Bartlett]

On Fri, 2016-09-30 at 08:26 -0700, Volker Lendecke wrote:
> On Wed, Sep 28, 2016 at 04:33:23PM -0700, Volker Lendecke wrote:
> > 
> > Looking for a scary patchset? Here it is....
> > 
> > Main reason for this patchset is the last two ones: This patches
> > passes down the event context that triggered a receiving message
> > callback to the handlers and only acts upon them in the right
> > context.
> > 
> > On the way there I've eliminated poll_funcs and made our
> > pthreadpool
> > safe against run-down with blocked threads. It's the last part that
> > again and again reminded me of
> > 
> > http://bholley.net/blog/2015/must-be-this-tall-to-write-multi-threa
> > ded-code.html

I love this link :-)

> > Consider it WIP, it has to survive a few autobuilds and the
> > tevent version needs to be bumped.
> > 
> > Review appreciated!
> 
> Attached find a version that survived two autobuilds. Thanks to metze
> for some insight on tevent_fd, see patch 30/30 :-)

Particularly with the patch for self-send, this looks like it might
address a number of our messaging/ldb interactions. 

I have a few comments and questions:

For "[PATCH 20/30] messages_dgm: Drop a segment if we can't ship it for
60 seconds" does this impact on the IRPC use case?  Under what
situation would this occur?

I ask because it is quite possible that we might be in a replication-
induced transaction for more than 60 seconds with the current code. 

Can you add a variation on "Subject: [PATCH 16/30] messaging: add an
overflow test" that confirms that all the messages are picked up?

It would also be really useful to have a unit test for the fire-and-
forget use of imessaging and IRPC used in ridalloc_poke_rid_manager(),
auth_sam_trigger_repl_secret() and notify_dns dns_notify_dnssrv_send()
because these seem to be part of our current problems.  Testing those
for overflow and normal moves in self-handled, self-handled-but-in-
another-loop and other-process-blocked states might move this from
'reproduce in a 4 hour autobuild' to 'debug in a unit test'.  

Volker, finally I want to say that I really appreciate the work you
have put in to making this area more reliable.  It was so good to be
able to talk about possible solutions to some of our most painful and
persistent issues.

Thanks!

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba