An interesting shutdown race in messaging

[Jeremy Allison]

On Tue, Jun 13, 2017 at 05:14:01PM +1200, Andrew Bartlett wrote:
> G'Day,
> 
> I have another messaging issue that has come up.
> 
> In my attempts to make progress on the ldb locking code, I tried moving
> all of the servers in 'make test' to the standard process model. 
> 
> However, this hits an issue in the auth_log tests, which use messaging
> to contact an external python process, the tests, which expect to be
> sent the audit messages.
> 
> The issue is that in -M standard and in smbd, the connection can be
> closed so fast that the messages that are written to the logs, and have
> presumably been dispached to a messaging thread, never make it to the
> other end.
> 
> I can work around it by holding the connection open in some of the
> tests, but I can't easily do that for the tests that call rpcclient
> over subprocess, or where the password fails on LDAP.
> 
> So, I wonder if there is a way to, before we call exit(0) in the
> standard process model, have it try again to dispatch the messages to
> somewhere safely on the other side of a connected or datagram fd?
> 
> This can be seen in the auth_log tests in 
> http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/ldb-safe-locking-for-master
> 
> The commit that causes the issue is 
> http://git.catalyst.net.nz/gitweb?p=samba.git;a=commitdiff;h=6cf5e9f9b9f27af56a5a42ea6d7259f06f35240c
> 
> (changing all "single" to "standard" in Samba4.pm).
> 
> Fixing this would allow folks to use the messaging transport for the
> auth audit messages in a reliable way (instead of reading the logs),
> which isn't currently promised (only listed as a developer feature
> right now), but could be quite handy.  
> 
> Any clues most appreciated,
> 
> (Jeremy CC'ed because I mentioned this to him a long while ago).

So what you need is a 'drain_sent_messages' kind of function,
that will block until all pending messages are flushed out of
the system or the receiver disconnects, or an error send gets
and error - yes ?