[PATCH] New debug backend "ringbuf"

[Ralph Böhme]

On Tue, Jan 10, 2017 at 08:21:56AM +1300, Andrew Bartlett wrote:
> On Mon, 2017-01-09 at 11:17 -0800, Jeremy Allison wrote:
> > On Sun, Jan 08, 2017 at 04:59:09PM +0100, Ralph Böhme wrote:
> > > On Sun, Jan 08, 2017 at 12:49:57PM +0100, Ralph Böhme wrote:
> > > > On Sun, Jan 08, 2017 at 10:48:26AM +0100, Volker Lendecke wrote:
> > > > > On Sun, Jan 08, 2017 at 10:08:53AM +0100, Ralph Böhme wrote:
> > > > > > This one made my day when trying to fix a subtle bug in some
> > > > > > asyc
> > > > > > code in ctdbd_conn.c I'm currently working on. The bug was
> > > > > > subject
> > > > > > to timing effects, I could easily reproduce it at log level
> > > > > > 1, but
> > > > > > as soon as I cranked up log level it hid away. Arg.
> > > > > > 
> > > > > > I tried moving the logfile to a memdisk, but even with that,
> > > > > > timings
> > > > > > were different enough so I still couldn't reproduce it.
> > > > > > Ahhhhhh...
> > > > > > 
> > > > > > That's when I thought of adding a new logging backend that
> > > > > > simply
> > > > > > writes all log messages into a memory buffer, et voila, with
> > > > > > this I
> > > > > > was able to reproduce the issue at log level 10. Bug found
> > > > > > and
> > > > > > fixed. :)
> > > > > > 
> > > > > > To make use of it when hunting for a bug, you must be able to
> > > > > > identify a place in the codepath where you know things are
> > > > > > gone
> > > > > > south and you can cause a panic. You then attach with gdb to
> > > > > > the
> > > > > > panicked (and in developer mode sleeping process, otherwise
> > > > > > grab the
> > > > > > corefile) and run:
> > > > > > 
> > > > > > (gdb) dump binary memory samba.log debug_ringbuf
> > > > > > debug_ringbuf+SIZE
> > > > > > 
> > > > > > The default size is 1 MB, but this can be changed by a
> > > > > > backend
> > > > > > option:
> > > > > > 
> > > > > >  logging = ringbuf:size=NBYTES
> > > > > > 
> > > > > > Fwiw, the ringbuffer is of course only allocated if the
> > > > > > backend is
> > > > > > used.
> > > > > 
> > > > > This looks really interesting. Question -- would you be fine if
> > > > > I
> > > > > wrote a smbcontrol to grab the log, similar to for example
> > > > > pool-usage?
> > > > 
> > > > great idea! If you're not going to do it, I will. Let's see who's
> > > > faster. :)
> > > 
> > > me. :)
> > > 
> > > Updated patchset attached. While I was at it I found it worthwhile
> > > improving
> > > messaging.idl and pidl to use and add support for mixed
> > > with/without value
> > > enums. What do you think?
> > 
> > Oh this looks *really* useful ! Thanks. I'll try and review shortly..
> 
> (bikeshedding) perhaps we should also add this to print during the
> panic?

nah, I don't think so. As Dumbledore puts it: he who is really in desperate need
of this, will know how to use it. :)

But if others like this idea as well, I can add code to do this.

Cheerio!
-slow