[WIP][PATCH] Safe LDB locking for better replication

[Andrew Bartlett]

On Fri, 2017-06-16 at 15:13 +0200, Stefan Metzmacher wrote:
> Am 16.06.2017 um 06:55 schrieb Andrew Bartlett:
> > On Thu, 2017-06-15 at 16:58 +1200, Andrew Bartlett via samba-technical
> > wrote:
> > > On Wed, 2017-06-14 at 22:02 +1200, Andrew Bartlett via samba-
> > > technical
> > > wrote:
> > > > > Further comments most welcome, and I plan to re-write it to use
> > > > > read_lock() and read_unlock() operations tomorrow. 
> > > 
> > > Attached is the patch set against master
> > > 
> > > As mentioned previously, my TODO list is:
> > >  - fix tests not to hang when failing
> > 
> > They don't seem to hang, at least when reverting the whole-db lock.
> > 
> > >  - add test for the talloc_free(req) unlock case
> > 
> > I've added such a test.
> > 
> > >  - add some kind of test for the partitions locking 
> > 
> > I've added a test for this.
> > 
> > >  - a dbcheck rule for out of date @ATTRIBUTES and @INDEXLIST (rather
> > > than a runtime check)
> > 
> > This is still TODO.
> > 
> > With the proviso that I need that dbcheck rule (or drop this aspect of
> > the patches), can I please get your review on this series.  I'll run
> > some more builds, but I'm pretty confident on the patch series at this
> > point and would like to get this into master as soon as you are
> > comfortable. 
> > 
> > It would be good to have at least a week of this in master before rc1
> > freezes, so we can deal with any unexpected fallout, despite our
> > thorough testing.
> 
> I've pushed the first bunch of patches.
> 
> You can also push the attaches two patches (which I split from one
> single commit that looked strange).

I agree, it was strange due to the dev history, and you were right to
pick as not ready.  I'll do that once I sort out a dbcheck rule and
explain that the change in behaviour is highly desired.

It seems likely that we will want the behaviour change, which is not to
rewrite @ATTRIBUTES automatically any more, as currently a 4.7 upgrade
will force a re-index, and at scale this can be very slow.  (The re-
index is to add the tag to show that the the feature-support is
supported). 

Anyway, that is a task for Monday :-)

> Are there any other pending ldb/tdb changes?

No.

> I'll look at the rest next week.

Thanks!  

I can thankfully advise that 5/8 autobuilds I ran on this my tree
passed last night, with 2 of those being the messaging crash that there
is already a fix for.  This is not an unknown pass rate, but is much
better than we have seen for a while.

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba