[WIP] TDB traverse lock changes for massive AD DC perf improvement

[Andrew Bartlett]

Garming recently discovered a bug in ldb_tdb, causing us not to hold
the allrecord lock in TDB after a lock/unlock cycle.  

The upshot of that is that a tdb traverse will do many more locks, but
still be protected against modification during the traverse by the
transaction lock.

However, testing this patch showed a lock ordering issue, triggered
EDEADLK, because if one process holds the allrecord lock, and the the
transaction lock (for the traverse), while the other process holds the
transaction lock (from a prepare commit), and tries to get the
allrecord lock (the finish the prepare commit), it will report a
deadlock in the prepare commit. 

We tried to change the ltdb code to simply obtain a transaction lock
first, but the tdb transaction lock is not exposed in an API.  

However, by swapping the traverse to the allrecord lock, we remove the
second lock that is the heart of the ordering issue.

I realise that while unimportant to the AD DC case (all modifies in a
transaction), I have traded off concurrency for performance.  Can
someone explain why tdb_traverse_read() needs a lock at all?

Attached is the WIP patch and test modifications, as well as a PDF of
the performance improvement. 

The branch is ldap-multi-process in the the Catalyst GIT server. 

Thoughts most welcome.

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tdb-perf-wip.patch
Type: text/x-patch
Size: 17274 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170331/0cf962b2/tdb-perf-wip.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: samba-tdb-locking-perf.pdf
Type: application/pdf
Size: 57170 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170331/0cf962b2/samba-tdb-locking-perf.pdf>