[PATCH] GUID index for LDB

[Andrew Bartlett]

On Fri, 2017-09-01 at 18:04 +1200, Andrew Bartlett via samba-technical
wrote:
> On Tue, 2017-08-29 at 17:20 +1200, Andrew Bartlett via samba-technical
> wrote:
> > On Tue, 2017-08-22 at 16:41 +1200, Andrew Bartlett via samba-
> > technical
> > wrote:
> > > 
> > > Finally, my GUID-index branch has been updated.  More work on
> > > tests,
> > > both new and existing will keep me busy for another week, but I'm
> > > pretty pleased with how it is working out.
> > 
> > I have continued to work on this.  
> 
> I'm continuing to test this branch.  There are some issues with other
> parts of Samba that worked due to DB ordering assumptions.  However
> overall, I think it is good.
> 
> The only other news is that while upgrades will be seamless, downgrades
> will require running a script (which I'll write) or modification of the
> @INDEXLIST on all the backend DBs.  We don't support a seamless
> downgrade.  
> 
> I'll do some performance work on it over the weekend.  The previous
> tests show a small performance cost (5% or so) on searches, and a big
> performance benefit (%40) on adding users. 

I've also run (on my workstation) the test to add users and add them
into a group in a 2-hour time limit, with fsync() disabled.

I was able to add almost 100k (99847) users in 2 hours, including
adding them to 4 groups, which is now the major factor.  (The previous
record was 85,000).

Currently the biggest bottleneck to LDB performance is not DB
technology, but simply the way we pack and unpack an LDB record!

BTW, the resulting database is only 860MB, significantly because the
new GUID index code produces a smaller index.

> I'll e-mail Tridge/Douglas/Catalyst for permission on a re-licence to
> LGPLv3 the binary search macros next week.
> 
> I know it is a massive patch set, but some feedback would be helpful.
> 
> http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/GUID-index-6
> 
Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba