[PATCH] LMDB full patch set

[Andrew Bartlett]

On Tue, 2018-05-22 at 17:41 +0200, Stefan Metzmacher via samba-
technical wrote:
> Am 22.05.2018 um 09:42 schrieb Andrew Bartlett via samba-technical:
> > On Sat, 2018-05-19 at 07:14 +1200, Andrew Bartlett via samba-technical
> > wrote:
> > > On Fri, 2018-05-18 at 13:13 +0200, Stefan Metzmacher wrote:
> > > > 
> > > > Thanks! I also compared autobuild times and got similar results.
> > > > 
> > > > One thing I was wondering about is if we should make use of virtual
> > > > databases within lmdb. 
> > > > I'm not asking to do that before we can push this, but we should
> > > > at least have ways for a seamless change in future. The compatibility
> > > > flags at the samba_dsdb level are not a valid way to detect this!
> > > > We need this at the ldb_lmdb layer.
> > > Yes, we have had many interesting ideas about ways we can properly
> > > leverage the virtual databases.  We certainly could solve a lot of
> > > problems that way, and (for example) if we use a sub-db per index we
> > > gain the ability to do >= searches but walking the keys in-order.
> > > 
> > > I agree, we shouldn't block on this.  Attached and included in the
> > > branch is a simple patch to recognise this probable future and abort on
> > > downgrade.
> > 
> > G'Day,
> > 
> > Having addressed all the various issues raised over the past weeks and
> > months, I would now like to push this to master.  
> > 
> > There will always be things to improve, but we will learn those best
> > when we have it available for use. 
> 
> Yes. One thing I tought about is starting with a small
> map_size and handle MDB_MAP_FULL and MDB_MAP_RESIZED.
> We could change the size in steps similar to tdb.
> That would remove the 32bit problem and avoids huge number in
> the 'top' output.

Yeah, that is also on a list of LMDB improvements we had been tossing
around the office.  I'll ensure these are listed somewhere we can build
in common today (eg the wiki). 

It would allow us to force allocation of the DB size so DB operation
can't run out of disk (currently the DB is sparse).  We don't have the
SIGSEGV issue TDB would have (avoids by allocating) because LMDB uses
write(), but it is a sharp edge I'm hoping to improve.

> > I would like to push this in the next day or so.  
> 
> I've removed some leading tabs/whitespaces
> and improved the commit message for ldb 1.4.0, see
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/master4-lmdb-full
> 
> Should we mark lmdb as experimental in the release commit message?

I think so.  I'll do that and push later today.  

When I write the WHATSNEW.txt I'll also include clear language on this
being new. 

> I also noticed that autobuilds with mdb are a little bit slower.
> I compared
> https://git.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=9b73105d902f
> https://git.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=95cfac07dc36
> https://git.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=0fd1ca4b8b33
> 
> "autobuild-private.sh samba" on the same host (at the same time)
> with mdb we use about 3 mins more of about 160 mins.
> and the same with "autobuild-private samba-ad-dc" with less difference.
> But in all cases autobuild finished earlier when not using mdb.

This is expected, as LMDB uses write() which goes via socket_wrapper. 

To avoid this confounding effect, Garming's test results were on a
branch that used user namespaces to replace socket_wrapper, while Joe's
were on a build on a bare VM. 

Thanks!

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba