LMDB key value backend for ldb_tdb (to be renamed ldb_key_val)

[William Brown]

On Tue, 2018-04-17 at 12:21 +1200, Garming Sam via samba-technical
wrote:
> On 17/04/18 01:11, Howard Chu via samba-technical wrote:
> > From: William Brown <william at blackhats.net.au>
> > > > > re-thinking the overall architecture e.g. consolidating the
> > > > > partitions
> > > > > into a single file using LMDB sub databases.
> > > 
> > > There are some arguments both ways here I think.
> > > 
> > > Keeping them seperate means that you can easily identify any
> > > component
> > > that has a size blow out (specific index, changelog etc), and
> > > potentially resolve it. This is pretty nice for admins to be able
> > > to
> > > deal with. It also makes it easier to export/import large files
> > > to
> > > resize them (lmdb files can't be resized iirc). If you have a
> > > combined
> > > DB and it's say ... 8GB, then you have to have at least 8GB free
> > > to
> > > export and import to a new db. If this was spilt to many files
> > > (say
> > > 0.5GB each) it becomes easier to manage this.
> > 
> > mdb_stat will tell you the sizes of each subDB. You can use
> > "mdb_dump
> > -s" to export a specific subDB. And "mdb_drop -s" to delete a
> > specific
> > subDB. Not sure what you mean about "LMDB files can't be resized" -
> > sure they can, that's what mdb_env_set_mapsize() does.
> > 
> 
> From Gary attempting an automatic resize mechanism, the caveats
> associated with mdb_env_set_mapsize() made it not straightforward to
> use, at least within the context of LDB. I presume William ran into
> similar complications. Right now, we've only used a fixed figure, but
> with the next iteration of our work, we'll probably be looking at how
> to
> get this right.

I've seen a number of sites get into a bit of an issue when they hit
the max size, and resizing didn't work without export/imports. I really
really just need to find some time to stress test mdb's edge cases so I
can give a report to Howard.

> 
> 
> Cheers,
> 
> Garming
> 
> > > However, joining these would be good for simpler transaction
> > > managament, especially to prevent deadlocking if tables are
> > > locked out-
> > > of-order. 
> 
>