LMDB key value backend for ldb_tdb (to be renamed ldb_key_val)

[William Brown]

On Thu, 2018-02-22 at 11:45 +1300, Garming Sam via samba-technical
wrote:
> Hi,
> 
> This is our current set of patches for implementing an LMDB based
> backend for LDB. The work is based on a prototype I wrote around this
> time last year inspired by Jakub's efforts. In saying that, the
> approach
> I took was completely different. The idea was to refactor ldb_tdb to
> be
> agnostic about which database backend was being used. The advantage
> has
> been quite minimal amount of code required to implement a functional
> 64-bit database backend. Many of the performance optimizations made
> for
> ldb_tdb can simply be reused, but conversely, for now we have
> deferred
> re-thinking the overall architecture e.g. consolidating the
> partitions
> into a single file using LMDB sub databases.

There are some arguments both ways here I think.

Keeping them seperate means that you can easily identify any component
that has a size blow out (specific index, changelog etc), and
potentially resolve it. This is pretty nice for admins to be able to
deal with. It also makes it easier to export/import large files to
resize them (lmdb files can't be resized iirc). If you have a combined
DB and it's say ... 8GB, then you have to have at least 8GB free to
export and import to a new db. If this was spilt to many files (say
0.5GB each) it becomes easier to manage this. 

However, joining these would be good for simpler transaction
managament, especially to prevent deadlocking if tables are locked out-
of-order. 

Provided you are really careful about transaction managament - for
example, if you take a write txn, you take a write txn on ALL mdb
files, you can "emulate" the behaviour of a single backend and gain
most of it's benefits, but you do pay an overhead to manage all those
transactions - and the cleanup of the reads when you close these. 

Hope my vague comments help,