[PATCH] LMDB full patch set
Andrew Bartlett
abartlet at samba.org
Fri May 18 19:14:31 UTC 2018
On Fri, 2018-05-18 at 13:13 +0200, Stefan Metzmacher wrote:
> Am 18.05.2018 um 10:34 schrieb Andrew Bartlett via samba-technical:
> > On Thu, 2018-05-17 at 16:15 +1200, Garming Sam wrote:
> > > I suspect Joe's concurrent stress-testing would tell us more about how
> > > it behaves under real load -- we definitely should block less now,
> > > although not as little as LMDB fully allows due to other requirements.
> >
> > Just to follow up on this with some numbers from an internal thread.
> >
> > We got a load of 167 (I think operations / second) on lmdb, essentially
> > the same as the 160 we saw on master. Windows 2012R2 gives 235 on the
> > same virtual hardware, for reference.
> >
> > So no massive gain, but no massive loss either. This will likely
> > change with larger user sets (I don't have the details to hand as to
> > what the DB size was).
>
> Thanks! I also compared autobuild times and got similar results.
>
> One thing I was wondering about is if we should make use of virtual
> databases within lmdb.
> I'm not asking to do that before we can push this, but we should
> at least have ways for a seamless change in future. The compatibility
> flags at the samba_dsdb level are not a valid way to detect this!
> We need this at the ldb_lmdb layer.
Yes, we have had many interesting ideas about ways we can properly
leverage the virtual databases. We certainly could solve a lot of
problems that way, and (for example) if we use a sub-db per index we
gain the ability to do >= searches but walking the keys in-order.
I agree, we shouldn't block on this. Attached and included in the
branch is a simple patch to recognise this probable future and abort on
downgrade.
Andrew Bartlett
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
-------------- next part --------------
From 558ec1190dd81b52798e87562f0f5ecdd3d40773 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sat, 19 May 2018 07:10:15 +1200
Subject: [PATCH] ldb: Reject a possible future ldb_mdb with the index in a
sub-database
This ensures we do not corrupt such an index by making changes to the
main database without knowing that the index values are now in a
sub-database.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
lib/ldb/ldb_tdb/ldb_cache.c | 16 +++++++++++++++-
lib/ldb/ldb_tdb/ldb_tdb.h | 9 +++++++++
lib/ldb/tests/python/index.py | 32 ++++++++++++++++++++++++++++++++
3 files changed, 56 insertions(+), 1 deletion(-)
diff --git a/lib/ldb/ldb_tdb/ldb_cache.c b/lib/ldb/ldb_tdb/ldb_cache.c
index a9d2113e707..018a1ee0990 100644
--- a/lib/ldb/ldb_tdb/ldb_cache.c
+++ b/lib/ldb/ldb_tdb/ldb_cache.c
@@ -238,7 +238,7 @@ static int ltdb_index_load(struct ldb_module *module,
{
struct ldb_context *ldb = ldb_module_get_ctx(module);
struct ldb_dn *indexlist_dn;
- int r;
+ int r, lmdb_subdb_version;
if (ldb->schema.index_handler_override) {
/*
@@ -291,6 +291,20 @@ static int ltdb_index_load(struct ldb_module *module,
= ldb_msg_find_attr_as_string(ltdb->cache->indexlist,
LTDB_IDX_DN_GUID, NULL);
+ lmdb_subdb_version
+ = ldb_msg_find_attr_as_int(ltdb->cache->indexlist,
+ LTDB_IDX_LMDB_SUBDB, 0);
+
+ if (lmdb_subdb_version != 0) {
+ ldb_set_errstring(ldb,
+ "FATAL: This ldb_mdb database has "
+ "been written in a new verson of LDB "
+ "using a sub-database index that "
+ "is not understood by ldb "
+ LDB_VERSION);
+ return -1;
+ }
+
return 0;
}
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index dfc0d5e3299..46af8a1d836 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -120,6 +120,15 @@ struct ltdb_reindex_context {
#define LTDB_IDXDN "@IDXDN"
#define LTDB_IDXGUID "@IDXGUID"
#define LTDB_IDX_DN_GUID "@IDX_DN_GUID"
+
+/*
+ * This will be used to indicate when a new, yet to be developed
+ * sub-database version of the indicies are in use, to ensure we do
+ * not load future databases unintentionally.
+ */
+
+#define LTDB_IDX_LMDB_SUBDB "@IDX_LMDB_SUBDB"
+
#define LTDB_BASEINFO "@BASEINFO"
#define LTDB_OPTIONS "@OPTIONS"
#define LTDB_ATTRIBUTES "@ATTRIBUTES"
diff --git a/lib/ldb/tests/python/index.py b/lib/ldb/tests/python/index.py
index 3379fb9374f..2613a4dbedd 100755
--- a/lib/ldb/tests/python/index.py
+++ b/lib/ldb/tests/python/index.py
@@ -1291,6 +1291,38 @@ class MaxIndexKeyLengthTestsLmdb(MaxIndexKeyLengthTests):
def tearDown(self):
super(MaxIndexKeyLengthTestsLmdb, self).tearDown()
+
+# Run the index truncation tests against an lmdb backend
+class RejectSubDBIndex(LdbBaseTest):
+
+ def setUp(self):
+ self.prefix = MDB_PREFIX
+ super(RejectSubDBIndex, self).setUp()
+ self.testdir = tempdir()
+ self.filename = os.path.join(self.testdir,
+ "reject_subidx_test.ldb")
+ self.l = ldb.Ldb(self.url(),
+ options=[
+ "modules:rdn_name"])
+
+ def tearDown(self):
+ super(RejectSubDBIndex, self).tearDown()
+
+ def test_try_subdb_index(self):
+ try:
+ self.l.add({"dn": "@INDEXLIST",
+ "@IDX_LMDB_SUBDB": [b"1"],
+ "@IDXONE": [b"1"],
+ "@IDXONE": [b"1"],
+ "@IDXGUID": [b"objectUUID"],
+ "@IDX_DN_GUID": [b"GUID"],
+ })
+ except ldb.LdbError as e:
+ code = e.args[0]
+ string = e.args[1]
+ self.assertEqual(ldb.ERR_OPERATIONS_ERROR, code)
+ self.assertIn("sub-database index", string)
+
if __name__ == '__main__':
import unittest
unittest.TestProgram()
--
2.14.3
More information about the samba-technical
mailing list