LMDB key value backend for ldb_tdb (to be renamed ldb_key_val)
Garming Sam
garming at catalyst.net.nz
Wed Feb 21 22:45:08 UTC 2018
Hi,
This is our current set of patches for implementing an LMDB based
backend for LDB. The work is based on a prototype I wrote around this
time last year inspired by Jakub's efforts. In saying that, the approach
I took was completely different. The idea was to refactor ldb_tdb to be
agnostic about which database backend was being used. The advantage has
been quite minimal amount of code required to implement a functional
64-bit database backend. Many of the performance optimizations made for
ldb_tdb can simply be reused, but conversely, for now we have deferred
re-thinking the overall architecture e.g. consolidating the partitions
into a single file using LMDB sub databases.
Currently this backend must have keys restricted to less than 511 bytes,
which is fine for our new GUID indexing scheme, but can run into issues
with our indexes. Gary is currently working on using truncated keys to
bypass this limit.
This current set of patches passes autobuild while still running with
the TDB backend. We have patches to pass the testsuite using the LMDB
backend, but a few of them still need tidying up and have been omitted
here for now.
Performance numbers at this point seem a bit tricky to obtain. Our
existing perf testing infrastructure relies on the test environment,
however, LMDB appears to run noticeably slower under cwrap due to some
calls being intercepted. Basic testing indicates better baseline figures
and better concurrency (reads no longer block writes, and writes
effectively only block reads during commit - not prepare commit), but
there needs to be a lot more testing to properly understand the
performance characteristics.
Somewhat related, I am currently investigating running our tests without
socket wrapper by using network namespaces. There are still areas where
being able to run socket wrapper is definitely useful, but performance
testing is definitely not one of them.
Noteworthy fixes required or bugs found:
- metadata.tdb needs to be committed last during a transaction
(getncchanges tests were causing replication errors as reads occurred
between the commit of metadata.tdb and the rest of the partitions).
- schema loading during a read lock needs a cached value (as writes can
happen during reads, long running read-locked operations could read new
metadata.tdb values).
Any thoughts or comments would be well appreciated. There is definitely
more to come in this space and using LMDB allows us to effectively
implement a number of further improvements like indexing for >= which
would make replication much faster.
Cheers,
Garming
-------------- next part --------------
From 371e10b98fafa6b07b55fbcc3f3dca8beb77c7e3 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 1 Feb 2018 17:16:13 +1300
Subject: [PATCH 01/34] partition: Use a transaction to write and a read lock
to read the LDB_METADATA_SEQ_NUM
This is critical as otherwise we can read a sequence number in advance of the data
that it represents and so have a false cache.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source4/dsdb/samdb/ldb_modules/partition.c | 12 +++---
.../dsdb/samdb/ldb_modules/partition_metadata.c | 48 +++++++++++++++++++---
2 files changed, 49 insertions(+), 11 deletions(-)
diff --git a/source4/dsdb/samdb/ldb_modules/partition.c b/source4/dsdb/samdb/ldb_modules/partition.c
index 426fce3..2cb05f9 100644
--- a/source4/dsdb/samdb/ldb_modules/partition.c
+++ b/source4/dsdb/samdb/ldb_modules/partition.c
@@ -861,7 +861,7 @@ static int partition_rename(struct ldb_module *module, struct ldb_request *req)
}
/* start a transaction */
-static int partition_start_trans(struct ldb_module *module)
+int partition_start_trans(struct ldb_module *module)
{
int i;
int ret;
@@ -923,7 +923,7 @@ static int partition_start_trans(struct ldb_module *module)
}
/* prepare for a commit */
-static int partition_prepare_commit(struct ldb_module *module)
+int partition_prepare_commit(struct ldb_module *module)
{
unsigned int i;
struct partition_private_data *data = talloc_get_type(ldb_module_get_private(module),
@@ -960,7 +960,7 @@ static int partition_prepare_commit(struct ldb_module *module)
/* end a transaction */
-static int partition_end_trans(struct ldb_module *module)
+int partition_end_trans(struct ldb_module *module)
{
int ret, ret2;
unsigned int i;
@@ -1006,7 +1006,7 @@ static int partition_end_trans(struct ldb_module *module)
}
/* delete a transaction */
-static int partition_del_trans(struct ldb_module *module)
+int partition_del_trans(struct ldb_module *module)
{
int ret, final_ret = LDB_SUCCESS;
unsigned int i;
@@ -1205,7 +1205,7 @@ static int partition_sequence_number(struct ldb_module *module, struct ldb_reque
}
/* lock all the backends */
-static int partition_read_lock(struct ldb_module *module)
+int partition_read_lock(struct ldb_module *module)
{
int i;
int ret;
@@ -1309,7 +1309,7 @@ static int partition_read_lock(struct ldb_module *module)
}
/* unlock all the backends */
-static int partition_read_unlock(struct ldb_module *module)
+int partition_read_unlock(struct ldb_module *module)
{
int i;
int ret = LDB_SUCCESS;
diff --git a/source4/dsdb/samdb/ldb_modules/partition_metadata.c b/source4/dsdb/samdb/ldb_modules/partition_metadata.c
index 3c5180b..3ba7905 100644
--- a/source4/dsdb/samdb/ldb_modules/partition_metadata.c
+++ b/source4/dsdb/samdb/ldb_modules/partition_metadata.c
@@ -319,13 +319,33 @@ int partition_metadata_init(struct ldb_module *module)
goto end;
}
+ ret = partition_start_trans(module);
+ if (ret != LDB_SUCCESS) {
+ talloc_free(data->metadata);
+ data->metadata = NULL;
+ }
+
ret = partition_metadata_set_sequence_number(module);
if (ret != LDB_SUCCESS) {
talloc_free(data->metadata);
data->metadata = NULL;
+ partition_del_trans(module);
+ } else {
+
+ ret = partition_prepare_commit(module);
+ if (ret != LDB_SUCCESS) {
+ talloc_free(data->metadata);
+ data->metadata = NULL;
+ } else {
+ ret = partition_end_trans(module);
+ if (ret != LDB_SUCCESS) {
+ talloc_free(data->metadata);
+ data->metadata = NULL;
+ }
+ }
}
-end:
+ end:
return ret;
}
@@ -335,10 +355,28 @@ end:
*/
int partition_metadata_sequence_number(struct ldb_module *module, uint64_t *value)
{
- return partition_metadata_get_uint64(module,
- LDB_METADATA_SEQ_NUM,
- value,
- 0);
+
+ /* We have to lock all the databases as otherwise we can
+ * return a sequence number that is higher than the DB values
+ * that we can see, as those transactions close after the
+ * metadata.tdb transaction closes */
+ int ret = partition_read_lock(module);
+ if (ret != LDB_SUCCESS) {
+ return ret;
+ }
+
+ ret = partition_metadata_get_uint64(module,
+ LDB_METADATA_SEQ_NUM,
+ value,
+ 0);
+ if (ret == LDB_SUCCESS) {
+ ret = partition_read_unlock(module);
+ } else {
+ /* Don't overwrite the error code */
+ partition_read_unlock(module);
+ }
+ return ret;
+
}
--
1.9.1
From 79d99c4606a9100a18723422bc58e712ed36abfd Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Fri, 2 Feb 2018 12:05:27 +1300
Subject: [PATCH 02/34] schema: Do not read different schema sequence values
during a read transaction
During a read lock, we find ourselves seeing an unchanged schema, but
reading any updates to the metadata.tdb (in the case of lmdb, where
reads do not block writes).
The alternative is to read-lock the entire metadata.tdb, however, this
allows more concurrency by allowing reads not to block writes.
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
source4/dsdb/samdb/ldb_modules/schema_load.c | 86 ++++++++++++++++++++++++++--
1 file changed, 80 insertions(+), 6 deletions(-)
diff --git a/source4/dsdb/samdb/ldb_modules/schema_load.c b/source4/dsdb/samdb/ldb_modules/schema_load.c
index 0bbd5fc..f1a01d7 100644
--- a/source4/dsdb/samdb/ldb_modules/schema_load.c
+++ b/source4/dsdb/samdb/ldb_modules/schema_load.c
@@ -36,8 +36,10 @@
#include "system/filesys.h"
struct schema_load_private_data {
struct ldb_module *module;
- bool in_transaction;
+ uint64_t in_transaction;
+ uint64_t in_read_transaction;
struct tdb_wrap *metadata;
+ uint64_t schema_seq_num_read_lock;
uint64_t schema_seq_num_cache;
int tdb_seqnum;
};
@@ -193,7 +195,7 @@ static struct dsdb_schema *dsdb_schema_refresh(struct ldb_module *module, struct
return schema;
}
- if (private_data->in_transaction) {
+ if (private_data->in_transaction > 0) {
/*
* If the refresh is not an expected part of a larger
@@ -221,7 +223,19 @@ static struct dsdb_schema *dsdb_schema_refresh(struct ldb_module *module, struct
* continue to hit the database to get the highest USN.
*/
- ret = schema_metadata_get_uint64(private_data, DSDB_METADATA_SCHEMA_SEQ_NUM, &schema_seq_num, 0);
+ if (private_data->in_read_transaction > 0) {
+ /*
+ * We must give a static value of the metadata sequence number
+ * during a read lock, otherwise, we will fail to load the
+ * schema at runtime.
+ */
+ schema_seq_num = private_data->schema_seq_num_read_lock;
+ ret = LDB_SUCCESS;
+ } else {
+ ret = schema_metadata_get_uint64(private_data,
+ DSDB_METADATA_SCHEMA_SEQ_NUM,
+ &schema_seq_num, 0);
+ }
if (schema != NULL) {
if (ret == LDB_SUCCESS) {
@@ -564,7 +578,7 @@ static int schema_load_start_transaction(struct ldb_module *module)
"schema_load_init: dsdb_get_schema failed");
return LDB_ERR_OPERATIONS_ERROR;
}
- private_data->in_transaction = true;
+ private_data->in_transaction++;
return ldb_next_start_trans(module);
}
@@ -573,8 +587,14 @@ static int schema_load_end_transaction(struct ldb_module *module)
{
struct schema_load_private_data *private_data =
talloc_get_type(ldb_module_get_private(module), struct schema_load_private_data);
+ struct ldb_context *ldb = ldb_module_get_ctx(module);
- private_data->in_transaction = false;
+ if (private_data->in_transaction == 0) {
+ ldb_debug_set(ldb, LDB_DEBUG_FATAL,
+ "schema_load_end_transaction: transaction mismatch");
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+ private_data->in_transaction--;
return ldb_next_end_trans(module);
}
@@ -583,8 +603,14 @@ static int schema_load_del_transaction(struct ldb_module *module)
{
struct schema_load_private_data *private_data =
talloc_get_type(ldb_module_get_private(module), struct schema_load_private_data);
+ struct ldb_context *ldb = ldb_module_get_ctx(module);
- private_data->in_transaction = false;
+ if (private_data->in_transaction == 0) {
+ ldb_debug_set(ldb, LDB_DEBUG_FATAL,
+ "schema_load_del_transaction: transaction mismatch");
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+ private_data->in_transaction--;
return ldb_next_del_trans(module);
}
@@ -618,6 +644,52 @@ static int schema_load_extended(struct ldb_module *module, struct ldb_request *r
return ldb_next_request(module, req);
}
+static int schema_read_lock(struct ldb_module *module)
+{
+ struct schema_load_private_data *private_data =
+ talloc_get_type(ldb_module_get_private(module), struct schema_load_private_data);
+ uint64_t schema_seq_num = 0;
+
+ if (private_data == NULL) {
+ return ldb_next_read_lock(module);
+ }
+
+ if (private_data->in_transaction == 0 &&
+ private_data->in_read_transaction == 0) {
+ /*
+ * This appears to fail during the init path, so do not bother
+ * checking the return, and return 0 (reload schema).
+ */
+ schema_metadata_get_uint64(private_data,
+ DSDB_METADATA_SCHEMA_SEQ_NUM,
+ &schema_seq_num, 0);
+
+ private_data->schema_seq_num_read_lock = schema_seq_num;
+ }
+ private_data->in_read_transaction++;
+
+ return ldb_next_read_lock(module);
+
+}
+
+static int schema_read_unlock(struct ldb_module *module)
+{
+ struct schema_load_private_data *private_data =
+ talloc_get_type(ldb_module_get_private(module), struct schema_load_private_data);
+
+ if (private_data == NULL) {
+ return ldb_next_read_unlock(module);
+ }
+
+ if (private_data->in_transaction == 0 &&
+ private_data->in_read_transaction == 1) {
+ private_data->schema_seq_num_read_lock = 0;
+ }
+ private_data->in_read_transaction--;
+
+ return ldb_next_read_unlock(module);
+}
+
static const struct ldb_module_ops ldb_schema_load_module_ops = {
.name = "schema_load",
@@ -627,6 +699,8 @@ static const struct ldb_module_ops ldb_schema_load_module_ops = {
.start_transaction = schema_load_start_transaction,
.end_transaction = schema_load_end_transaction,
.del_transaction = schema_load_del_transaction,
+ .read_lock = schema_read_lock,
+ .read_unlock = schema_read_unlock,
};
int ldb_schema_load_module_init(const char *version)
--
1.9.1
From b04b17a5d13717e37d7f954f16bc90afc39e23a9 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Wed, 7 Feb 2018 23:21:45 +1300
Subject: [PATCH 03/34] partition: Leave metadata.tdb unlocking until last
With the lmdb patches, I have cleanly observed the database being read
in between the commit of the metadata.tdb and the eventual commits of
the individual partitions.
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
source4/dsdb/samdb/ldb_modules/partition.c | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/source4/dsdb/samdb/ldb_modules/partition.c b/source4/dsdb/samdb/ldb_modules/partition.c
index 2cb05f9..422ed36 100644
--- a/source4/dsdb/samdb/ldb_modules/partition.c
+++ b/source4/dsdb/samdb/ldb_modules/partition.c
@@ -976,10 +976,6 @@ int partition_end_trans(struct ldb_module *module)
data->in_transaction--;
}
- ret2 = partition_metadata_end_trans(module);
- if (ret2 != LDB_SUCCESS) {
- ret = ret2;
- }
for (i=0; data && data->partitions && data->partitions[i]; i++) {
if ((module && ldb_module_flags(ldb_module_get_ctx(module)) & LDB_FLG_ENABLE_TRACING)) {
@@ -1002,6 +998,12 @@ int partition_end_trans(struct ldb_module *module)
if (ret2 != LDB_SUCCESS) {
ret = ret2;
}
+
+ ret2 = partition_metadata_end_trans(module);
+ if (ret2 != LDB_SUCCESS) {
+ ret = ret2;
+ }
+
return ret;
}
@@ -1012,10 +1014,6 @@ int partition_del_trans(struct ldb_module *module)
unsigned int i;
struct partition_private_data *data = talloc_get_type(ldb_module_get_private(module),
struct partition_private_data);
- ret = partition_metadata_del_trans(module);
- if (ret != LDB_SUCCESS) {
- final_ret = ret;
- }
for (i=0; data && data->partitions && data->partitions[i]; i++) {
if ((module && ldb_module_flags(ldb_module_get_ctx(module)) & LDB_FLG_ENABLE_TRACING)) {
@@ -1044,6 +1042,12 @@ int partition_del_trans(struct ldb_module *module)
if (ret != LDB_SUCCESS) {
final_ret = ret;
}
+
+ ret = partition_metadata_del_trans(module);
+ if (ret != LDB_SUCCESS) {
+ final_ret = ret;
+ }
+
return final_ret;
}
--
1.9.1
From 827e2ec53dd14d2eb58fbe007f2cee8c60fb5173 Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Thu, 11 Jan 2018 14:27:40 +1300
Subject: [PATCH 04/34] Change name to sam.ldb to align with new assumptions
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
python/samba/tests/samba3sam.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/samba/tests/samba3sam.py b/python/samba/tests/samba3sam.py
index 929523b..bfc7932 100644
--- a/python/samba/tests/samba3sam.py
+++ b/python/samba/tests/samba3sam.py
@@ -73,7 +73,7 @@ class MapBaseTestCase(TestCaseInTempDir):
def make_s4dn(basedn, rdn):
return "%s,%s" % (rdn, basedn)
- self.ldbfile = os.path.join(self.tempdir, "test.ldb")
+ self.ldbfile = os.path.join(self.tempdir, "sam.ldb")
self.ldburl = "tdb://" + self.ldbfile
tempdir = self.tempdir
--
1.9.1
From da1df145ba02921700d5df2a316c66966c712f2e Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Tue, 23 Jan 2018 11:02:28 +1300
Subject: [PATCH 05/34] ldb_mod_op_test: Fix core dump on
ldb_case_attrs_index_test_teardown
With no schema syntax, this would occasionally crash as it dereferenced
some possibly NULL sequence of memory.
Note: Removing all tests except this one, made it crash reliably.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/tests/ldb_mod_op_test.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/lib/ldb/tests/ldb_mod_op_test.c b/lib/ldb/tests/ldb_mod_op_test.c
index cf2288c..2ab1c2e 100644
--- a/lib/ldb/tests/ldb_mod_op_test.c
+++ b/lib/ldb/tests/ldb_mod_op_test.c
@@ -2587,6 +2587,14 @@ static void test_ldb_attrs_index_handler(void **state)
syntax, &cn_attr_2);
assert_int_equal(ret, LDB_SUCCESS);
+ syntax = ldb_standard_syntax_by_name(ldb, LDB_SYNTAX_OCTET_STRING);
+ assert_non_null(syntax);
+
+ ret = ldb_schema_attribute_fill_with_syntax(ldb, ldb,
+ "", 0,
+ syntax, &default_attr);
+ assert_int_equal(ret, LDB_SUCCESS);
+
/*
* Set an attribute handler
*/
--
1.9.1
From 3094bbe7376e9bcaeb5f2608746b8a287a96b77f Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Fri, 19 Jan 2018 09:16:04 +1300
Subject: [PATCH 06/34] remove_dc.py: Abort transaction before throwing an
exception
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
python/samba/remove_dc.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/python/samba/remove_dc.py b/python/samba/remove_dc.py
index f273a51..457b449 100644
--- a/python/samba/remove_dc.py
+++ b/python/samba/remove_dc.py
@@ -382,11 +382,13 @@ def remove_dc(samdb, logger, dc_name):
"(cn=%s))"
% ldb.binary_encode(dc_name))
except LdbError as (enum, estr):
+ samdb.transaction_cancel()
raise DemoteException("Failure checking if %s is an server "
"object in %s: "
% (dc_name, samdb.domain_dns_name()), estr)
if (len(server_msgs) == 0):
+ samdb.transaction_cancel()
raise DemoteException("%s is not an AD DC in %s"
% (dc_name, samdb.domain_dns_name()))
server_dn = server_msgs[0].dn
@@ -404,6 +406,7 @@ def remove_dc(samdb, logger, dc_name):
ntds_msgs = []
pass
else:
+ samdb.transaction_cancel()
raise DemoteException("Failure checking if %s is an NTDS DSA in %s: "
% (ntds_dn, samdb.domain_dns_name()), estr)
@@ -411,6 +414,7 @@ def remove_dc(samdb, logger, dc_name):
# object, just remove the server object located above
if (len(ntds_msgs) == 0):
if server_dn is None:
+ samdb.transaction_cancel()
raise DemoteException("%s is not an AD DC in %s"
% (dc_name, samdb.domain_dns_name()))
--
1.9.1
From f4b3da764769b1f75394fbb45a1a45eeea685c29 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Wed, 22 Nov 2017 12:37:07 +1300
Subject: [PATCH 07/34] schema_set: Add a missing newline between functions
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
source4/dsdb/schema/schema_set.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/source4/dsdb/schema/schema_set.c b/source4/dsdb/schema/schema_set.c
index 226e31e..8674757 100644
--- a/source4/dsdb/schema/schema_set.c
+++ b/source4/dsdb/schema/schema_set.c
@@ -50,6 +50,7 @@ const struct ldb_schema_attribute *dsdb_attribute_handler_override(struct ldb_co
}
return a->ldb_schema_attribute;
}
+
/*
* Set the attribute handlers onto the LDB, and potentially write the
* @INDEXLIST, @IDXONE and @ATTRIBUTES records. The @ATTRIBUTES records
--
1.9.1
From 32f841c66f6b0986ec759c3d46ac5065d25645e0 Mon Sep 17 00:00:00 2001
From: Bob Campbell <bobcampbell at catalyst.net.nz>
Date: Tue, 11 Jul 2017 16:40:14 +1200
Subject: [PATCH 08/34] samdb/schema_load: do schema loading with one search
It appears that there was a race condition between searching for the
attribute & class definitions, and searching for the schema object, if
the schema was changed in-between the two searches.
This is likely the cause of ldap_schema being flapping.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12889
Signed-off-by: Bob Campbell <bobcampbell at catalyst.net.nz>
---
source4/dsdb/samdb/ldb_modules/schema_load.c | 60 ++++++++++++++--------------
source4/dsdb/schema/schema_init.c | 23 ++++++++---
2 files changed, 48 insertions(+), 35 deletions(-)
diff --git a/source4/dsdb/samdb/ldb_modules/schema_load.c b/source4/dsdb/samdb/ldb_modules/schema_load.c
index f1a01d7..caaeb7b 100644
--- a/source4/dsdb/samdb/ldb_modules/schema_load.c
+++ b/source4/dsdb/samdb/ldb_modules/schema_load.c
@@ -296,20 +296,17 @@ static int dsdb_schema_from_db(struct ldb_module *module,
struct ldb_context *ldb = ldb_module_get_ctx(module);
TALLOC_CTX *tmp_ctx;
char *error_string;
- int ret;
+ int ret, i;
struct ldb_dn *schema_dn = ldb_get_schema_basedn(ldb);
- struct ldb_result *schema_res;
struct ldb_result *res;
- static const char *schema_head_attrs[] = {
- "prefixMap",
- "schemaInfo",
- "fSMORoleOwner",
- NULL
- };
+ struct ldb_message *schema_msg = NULL;
static const char *schema_attrs[] = {
DSDB_SCHEMA_COMMON_ATTRS,
DSDB_SCHEMA_ATTR_ATTRS,
DSDB_SCHEMA_CLASS_ATTRS,
+ "prefixMap",
+ "schemaInfo",
+ "fSMORoleOwner",
NULL
};
unsigned flags;
@@ -324,33 +321,18 @@ static int dsdb_schema_from_db(struct ldb_module *module,
ldb_set_flags(ldb, flags & ~LDB_FLG_ENABLE_TRACING);
/*
- * setup the prefix mappings and schema info
- */
- ret = dsdb_module_search_dn(module, tmp_ctx, &schema_res,
- schema_dn, schema_head_attrs,
- DSDB_FLAG_NEXT_MODULE, NULL);
- if (ret == LDB_ERR_NO_SUCH_OBJECT) {
- ldb_reset_err_string(ldb);
- ldb_debug(ldb, LDB_DEBUG_WARNING,
- "schema_load_init: no schema head present: (skip schema loading)\n");
- goto failed;
- } else if (ret != LDB_SUCCESS) {
- ldb_asprintf_errstring(ldb,
- "dsdb_schema: failed to search the schema head: %s",
- ldb_errstring(ldb));
- goto failed;
- }
-
- /*
- * load the attribute definitions.
+ * Load the attribute and class definitions, as well as
+ * the schema object. We do this in one search and then
+ * split it so that there isn't a race condition when
+ * the schema is changed between two searches.
*/
ret = dsdb_module_search(module, tmp_ctx, &res,
- schema_dn, LDB_SCOPE_ONELEVEL,
+ schema_dn, LDB_SCOPE_SUBTREE,
schema_attrs,
DSDB_FLAG_NEXT_MODULE |
DSDB_SEARCH_SHOW_DN_IN_STORAGE_FORMAT,
NULL,
- "(|(objectClass=attributeSchema)(objectClass=classSchema))");
+ "(|(objectClass=attributeSchema)(objectClass=classSchema)(objectClass=dMD))");
if (ret != LDB_SUCCESS) {
ldb_asprintf_errstring(ldb,
"dsdb_schema: failed to search attributeSchema and classSchema objects: %s",
@@ -358,8 +340,26 @@ static int dsdb_schema_from_db(struct ldb_module *module,
goto failed;
}
+ /*
+ * Separate the schema object from the attribute and
+ * class objects.
+ */
+ for (i = 0; i < res->count; i++) {
+ if (ldb_msg_find_element(res->msgs[i], "prefixMap")) {
+ schema_msg = res->msgs[i];
+ break;
+ }
+ }
+
+ if (schema_msg == NULL) {
+ ldb_asprintf_errstring(ldb,
+ "dsdb_schema load failed: failed to find prefixMap");
+ ret = LDB_ERR_NO_SUCH_ATTRIBUTE;
+ goto failed;
+ }
+
ret = dsdb_schema_from_ldb_results(tmp_ctx, ldb,
- schema_res, res, schema, &error_string);
+ schema_msg, res, schema, &error_string);
if (ret != LDB_SUCCESS) {
ldb_asprintf_errstring(ldb,
"dsdb_schema load failed: %s",
diff --git a/source4/dsdb/schema/schema_init.c b/source4/dsdb/schema/schema_init.c
index dbd5045..22ad599 100644
--- a/source4/dsdb/schema/schema_init.c
+++ b/source4/dsdb/schema/schema_init.c
@@ -904,10 +904,23 @@ int dsdb_load_ldb_results_into_schema(TALLOC_CTX *mem_ctx, struct ldb_context *l
char **error_string)
{
unsigned int i;
+ char *prefixMap;
+ WERROR status;
schema->ts_last_change = 0;
for (i=0; i < attrs_class_res->count; i++) {
- WERROR status = dsdb_schema_set_el_from_ldb_msg(ldb, schema, attrs_class_res->msgs[i]);
+ /*
+ * attrs_class_res also includes the schema object;
+ * we only want to process classes & attributes
+ */
+ prefixMap = ldb_msg_find_attr_as_string(
+ attrs_class_res->msgs[i],
+ "prefixMap", NULL);
+ if (prefixMap != NULL) {
+ continue;
+ }
+
+ status = dsdb_schema_set_el_from_ldb_msg(ldb, schema, attrs_class_res->msgs[i]);
if (!W_ERROR_IS_OK(status)) {
*error_string = talloc_asprintf(mem_ctx,
"dsdb_load_ldb_results_into_schema: failed to load attribute or class definition: %s:%s",
@@ -928,7 +941,7 @@ int dsdb_load_ldb_results_into_schema(TALLOC_CTX *mem_ctx, struct ldb_context *l
*/
int dsdb_schema_from_ldb_results(TALLOC_CTX *mem_ctx, struct ldb_context *ldb,
- struct ldb_result *schema_res,
+ struct ldb_message *schema_msg,
struct ldb_result *attrs_class_res,
struct dsdb_schema **schema_out,
char **error_string)
@@ -961,7 +974,7 @@ int dsdb_schema_from_ldb_results(TALLOC_CTX *mem_ctx, struct ldb_context *ldb,
false);
}
- prefix_val = ldb_msg_find_ldb_val(schema_res->msgs[0], "prefixMap");
+ prefix_val = ldb_msg_find_ldb_val(schema_msg, "prefixMap");
if (!prefix_val) {
*error_string = talloc_asprintf(mem_ctx,
"schema_fsmo_init: no prefixMap attribute found");
@@ -969,7 +982,7 @@ int dsdb_schema_from_ldb_results(TALLOC_CTX *mem_ctx, struct ldb_context *ldb,
talloc_free(tmp_ctx);
return LDB_ERR_CONSTRAINT_VIOLATION;
}
- info_val = ldb_msg_find_ldb_val(schema_res->msgs[0], "schemaInfo");
+ info_val = ldb_msg_find_ldb_val(schema_msg, "schemaInfo");
if (!info_val) {
status = dsdb_schema_info_blob_new(mem_ctx, &info_val_default);
if (!W_ERROR_IS_OK(status)) {
@@ -999,7 +1012,7 @@ int dsdb_schema_from_ldb_results(TALLOC_CTX *mem_ctx, struct ldb_context *ldb,
return ret;
}
- schema->fsmo.master_dn = ldb_msg_find_attr_as_dn(ldb, schema, schema_res->msgs[0], "fSMORoleOwner");
+ schema->fsmo.master_dn = ldb_msg_find_attr_as_dn(ldb, schema, schema_msg, "fSMORoleOwner");
if (ldb_dn_compare(samdb_ntds_settings_dn(ldb, tmp_ctx), schema->fsmo.master_dn) == 0) {
schema->fsmo.we_are_master = true;
} else {
--
1.9.1
From 8518b421bc1721e073eba830ac522ac32b6fd772 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Wed, 22 Nov 2017 12:36:57 +1300
Subject: [PATCH 09/34] schema_init: Reduce scope of variable
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
source4/dsdb/schema/schema_init.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source4/dsdb/schema/schema_init.c b/source4/dsdb/schema/schema_init.c
index 22ad599..a4969a4 100644
--- a/source4/dsdb/schema/schema_init.c
+++ b/source4/dsdb/schema/schema_init.c
@@ -904,11 +904,11 @@ int dsdb_load_ldb_results_into_schema(TALLOC_CTX *mem_ctx, struct ldb_context *l
char **error_string)
{
unsigned int i;
- char *prefixMap;
WERROR status;
schema->ts_last_change = 0;
for (i=0; i < attrs_class_res->count; i++) {
+ const char *prefixMap = NULL;
/*
* attrs_class_res also includes the schema object;
* we only want to process classes & attributes
--
1.9.1
From 5e10d94850f6277262cc63c6d7d7e0b44b222140 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Tue, 10 Jan 2017 19:05:40 +1300
Subject: [PATCH 10/34] ldb_tdb: Begin abstracting out the base key value
operations
This will allow us to change the backend from tdb to lmdb.
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_tdb.c | 36 ++++++++++++++++++++++++++++++++++++
lib/ldb/ldb_tdb/ldb_tdb.h | 12 ++++++++++++
2 files changed, 48 insertions(+)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 16e4b8e..4a35028 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -413,6 +413,16 @@ static int ltdb_modified(struct ldb_module *module, struct ldb_dn *dn)
return ret;
}
+static int ltdb_tdb_store(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags)
+{
+ return tdb_store(ltdb->tdb, key, data, flags);
+}
+
+static int ltdb_error(struct ltdb_private *ltdb)
+{
+ return ltdb_err_map(tdb_error(ltdb->tdb));
+}
+
/*
store a record into the db
*/
@@ -635,6 +645,16 @@ static int ltdb_add(struct ltdb_context *ctx)
return ret;
}
+static int ltdb_tdb_exists(struct ltdb_private *ltdb, TDB_DATA key)
+{
+ return tdb_exists(ltdb->tdb, key);
+}
+
+static int ltdb_tdb_delete(struct ltdb_private *ltdb, TDB_DATA tdb_key)
+{
+ return tdb_delete(ltdb->tdb, tdb_key);
+}
+
/*
delete a record from the database, not updating indexes (used for deleting
index records)
@@ -1671,6 +1691,21 @@ static void ltdb_handle_extended(struct ltdb_context *ctx)
ltdb_request_extended_done(ctx, ext, ret);
}
+static const char * ltdb_tdb_name(struct ltdb_private *ltdb)
+{
+ return tdb_name(ltdb->tdb);
+}
+
+static struct kv_db_ops key_value_ops = {
+ .store = ltdb_tdb_store,
+ .delete = ltdb_tdb_delete,
+ .exists = ltdb_tdb_exists,
+ .lock_read = ltdb_lock_read,
+ .unlock_read = ltdb_unlock_read,
+ .error = ltdb_error,
+ .name = ltdb_tdb_name,
+};
+
static void ltdb_callback(struct tevent_context *ev,
struct tevent_timer *te,
struct timeval t,
@@ -1934,6 +1969,7 @@ static int ltdb_connect(struct ldb_context *ldb, const char *url,
}
ltdb->sequence_number = 0;
+ ltdb->kv_ops = &key_value_ops;
module = ldb_module_new(ldb, ldb, "ldb_tdb backend", <db_ops);
if (!module) {
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 7e18249..f2a6e7a 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -4,9 +4,21 @@
#include "tdb.h"
#include "ldb_module.h"
+struct ltdb_private;
+struct kv_db_ops {
+ int (*store)(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags);
+ int (*delete)(struct ltdb_private *ltdb, TDB_DATA key);
+ int (*exists)(struct ltdb_private *ltdb, TDB_DATA key);
+ int (*lock_read)(struct ldb_module *);
+ int (*unlock_read)(struct ldb_module *);
+ int (*error)(struct ltdb_private *ltdb);
+ const char * (*name)(struct ltdb_private *ltdb);
+};
+
/* this private structure is used by the ltdb backend in the
ldb_context */
struct ltdb_private {
+ struct kv_db_ops *kv_ops;
TDB_CONTEXT *tdb;
unsigned int connect_flags;
--
1.9.1
From 6f62b4005b5d69293022c9b41cf8ff79e6b1372a Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Tue, 10 Jan 2017 20:45:02 +1300
Subject: [PATCH 11/34] ldb_tdb: Replace exists, name and error_map with key
value ops
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_index.c | 2 +-
lib/ldb/ldb_tdb/ldb_search.c | 2 +-
lib/ldb/ldb_tdb/ldb_tdb.c | 16 ++++++++--------
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_index.c b/lib/ldb/ldb_tdb/ldb_index.c
index f2fce42..0984760 100644
--- a/lib/ldb/ldb_tdb/ldb_index.c
+++ b/lib/ldb/ldb_tdb/ldb_index.c
@@ -2647,7 +2647,7 @@ int ltdb_reindex(struct ldb_module *module)
ldb_debug(ldb_module_get_ctx(module),
LDB_DEBUG_WARNING, "Reindexing: re_index successful on %s, "
"final index write-out will be in transaction commit",
- tdb_name(ltdb->tdb));
+ ltdb->kv_ops->name(ltdb));
}
return LDB_SUCCESS;
}
diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c
index 0af230f..fb618fc 100644
--- a/lib/ldb/ldb_tdb/ldb_search.c
+++ b/lib/ldb/ldb_tdb/ldb_search.c
@@ -256,7 +256,7 @@ int ltdb_search_key(struct ldb_module *module, struct ltdb_private *ltdb,
ltdb_parse_data_unpack, &ctx);
if (ret == -1) {
- ret = ltdb_err_map(tdb_error(ltdb->tdb));
+ ret = ltdb->kv_ops->error(ltdb);
if (ret == LDB_SUCCESS) {
/*
* Just to be sure we don't turn errors
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 4a35028..dbe9dc1 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -391,7 +391,7 @@ static int ltdb_modified(struct ldb_module *module, struct ldb_dn *dn)
if (ltdb->warn_reindex) {
ldb_debug(ldb_module_get_ctx(module),
LDB_DEBUG_ERROR, "Reindexing %s due to modification on %s",
- tdb_name(ltdb->tdb), ldb_dn_get_linearized(dn));
+ ltdb->kv_ops->name(ltdb), ldb_dn_get_linearized(dn));
}
ret = ltdb_reindex(module);
}
@@ -459,10 +459,10 @@ int ltdb_store(struct ldb_module *module, const struct ldb_message *msg, int flg
tdb_data.dptr = ldb_data.data;
tdb_data.dsize = ldb_data.length;
- ret = tdb_store(ltdb->tdb, tdb_key, tdb_data, flgs);
+ ret = ltdb->kv_ops->store(ltdb, tdb_key, tdb_data, flgs);
if (ret != 0) {
bool is_special = ldb_dn_is_special(msg->dn);
- ret = ltdb_err_map(tdb_error(ltdb->tdb));
+ ret = ltdb->kv_ops->error(ltdb);
/*
* LDB_ERR_ENTRY_ALREADY_EXISTS means the DN, not
@@ -682,11 +682,11 @@ int ltdb_delete_noindex(struct ldb_module *module,
return LDB_ERR_OTHER;
}
- ret = tdb_delete(ltdb->tdb, tdb_key);
+ ret = ltdb->kv_ops->delete(ltdb, tdb_key);
TALLOC_FREE(tdb_key_ctx);
if (ret != 0) {
- ret = ltdb_err_map(tdb_error(ltdb->tdb));
+ ret = ltdb->kv_ops->error(ltdb);
}
return ret;
@@ -1417,7 +1417,7 @@ static int ltdb_start_trans(struct ldb_module *module)
}
if (tdb_transaction_start(ltdb->tdb) != 0) {
- return ltdb_err_map(tdb_error(ltdb->tdb));
+ return ltdb->kv_ops->error(ltdb);
}
ltdb->in_transaction++;
@@ -1478,7 +1478,7 @@ static int ltdb_end_trans(struct ldb_module *module)
ltdb->prepared_commit = false;
if (tdb_transaction_commit(ltdb->tdb) != 0) {
- ret = ltdb_err_map(tdb_error(ltdb->tdb));
+ ret = ltdb->kv_ops->error(ltdb);
ldb_asprintf_errstring(ldb_module_get_ctx(module),
"Failure during tdb_transaction_commit(): %s -> %s",
tdb_errorstr(ltdb->tdb),
@@ -1498,7 +1498,7 @@ static int ltdb_del_trans(struct ldb_module *module)
if (ltdb_index_transaction_cancel(module) != 0) {
tdb_transaction_cancel(ltdb->tdb);
- return ltdb_err_map(tdb_error(ltdb->tdb));
+ return ltdb->kv_ops->error(ltdb);
}
tdb_transaction_cancel(ltdb->tdb);
--
1.9.1
From 9b01f05dd9109d6c19bfe043922deedf26735eff Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Tue, 10 Jan 2017 21:44:11 +1300
Subject: [PATCH 12/34] ldb_tdb: Replace tdb transaction code with generic key
value ones
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_cache.c | 6 ++++--
lib/ldb/ldb_tdb/ldb_tdb.c | 38 +++++++++++++++++++++++++++++++-------
lib/ldb/ldb_tdb/ldb_tdb.h | 4 ++++
3 files changed, 39 insertions(+), 9 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_cache.c b/lib/ldb/ldb_tdb/ldb_cache.c
index 5b90bd9..600b73f 100644
--- a/lib/ldb/ldb_tdb/ldb_cache.c
+++ b/lib/ldb/ldb_tdb/ldb_cache.c
@@ -415,7 +415,7 @@ int ltdb_cache_load(struct ldb_module *module)
/* possibly initialise the baseinfo */
if (r == LDB_ERR_NO_SUCH_OBJECT) {
- if (tdb_transaction_start(ltdb->tdb) != 0) {
+ if (ltdb->kv_ops->begin_write(ltdb) != 0) {
goto failed;
}
@@ -423,7 +423,9 @@ int ltdb_cache_load(struct ldb_module *module)
looking for the record again. */
ltdb_baseinfo_init(module);
- tdb_transaction_commit(ltdb->tdb);
+ if (ltdb->kv_ops->finish_write(ltdb) != 0) {
+ goto failed;
+ }
if (ltdb_search_dn1(module, baseinfo_dn, baseinfo, 0) != LDB_SUCCESS) {
goto failed;
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index dbe9dc1..5caeea9 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -1406,6 +1406,26 @@ static int ltdb_rename(struct ltdb_context *ctx)
return ret;
}
+static int ltdb_tdb_transaction_start(struct ltdb_private *ltdb)
+{
+ return tdb_transaction_start(ltdb->tdb);
+}
+
+static int ltdb_tdb_transaction_cancel(struct ltdb_private *ltdb)
+{
+ return tdb_transaction_cancel(ltdb->tdb);
+}
+
+static int ltdb_tdb_transaction_prepare_commit(struct ltdb_private *ltdb)
+{
+ return tdb_transaction_prepare_commit(ltdb->tdb);
+}
+
+static int ltdb_tdb_transaction_commit(struct ltdb_private *ltdb)
+{
+ return tdb_transaction_commit(ltdb->tdb);
+}
+
static int ltdb_start_trans(struct ldb_module *module)
{
void *data = ldb_module_get_private(module);
@@ -1416,7 +1436,7 @@ static int ltdb_start_trans(struct ldb_module *module)
return LDB_ERR_UNWILLING_TO_PERFORM;
}
- if (tdb_transaction_start(ltdb->tdb) != 0) {
+ if (ltdb->kv_ops->begin_write(ltdb) != 0) {
return ltdb->kv_ops->error(ltdb);
}
@@ -1439,13 +1459,13 @@ static int ltdb_prepare_commit(struct ldb_module *module)
ret = ltdb_index_transaction_commit(module);
if (ret != LDB_SUCCESS) {
- tdb_transaction_cancel(ltdb->tdb);
+ ltdb->kv_ops->abort_write(ltdb);
ltdb->in_transaction--;
return ret;
}
- if (tdb_transaction_prepare_commit(ltdb->tdb) != 0) {
- ret = ltdb_err_map(tdb_error(ltdb->tdb));
+ if (ltdb->kv_ops->prepare_write(ltdb) != 0) {
+ ret = ltdb->kv_ops->error(ltdb);
ltdb->in_transaction--;
ldb_debug_set(ldb_module_get_ctx(module),
LDB_DEBUG_FATAL,
@@ -1477,7 +1497,7 @@ static int ltdb_end_trans(struct ldb_module *module)
ltdb->in_transaction--;
ltdb->prepared_commit = false;
- if (tdb_transaction_commit(ltdb->tdb) != 0) {
+ if (ltdb->kv_ops->finish_write(ltdb) != 0) {
ret = ltdb->kv_ops->error(ltdb);
ldb_asprintf_errstring(ldb_module_get_ctx(module),
"Failure during tdb_transaction_commit(): %s -> %s",
@@ -1497,11 +1517,11 @@ static int ltdb_del_trans(struct ldb_module *module)
ltdb->in_transaction--;
if (ltdb_index_transaction_cancel(module) != 0) {
- tdb_transaction_cancel(ltdb->tdb);
+ ltdb->kv_ops->abort_write(ltdb);
return ltdb->kv_ops->error(ltdb);
}
- tdb_transaction_cancel(ltdb->tdb);
+ ltdb->kv_ops->abort_write(ltdb);
return LDB_SUCCESS;
}
@@ -1702,6 +1722,10 @@ static struct kv_db_ops key_value_ops = {
.exists = ltdb_tdb_exists,
.lock_read = ltdb_lock_read,
.unlock_read = ltdb_unlock_read,
+ .begin_write = ltdb_tdb_transaction_start,
+ .prepare_write = ltdb_tdb_transaction_prepare_commit,
+ .finish_write = ltdb_tdb_transaction_commit,
+ .abort_write = ltdb_tdb_transaction_cancel,
.error = ltdb_error,
.name = ltdb_tdb_name,
};
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index f2a6e7a..3cd3aa9 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -11,6 +11,10 @@ struct kv_db_ops {
int (*exists)(struct ltdb_private *ltdb, TDB_DATA key);
int (*lock_read)(struct ldb_module *);
int (*unlock_read)(struct ldb_module *);
+ int (*begin_write)(struct ltdb_private *);
+ int (*prepare_write)(struct ltdb_private *);
+ int (*abort_write)(struct ltdb_private *);
+ int (*finish_write)(struct ltdb_private *);
int (*error)(struct ltdb_private *ltdb);
const char * (*name)(struct ltdb_private *ltdb);
};
--
1.9.1
From 749d168684d64092a3b47299ba0279a376d48ba1 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Tue, 10 Jan 2017 23:19:55 +1300
Subject: [PATCH 13/34] ldb_tdb: Add lock_read and unlock_read to key value ops
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_search.c | 12 ++++++------
lib/ldb/ldb_tdb/ldb_tdb.c | 31 ++++++++++++++++++++++++-------
lib/ldb/ldb_tdb/ldb_tdb.h | 3 +--
3 files changed, 31 insertions(+), 15 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c
index fb618fc..f7a7bd1 100644
--- a/lib/ldb/ldb_tdb/ldb_search.c
+++ b/lib/ldb/ldb_tdb/ldb_search.c
@@ -721,17 +721,17 @@ int ltdb_search(struct ltdb_context *ctx)
ldb_request_set_state(req, LDB_ASYNC_PENDING);
- if (ltdb_lock_read(module) != 0) {
+ if (ltdb->kv_ops->lock_read(module) != 0) {
return LDB_ERR_OPERATIONS_ERROR;
}
if (ltdb_cache_load(module) != 0) {
- ltdb_unlock_read(module);
+ ltdb->kv_ops->unlock_read(module);
return LDB_ERR_OPERATIONS_ERROR;
}
if (req->op.search.tree == NULL) {
- ltdb_unlock_read(module);
+ ltdb->kv_ops->unlock_read(module);
return LDB_ERR_OPERATIONS_ERROR;
}
@@ -778,7 +778,7 @@ int ltdb_search(struct ltdb_context *ctx)
*/
ret = ltdb_search_and_return_base(ltdb, ctx);
- ltdb_unlock_read(module);
+ ltdb->kv_ops->unlock_read(module);
return ret;
@@ -840,7 +840,7 @@ int ltdb_search(struct ltdb_context *ctx)
* full search or we may return
* duplicate entries
*/
- ltdb_unlock_read(module);
+ ltdb->kv_ops->unlock_read(module);
return LDB_ERR_OPERATIONS_ERROR;
}
ret = ltdb_search_full(ctx);
@@ -850,7 +850,7 @@ int ltdb_search(struct ltdb_context *ctx)
}
}
- ltdb_unlock_read(module);
+ ltdb->kv_ops->unlock_read(module);
return ret;
}
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 5caeea9..135c740 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -94,7 +94,7 @@ int ltdb_err_map(enum TDB_ERROR tdb_code)
/*
lock the database for read - use by ltdb_search and ltdb_sequence_number
*/
-int ltdb_lock_read(struct ldb_module *module)
+static int ltdb_lock_read(struct ldb_module *module)
{
void *data = ldb_module_get_private(module);
struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
@@ -124,7 +124,7 @@ int ltdb_lock_read(struct ldb_module *module)
/*
unlock the database after a ltdb_lock_read()
*/
-int ltdb_unlock_read(struct ldb_module *module)
+static int ltdb_unlock_read(struct ldb_module *module)
{
void *data = ldb_module_get_private(module);
struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
@@ -1534,6 +1534,8 @@ static int ltdb_sequence_number(struct ltdb_context *ctx,
struct ldb_context *ldb;
struct ldb_module *module = ctx->module;
struct ldb_request *req = ctx->req;
+ void *data = ldb_module_get_private(module);
+ struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
TALLOC_CTX *tmp_ctx = NULL;
struct ldb_seqnum_request *seq;
struct ldb_seqnum_result *res;
@@ -1552,7 +1554,7 @@ static int ltdb_sequence_number(struct ltdb_context *ctx,
ldb_request_set_state(req, LDB_ASYNC_PENDING);
- if (ltdb_lock_read(module) != 0) {
+ if (ltdb->kv_ops->lock_read(module) != 0) {
return LDB_ERR_OPERATIONS_ERROR;
}
@@ -1614,7 +1616,8 @@ static int ltdb_sequence_number(struct ltdb_context *ctx,
done:
talloc_free(tmp_ctx);
- ltdb_unlock_read(module);
+
+ ltdb->kv_ops->unlock_read(module);
return ret;
}
@@ -1878,6 +1881,21 @@ static int ltdb_init_rootdse(struct ldb_module *module)
return LDB_SUCCESS;
}
+
+static int generic_lock_read(struct ldb_module *module)
+{
+ void *data = ldb_module_get_private(module);
+ struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
+ return ltdb->kv_ops->lock_read(module);
+}
+
+static int generic_unlock_read(struct ldb_module *module)
+{
+ void *data = ldb_module_get_private(module);
+ struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
+ return ltdb->kv_ops->unlock_read(module);
+}
+
static const struct ldb_module_ops ltdb_ops = {
.name = "tdb",
.init_context = ltdb_init_rootdse,
@@ -1891,8 +1909,8 @@ static const struct ldb_module_ops ltdb_ops = {
.end_transaction = ltdb_end_trans,
.prepare_commit = ltdb_prepare_commit,
.del_transaction = ltdb_del_trans,
- .read_lock = ltdb_lock_read,
- .read_unlock = ltdb_unlock_read,
+ .read_lock = generic_lock_read,
+ .read_unlock = generic_unlock_read,
};
/*
@@ -1911,7 +1929,6 @@ static int ltdb_connect(struct ldb_context *ldb, const char *url,
* We hold locks, so we must use a private event context
* on each returned handle
*/
-
ldb_set_require_private_event_context(ldb);
/* parse the url */
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 3cd3aa9..b494817 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -159,8 +159,6 @@ int ltdb_filter_attrs(TALLOC_CTX *mem_ctx,
int ltdb_search(struct ltdb_context *ctx);
/* The following definitions come from lib/ldb/ldb_tdb/ldb_tdb.c */
-int ltdb_lock_read(struct ldb_module *module);
-int ltdb_unlock_read(struct ldb_module *module);
/*
* Determine if this key could hold a record. We allow the new GUID
* index, the old DN index and a possible future ID=
@@ -179,6 +177,7 @@ int ltdb_idx_to_key(struct ldb_module *module,
TALLOC_CTX *mem_ctx,
const struct ldb_val *idx_val,
TDB_DATA *key);
+TDB_DATA ltdb_key(struct ldb_module *module, struct ldb_dn *dn);
int ltdb_store(struct ldb_module *module, const struct ldb_message *msg, int flgs);
int ltdb_modify_internal(struct ldb_module *module, const struct ldb_message *msg, struct ldb_request *req);
int ltdb_delete_noindex(struct ldb_module *module,
--
1.9.1
From 6d5d34f7e0bf9f8821342b0c3424480c8fb0bb24 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Tue, 10 Jan 2017 23:23:22 +1300
Subject: [PATCH 14/34] ldb_tdb: Remove tdb_get_seqnum and use a generic
'has_changed'
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_cache.c | 8 ++++----
lib/ldb/ldb_tdb/ldb_tdb.c | 10 ++++++++++
lib/ldb/ldb_tdb/ldb_tdb.h | 1 +
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_cache.c b/lib/ldb/ldb_tdb/ldb_cache.c
index 600b73f..4790bcd 100644
--- a/lib/ldb/ldb_tdb/ldb_cache.c
+++ b/lib/ldb/ldb_tdb/ldb_cache.c
@@ -391,8 +391,7 @@ int ltdb_cache_load(struct ldb_module *module)
ldb = ldb_module_get_ctx(module);
/* a very fast check to avoid extra database reads */
- if (ltdb->cache != NULL &&
- tdb_get_seqnum(ltdb->tdb) == ltdb->tdb_seqnum) {
+ if (ltdb->cache != NULL && !ltdb->kv_ops->has_changed(ltdb)) {
return 0;
}
@@ -432,7 +431,8 @@ int ltdb_cache_load(struct ldb_module *module)
}
}
- ltdb->tdb_seqnum = tdb_get_seqnum(ltdb->tdb);
+ /* Ignore the result, and update the sequence number */
+ ltdb->kv_ops->has_changed(ltdb);
/* if the current internal sequence number is the same as the one
in the database then assume the rest of the cache is OK */
@@ -594,7 +594,7 @@ int ltdb_increase_sequence_number(struct ldb_module *module)
/* updating the tdb_seqnum here avoids us reloading the cache
records due to our own modification */
- ltdb->tdb_seqnum = tdb_get_seqnum(ltdb->tdb);
+ ltdb->kv_ops->has_changed(ltdb);
return ret;
}
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 135c740..108565c 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -1719,6 +1719,15 @@ static const char * ltdb_tdb_name(struct ltdb_private *ltdb)
return tdb_name(ltdb->tdb);
}
+static bool ltdb_tdb_changed(struct ltdb_private *ltdb)
+{
+ bool ret = (tdb_get_seqnum(ltdb->tdb) != ltdb->tdb_seqnum);
+
+ ltdb->tdb_seqnum = tdb_get_seqnum(ltdb->tdb);
+
+ return ret;
+}
+
static struct kv_db_ops key_value_ops = {
.store = ltdb_tdb_store,
.delete = ltdb_tdb_delete,
@@ -1731,6 +1740,7 @@ static struct kv_db_ops key_value_ops = {
.abort_write = ltdb_tdb_transaction_cancel,
.error = ltdb_error,
.name = ltdb_tdb_name,
+ .has_changed = ltdb_tdb_changed,
};
static void ltdb_callback(struct tevent_context *ev,
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index b494817..41d2a8e 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -17,6 +17,7 @@ struct kv_db_ops {
int (*finish_write)(struct ltdb_private *);
int (*error)(struct ltdb_private *ltdb);
const char * (*name)(struct ltdb_private *ltdb);
+ bool (*has_changed)(struct ltdb_private *ltdb);
};
/* this private structure is used by the ltdb backend in the
--
1.9.1
From 66f9f76229d5a0d9adf9e9a76b2fa7e63b94d43f Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Tue, 13 Feb 2018 15:21:34 +1300
Subject: [PATCH 15/34] ldb_tdb: Add errorstr to the key value ops
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_tdb.c | 12 +++++++++---
lib/ldb/ldb_tdb/ldb_tdb.h | 1 +
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 108565c..2aa0583 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -423,6 +423,11 @@ static int ltdb_error(struct ltdb_private *ltdb)
return ltdb_err_map(tdb_error(ltdb->tdb));
}
+static const char *ltdb_errorstr(struct ltdb_private *ltdb)
+{
+ return tdb_errorstr(ltdb->tdb);
+}
+
/*
store a record into the db
*/
@@ -1470,8 +1475,8 @@ static int ltdb_prepare_commit(struct ldb_module *module)
ldb_debug_set(ldb_module_get_ctx(module),
LDB_DEBUG_FATAL,
"Failure during "
- "tdb_transaction_prepare_commit(): %s -> %s",
- tdb_errorstr(ltdb->tdb),
+ "prepare_write): %s -> %s",
+ ltdb->kv_ops->errorstr(ltdb),
ldb_strerror(ret));
return ret;
}
@@ -1501,7 +1506,7 @@ static int ltdb_end_trans(struct ldb_module *module)
ret = ltdb->kv_ops->error(ltdb);
ldb_asprintf_errstring(ldb_module_get_ctx(module),
"Failure during tdb_transaction_commit(): %s -> %s",
- tdb_errorstr(ltdb->tdb),
+ ltdb->kv_ops->errorstr(ltdb),
ldb_strerror(ret));
return ret;
}
@@ -1739,6 +1744,7 @@ static struct kv_db_ops key_value_ops = {
.finish_write = ltdb_tdb_transaction_commit,
.abort_write = ltdb_tdb_transaction_cancel,
.error = ltdb_error,
+ .errorstr = ltdb_errorstr,
.name = ltdb_tdb_name,
.has_changed = ltdb_tdb_changed,
};
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 41d2a8e..bf8721c 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -16,6 +16,7 @@ struct kv_db_ops {
int (*abort_write)(struct ltdb_private *);
int (*finish_write)(struct ltdb_private *);
int (*error)(struct ltdb_private *ltdb);
+ const char * (*errorstr)(struct ltdb_private *ltdb);
const char * (*name)(struct ltdb_private *ltdb);
bool (*has_changed)(struct ltdb_private *ltdb);
};
--
1.9.1
From 9bb2c9a6a7cb3344d49832c70b92c5408d4ee2d9 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Wed, 11 Jan 2017 11:36:48 +1300
Subject: [PATCH 16/34] ldb_tdb: factor out the (to be) common init code
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_tdb.c | 72 +++++++++++++++++++++++++++--------------------
lib/ldb/ldb_tdb/ldb_tdb.h | 3 ++
2 files changed, 45 insertions(+), 30 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 2aa0583..2701270 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -1929,6 +1929,45 @@ static const struct ldb_module_ops ltdb_ops = {
.read_unlock = generic_unlock_read,
};
+int init_store(struct ltdb_private *ltdb,
+ const char *name,
+ struct ldb_context *ldb,
+ const char *options[],
+ struct ldb_module **_module)
+{
+ struct ldb_module *module;
+
+ if (getenv("LDB_WARN_UNINDEXED")) {
+ ltdb->warn_unindexed = true;
+ }
+
+ if (getenv("LDB_WARN_REINDEX")) {
+ ltdb->warn_reindex = true;
+ }
+
+ ltdb->sequence_number = 0;
+
+ module = ldb_module_new(ldb, ldb, name, <db_ops);
+ if (!module) {
+ ldb_oom(ldb);
+ talloc_free(ltdb);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+ ldb_module_set_private(module, ltdb);
+ talloc_steal(module, ltdb);
+
+ if (ltdb_cache_load(module) != 0) {
+ ldb_asprintf_errstring(ldb, "Unable to load ltdb cache "
+ "records for backend '%s'", name);
+ talloc_free(module);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
+ *_module = module;
+
+ return LDB_SUCCESS;
+}
+
/*
connect to the database
*/
@@ -1936,7 +1975,6 @@ static int ltdb_connect(struct ldb_context *ldb, const char *url,
unsigned int flags, const char *options[],
struct ldb_module **_module)
{
- struct ldb_module *module;
const char *path;
int tdb_flags, open_flags;
struct ltdb_private *ltdb;
@@ -2001,6 +2039,8 @@ static int ltdb_connect(struct ldb_context *ldb, const char *url,
open_flags = O_CREAT | O_RDWR;
}
+ ltdb->kv_ops = &key_value_ops;
+
/* note that we use quite a large default hash size */
ltdb->tdb = ltdb_wrap_open(ltdb, path, 10000,
tdb_flags, open_flags,
@@ -2017,35 +2057,7 @@ static int ltdb_connect(struct ldb_context *ldb, const char *url,
return LDB_ERR_OPERATIONS_ERROR;
}
- if (getenv("LDB_WARN_UNINDEXED")) {
- ltdb->warn_unindexed = true;
- }
-
- if (getenv("LDB_WARN_REINDEX")) {
- ltdb->warn_reindex = true;
- }
-
- ltdb->sequence_number = 0;
- ltdb->kv_ops = &key_value_ops;
-
- module = ldb_module_new(ldb, ldb, "ldb_tdb backend", <db_ops);
- if (!module) {
- ldb_oom(ldb);
- talloc_free(ltdb);
- return LDB_ERR_OPERATIONS_ERROR;
- }
- ldb_module_set_private(module, ltdb);
- talloc_steal(module, ltdb);
-
- if (ltdb_cache_load(module) != 0) {
- ldb_asprintf_errstring(ldb,
- "Unable to load ltdb cache records of tdb '%s'", path);
- talloc_free(module);
- return LDB_ERR_OPERATIONS_ERROR;
- }
-
- *_module = module;
- return LDB_SUCCESS;
+ return init_store(ltdb, "ldb_tdb backend", ldb, options, _module);
}
int ldb_tdb_init(const char *version)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index bf8721c..b2b5991 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -190,3 +190,6 @@ struct tdb_context *ltdb_wrap_open(TALLOC_CTX *mem_ctx,
const char *path, int hash_size, int tdb_flags,
int open_flags, mode_t mode,
struct ldb_context *ldb);
+int init_store(struct ltdb_private *ltdb, const char *name,
+ struct ldb_context *ldb, const char *options[],
+ struct ldb_module **_module);
--
1.9.1
From 11e3f13347d36285eb25201038dacb47ddc453dd Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Tue, 10 Jan 2017 20:43:38 +1300
Subject: [PATCH 17/34] ldb_tdb: Use key value ops for fetch command
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_search.c | 6 +++---
lib/ldb/ldb_tdb/ldb_tdb.c | 9 +++++++++
lib/ldb/ldb_tdb/ldb_tdb.h | 4 ++++
3 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c
index f7a7bd1..b0322d5 100644
--- a/lib/ldb/ldb_tdb/ldb_search.c
+++ b/lib/ldb/ldb_tdb/ldb_search.c
@@ -252,9 +252,9 @@ int ltdb_search_key(struct ldb_module *module, struct ltdb_private *ltdb,
msg->num_elements = 0;
msg->elements = NULL;
- ret = tdb_parse_record(ltdb->tdb, tdb_key,
- ltdb_parse_data_unpack, &ctx);
-
+ ret = ltdb->kv_ops->fetch_and_parse(ltdb, tdb_key,
+ ltdb_parse_data_unpack, &ctx);
+
if (ret == -1) {
ret = ltdb->kv_ops->error(ltdb);
if (ret == LDB_SUCCESS) {
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 2701270..e458bc2 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -1719,6 +1719,14 @@ static void ltdb_handle_extended(struct ltdb_context *ctx)
ltdb_request_extended_done(ctx, ext, ret);
}
+static int ltdb_tdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
+ int (*parser)(TDB_DATA key, TDB_DATA data,
+ void *private_data),
+ void *ctx)
+{
+ return tdb_parse_record(ltdb->tdb, key, parser, ctx);
+}
+
static const char * ltdb_tdb_name(struct ltdb_private *ltdb)
{
return tdb_name(ltdb->tdb);
@@ -1737,6 +1745,7 @@ static struct kv_db_ops key_value_ops = {
.store = ltdb_tdb_store,
.delete = ltdb_tdb_delete,
.exists = ltdb_tdb_exists,
+ .fetch_and_parse = ltdb_tdb_parse_record,
.lock_read = ltdb_lock_read,
.unlock_read = ltdb_unlock_read,
.begin_write = ltdb_tdb_transaction_start,
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index b2b5991..44e5cf7 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -9,6 +9,10 @@ struct kv_db_ops {
int (*store)(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags);
int (*delete)(struct ltdb_private *ltdb, TDB_DATA key);
int (*exists)(struct ltdb_private *ltdb, TDB_DATA key);
+ int (*fetch_and_parse)(struct ltdb_private *ltdb, TDB_DATA key,
+ int (*parser)(TDB_DATA key, TDB_DATA data,
+ void *private_data),
+ void *ctx);
int (*lock_read)(struct ldb_module *);
int (*unlock_read)(struct ldb_module *);
int (*begin_write)(struct ltdb_private *);
--
1.9.1
From c76faa17a32a8d53049a737af1549af52639ceae Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Fri, 16 Feb 2018 13:06:31 +1300
Subject: [PATCH 18/34] ldb_tdb: Implement a traversal function in key value
ops
This can handle both read-only and writable traverses.
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_index.c | 75 +++++++++++++-------------------------------
lib/ldb/ldb_tdb/ldb_search.c | 20 +++++-------
lib/ldb/ldb_tdb/ldb_tdb.c | 74 ++++++++++++++++++++++++++++++++++++++++++-
lib/ldb/ldb_tdb/ldb_tdb.h | 14 +++++++++
4 files changed, 117 insertions(+), 66 deletions(-)
diff --git a/lib/ldb/ldb_tdb/ldb_index.c b/lib/ldb/ldb_tdb/ldb_index.c
index 0984760..baf56d9 100644
--- a/lib/ldb/ldb_tdb/ldb_index.c
+++ b/lib/ldb/ldb_tdb/ldb_index.c
@@ -2316,17 +2316,16 @@ int ltdb_index_delete(struct ldb_module *module, const struct ldb_message *msg)
commit, which in turn greatly reduces DB churn as we will likely
be able to do a direct update into the old record.
*/
-static int delete_index(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, void *state)
+static int delete_index(struct ltdb_private *ltdb, struct ldb_val key, struct ldb_val data, void *state)
{
struct ldb_module *module = state;
- struct ltdb_private *ltdb = talloc_get_type(ldb_module_get_private(module), struct ltdb_private);
const char *dnstr = "DN=" LTDB_INDEX ":";
struct dn_list list;
struct ldb_dn *dn;
struct ldb_val v;
int ret;
- if (strncmp((char *)key.dptr, dnstr, strlen(dnstr)) != 0) {
+ if (strncmp((char *)key.data, dnstr, strlen(dnstr)) != 0) {
return 0;
}
/* we need to put a empty list in the internal tdb for this
@@ -2335,8 +2334,8 @@ static int delete_index(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, vo
list.count = 0;
/* the offset of 3 is to remove the DN= prefix. */
- v.data = key.dptr + 3;
- v.length = strnlen((char *)key.dptr, key.dsize) - 3;
+ v.data = key.data + 3;
+ v.length = strnlen((char *)key.data, key.length) - 3;
dn = ldb_dn_from_ldb_val(ltdb, ldb_module_get_ctx(module), &v);
@@ -2356,29 +2355,23 @@ static int delete_index(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, vo
return 0;
}
-struct ltdb_reindex_context {
- struct ldb_module *module;
- int error;
- uint32_t count;
-};
-
/*
- traversal function that adds @INDEX records during a re index
+ traversal function that adds @INDEX records during a re index TODO wrong comment
*/
-static int re_key(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, void *state)
+static int re_key(struct ltdb_private *ltdb, struct ldb_val ldb_key, struct ldb_val val, void *state)
{
struct ldb_context *ldb;
struct ltdb_reindex_context *ctx = (struct ltdb_reindex_context *)state;
struct ldb_module *module = ctx->module;
struct ldb_message *msg;
unsigned int nb_elements_in_db;
- const struct ldb_val val = {
- .data = data.dptr,
- .length = data.dsize,
- };
int ret;
TDB_DATA key2;
bool is_record;
+ TDB_DATA key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
+ };
ldb = ldb_module_get_ctx(module);
@@ -2433,32 +2426,11 @@ static int re_key(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, void *st
}
if (key.dsize != key2.dsize ||
(memcmp(key.dptr, key2.dptr, key.dsize) != 0)) {
- int tdb_ret;
- tdb_ret = tdb_delete(tdb, key);
- if (tdb_ret != 0) {
- ldb_debug(ldb, LDB_DEBUG_ERROR,
- "Failed to delete %*.*s "
- "for rekey as %*.*s: %s",
- (int)key.dsize, (int)key.dsize,
- (const char *)key.dptr,
- (int)key2.dsize, (int)key2.dsize,
- (const char *)key.dptr,
- tdb_errorstr(tdb));
- ctx->error = ltdb_err_map(tdb_error(tdb));
- return -1;
- }
- tdb_ret = tdb_store(tdb, key2, data, 0);
- if (tdb_ret != 0) {
- ldb_debug(ldb, LDB_DEBUG_ERROR,
- "Failed to rekey %*.*s as %*.*s: %s",
- (int)key.dsize, (int)key.dsize,
- (const char *)key.dptr,
- (int)key2.dsize, (int)key2.dsize,
- (const char *)key.dptr,
- tdb_errorstr(tdb));
- ctx->error = ltdb_err_map(tdb_error(tdb));
- return -1;
- }
+ TDB_DATA data = {
+ .dptr = val.data,
+ .dsize = val.length
+ };
+ ltdb->kv_ops->update_in_iterate(ltdb, key, key2, data, ctx);
}
talloc_free(key2.dptr);
@@ -2477,18 +2449,16 @@ static int re_key(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, void *st
/*
traversal function that adds @INDEX records during a re index
*/
-static int re_index(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, void *state)
+static int re_index(struct ltdb_private *ltdb, struct ldb_val ldb_key, struct ldb_val val, void *state)
{
struct ldb_context *ldb;
struct ltdb_reindex_context *ctx = (struct ltdb_reindex_context *)state;
struct ldb_module *module = ctx->module;
- struct ltdb_private *ltdb = talloc_get_type(ldb_module_get_private(module),
- struct ltdb_private);
struct ldb_message *msg;
unsigned int nb_elements_in_db;
- const struct ldb_val val = {
- .data = data.dptr,
- .length = data.dsize,
+ TDB_DATA key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
};
int ret;
bool is_record;
@@ -2598,7 +2568,7 @@ int ltdb_reindex(struct ldb_module *module)
/* first traverse the database deleting any @INDEX records by
* putting NULL entries in the in-memory tdb
*/
- ret = tdb_traverse(ltdb->tdb, delete_index, module);
+ ret = ltdb->kv_ops->iterate(ltdb, delete_index, module);
if (ret < 0) {
struct ldb_context *ldb = ldb_module_get_ctx(module);
ldb_asprintf_errstring(ldb, "index deletion traverse failed: %s",
@@ -2610,8 +2580,7 @@ int ltdb_reindex(struct ldb_module *module)
ctx.error = 0;
ctx.count = 0;
- /* now traverse adding any indexes for normal LDB records */
- ret = tdb_traverse(ltdb->tdb, re_key, &ctx);
+ ret = ltdb->kv_ops->iterate(ltdb, re_key, &ctx);
if (ret < 0) {
struct ldb_context *ldb = ldb_module_get_ctx(module);
ldb_asprintf_errstring(ldb, "key correction traverse failed: %s",
@@ -2629,7 +2598,7 @@ int ltdb_reindex(struct ldb_module *module)
ctx.count = 0;
/* now traverse adding any indexes for normal LDB records */
- ret = tdb_traverse(ltdb->tdb, re_index, &ctx);
+ ret = ltdb->kv_ops->iterate(ltdb, re_index, &ctx);
if (ret < 0) {
struct ldb_context *ldb = ldb_module_get_ctx(module);
ldb_asprintf_errstring(ldb, "reindexing traverse failed: %s",
diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c
index b0322d5..0d205b3 100644
--- a/lib/ldb/ldb_tdb/ldb_search.c
+++ b/lib/ldb/ldb_tdb/ldb_search.c
@@ -497,23 +497,23 @@ failed:
/*
search function for a non-indexed search
*/
-static int search_func(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, void *state)
+static int search_func(struct ltdb_private *ltdb, struct ldb_val key, struct ldb_val val, void *state)
{
struct ldb_context *ldb;
struct ltdb_context *ac;
struct ldb_message *msg, *filtered_msg;
- const struct ldb_val val = {
- .data = data.dptr,
- .length = data.dsize,
- };
int ret;
bool matched;
unsigned int nb_elements_in_db;
+ TDB_DATA tdb_key = {
+ .dptr = key.data,
+ .dsize = key.length
+ };
ac = talloc_get_type(state, struct ltdb_context);
ldb = ldb_module_get_ctx(ac->module);
- if (ltdb_key_is_record(key) == false) {
+ if (ltdb_key_is_record(tdb_key) == false) {
return 0;
}
@@ -538,7 +538,7 @@ static int search_func(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data, voi
if (!msg->dn) {
msg->dn = ldb_dn_new(msg, ldb,
- (char *)key.dptr + 3);
+ (char *)key.data + 3);
if (msg->dn == NULL) {
talloc_free(msg);
ac->error = LDB_ERR_OPERATIONS_ERROR;
@@ -591,11 +591,7 @@ static int ltdb_search_full(struct ltdb_context *ctx)
int ret;
ctx->error = LDB_SUCCESS;
- if (ltdb->in_transaction != 0) {
- ret = tdb_traverse(ltdb->tdb, search_func, ctx);
- } else {
- ret = tdb_traverse_read(ltdb->tdb, search_func, ctx);
- }
+ ret = ltdb->kv_ops->iterate(ltdb, search_func, ctx);
if (ret < 0) {
return LDB_ERR_OPERATIONS_ERROR;
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index e458bc2..a2d37fb 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -931,7 +931,6 @@ static int msg_delete_element(struct ldb_module *module,
return LDB_ERR_NO_SUCH_ATTRIBUTE;
}
-
/*
modify a record - internal interface
@@ -1719,6 +1718,77 @@ static void ltdb_handle_extended(struct ltdb_context *ctx)
ltdb_request_extended_done(ctx, ext, ret);
}
+struct kv_ctx {
+ ldb_kv_traverse_fn kv_traverse_fn;
+ void *ctx;
+ struct ltdb_private *ltdb;
+};
+
+static int ldb_tdb_traverse_fn_wrapper(struct tdb_context *tdb, TDB_DATA tdb_key, TDB_DATA tdb_data, void *ctx)
+{
+ struct kv_ctx *kv_ctx = ctx;
+ struct ldb_val key = {
+ .length = tdb_key.dsize,
+ .data = tdb_key.dptr,
+ };
+ struct ldb_val data = {
+ .length = tdb_data.dsize,
+ .data = tdb_data.dptr,
+ };
+ return kv_ctx->kv_traverse_fn(kv_ctx->ltdb, key, data, kv_ctx->ctx);
+}
+
+static int ltdb_tdb_traverse_fn(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn, void *ctx)
+{
+ struct kv_ctx kv_ctx = {
+ .kv_traverse_fn = fn,
+ .ctx = ctx,
+ .ltdb = ltdb
+ };
+ if (ltdb->in_transaction != 0) {
+ return tdb_traverse(ltdb->tdb, ldb_tdb_traverse_fn_wrapper, &kv_ctx);
+ } else {
+ return tdb_traverse_read(ltdb->tdb, ldb_tdb_traverse_fn_wrapper, &kv_ctx);
+ }
+}
+
+static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA key2, TDB_DATA data, void *state)
+{
+ int tdb_ret;
+ struct ldb_context *ldb;
+ struct ltdb_reindex_context *ctx = (struct ltdb_reindex_context *)state;
+ struct ldb_module *module = ctx->module;
+
+ ldb = ldb_module_get_ctx(module);
+
+ tdb_ret = tdb_delete(ltdb->tdb, key);
+ if (tdb_ret != 0) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR,
+ "Failed to delete %*.*s "
+ "for rekey as %*.*s: %s",
+ (int)key.dsize, (int)key.dsize,
+ (const char *)key.dptr,
+ (int)key2.dsize, (int)key2.dsize,
+ (const char *)key.dptr,
+ tdb_errorstr(ltdb->tdb));
+ ctx->error = ltdb_err_map(tdb_error(ltdb->tdb));
+ return -1;
+ }
+ tdb_ret = tdb_store(ltdb->tdb, key2, data, 0);
+ if (tdb_ret != 0) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR,
+ "Failed to rekey %*.*s as %*.*s: %s",
+ (int)key.dsize, (int)key.dsize,
+ (const char *)key.dptr,
+ (int)key2.dsize, (int)key2.dsize,
+ (const char *)key.dptr,
+ tdb_errorstr(ltdb->tdb));
+ ctx->error = ltdb_err_map(tdb_error(ltdb->tdb));
+ return -1;
+ }
+ return tdb_ret;
+}
+
static int ltdb_tdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
int (*parser)(TDB_DATA key, TDB_DATA data,
void *private_data),
@@ -1745,6 +1815,8 @@ static struct kv_db_ops key_value_ops = {
.store = ltdb_tdb_store,
.delete = ltdb_tdb_delete,
.exists = ltdb_tdb_exists,
+ .iterate = ltdb_tdb_traverse_fn,
+ .update_in_iterate = ltdb_tdb_update_in_iterate,
.fetch_and_parse = ltdb_tdb_parse_record,
.lock_read = ltdb_lock_read,
.unlock_read = ltdb_unlock_read,
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 44e5cf7..a03e13b 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -5,10 +5,17 @@
#include "ldb_module.h"
struct ltdb_private;
+typedef int (*ldb_kv_traverse_fn)(struct ltdb_private *ltdb,
+ struct ldb_val key, struct ldb_val data,
+ void *ctx);
+
struct kv_db_ops {
int (*store)(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags);
int (*delete)(struct ltdb_private *ltdb, TDB_DATA key);
int (*exists)(struct ltdb_private *ltdb, TDB_DATA key);
+ int (*iterate)(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn, void *ctx);
+ int (*update_in_iterate)(struct ltdb_private *ltdb, TDB_DATA key,
+ TDB_DATA key2, TDB_DATA data, void *ctx);
int (*fetch_and_parse)(struct ltdb_private *ltdb, TDB_DATA key,
int (*parser)(TDB_DATA key, TDB_DATA data,
void *private_data),
@@ -80,6 +87,13 @@ struct ltdb_context {
int error;
};
+struct ltdb_reindex_context {
+ struct ldb_module *module;
+ int error;
+ uint32_t count;
+};
+
+
/* special record types */
#define LTDB_INDEX "@INDEX"
#define LTDB_INDEXLIST "@INDEXLIST"
--
1.9.1
From 75977f417f9f5d2037a1c013228957310acf034d Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Fri, 13 Jan 2017 11:32:14 +1300
Subject: [PATCH 19/34] partition: Allow a different backend store from
@PARTITION
By default, use tdb, but otherwise read the value from backendStore.
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
source4/dsdb/samdb/ldb_modules/partition.h | 2 ++
source4/dsdb/samdb/ldb_modules/partition_init.c | 37 ++++++++++++++++---------
2 files changed, 26 insertions(+), 13 deletions(-)
diff --git a/source4/dsdb/samdb/ldb_modules/partition.h b/source4/dsdb/samdb/ldb_modules/partition.h
index ea05e94..1bf213f 100644
--- a/source4/dsdb/samdb/ldb_modules/partition.h
+++ b/source4/dsdb/samdb/ldb_modules/partition.h
@@ -57,6 +57,8 @@ struct partition_private_data {
uint32_t in_transaction;
struct ldb_message *forced_module_msg;
+
+ const char *backend_db_store;
};
#include "dsdb/samdb/ldb_modules/partition_proto.h"
diff --git a/source4/dsdb/samdb/ldb_modules/partition_init.c b/source4/dsdb/samdb/ldb_modules/partition_init.c
index 3e2648e..9a6ac0c 100644
--- a/source4/dsdb/samdb/ldb_modules/partition_init.c
+++ b/source4/dsdb/samdb/ldb_modules/partition_init.c
@@ -139,7 +139,7 @@ static int partition_reload_metadata(struct ldb_module *module, struct partition
struct ldb_result *res;
struct ldb_context *ldb = ldb_module_get_ctx(module);
const char *attrs[] = { "partition", "replicateEntries", "modules", "ldapBackend",
- "partialReplica", NULL };
+ "partialReplica", "backendStore", NULL };
/* perform search for @PARTITION, looking for module, replicateEntries and ldapBackend */
ret = dsdb_module_search_dn(module, mem_ctx, &res,
ldb_dn_new(mem_ctx, ldb, DSDB_PARTITION_DN),
@@ -201,8 +201,8 @@ static const char **find_modules_for_dn(struct partition_private_data *data, str
static int new_partition_from_dn(struct ldb_context *ldb, struct partition_private_data *data,
TALLOC_CTX *mem_ctx,
struct ldb_dn *dn, const char *filename,
+ const char *backend_db_store,
struct dsdb_partition **partition) {
- const char *backend_url;
struct dsdb_control_current_partition *ctrl;
struct ldb_module *backend_module;
struct ldb_module *module_chain;
@@ -225,29 +225,31 @@ static int new_partition_from_dn(struct ldb_context *ldb, struct partition_priva
(*partition)->backend_url = data->ldapBackend;
} else {
/* the backend LDB is the DN (base64 encoded if not 'plain') followed by .ldb */
- backend_url = ldb_relative_path(ldb,
- *partition,
- filename);
- if (!backend_url) {
+ char *backend_path = ldb_relative_path(ldb,
+ *partition,
+ filename);
+ if (!backend_path) {
ldb_asprintf_errstring(ldb,
"partition_init: unable to determine an relative path for partition: %s", filename);
talloc_free(*partition);
return LDB_ERR_OPERATIONS_ERROR;
}
- (*partition)->backend_url = talloc_steal((*partition), backend_url);
+ (*partition)->backend_url = talloc_asprintf(*partition, "%s://%s",
+ backend_db_store,
+ backend_path);
if (!(ldb_module_flags(ldb) & LDB_FLG_RDONLY)) {
char *p;
- char *backend_dir = talloc_strdup(*partition, backend_url);
-
- p = strrchr(backend_dir, '/');
+ char *backend_dir;
+
+ p = strrchr(backend_path, '/');
if (p) {
p[0] = '\0';
}
+ backend_dir = backend_path;
/* Failure is quite reasonable, it might alredy exist */
mkdir(backend_dir, 0700);
- talloc_free(backend_dir);
}
}
@@ -417,6 +419,13 @@ int partition_reload_if_required(struct ldb_module *module,
partition_attributes = ldb_msg_find_element(msg, "partition");
partial_replicas = ldb_msg_find_element(msg, "partialReplica");
+ data->backend_db_store
+ = talloc_strdup(data, ldb_msg_find_attr_as_string(msg, "backendStore", "tdb"));
+
+ if (data->backend_db_store == NULL) {
+ talloc_free(mem_ctx);
+ return ldb_module_oom(module);
+ }
for (i=0; partition_attributes && i < partition_attributes->num_values; i++) {
unsigned int j;
@@ -498,7 +507,7 @@ int partition_reload_if_required(struct ldb_module *module,
* correctly. We don't want to mess that up as the
* schema isn't loaded yet */
ret = new_partition_from_dn(ldb, data, data->partitions, dn,
- filename,
+ filename, data->backend_db_store,
&partition);
if (ret != LDB_SUCCESS) {
talloc_free(mem_ctx);
@@ -816,7 +825,9 @@ int partition_create(struct ldb_module *module, struct ldb_request *req)
}
/* Make a partition structure for this new partition, so we can copy in the template structure */
- ret = new_partition_from_dn(ldb, data, req, ldb_dn_copy(req, dn), filename, &partition);
+ ret = new_partition_from_dn(ldb, data, req, ldb_dn_copy(req, dn),
+ filename, data->backend_db_store,
+ &partition);
if (ret != LDB_SUCCESS) {
return ret;
}
--
1.9.1
From 13639ed662f826c233d0c5f13ebce13097388fdf Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Fri, 16 Feb 2018 13:26:46 +1300
Subject: [PATCH 20/34] ldb_tdb: Build a key value operation library
This allows sharing of the originally ldb_tdb operations to the new
ldb_mdb backend.
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_tdb/ldb_tdb.c | 12 +++------
lib/ldb/ldb_tdb/ldb_tdb.h | 4 +++
lib/ldb/ldb_tdb/ldb_tdb_init.c | 59 ++++++++++++++++++++++++++++++++++++++++++
lib/ldb/wscript | 12 ++++++---
4 files changed, 75 insertions(+), 12 deletions(-)
create mode 100644 lib/ldb/ldb_tdb/ldb_tdb_init.c
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index a2d37fb..8993340 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -2052,9 +2052,9 @@ int init_store(struct ltdb_private *ltdb,
/*
connect to the database
*/
-static int ltdb_connect(struct ldb_context *ldb, const char *url,
- unsigned int flags, const char *options[],
- struct ldb_module **_module)
+int ltdb_connect(struct ldb_context *ldb, const char *url,
+ unsigned int flags, const char *options[],
+ struct ldb_module **_module)
{
const char *path;
int tdb_flags, open_flags;
@@ -2140,9 +2140,3 @@ static int ltdb_connect(struct ldb_context *ldb, const char *url,
return init_store(ltdb, "ldb_tdb backend", ldb, options, _module);
}
-
-int ldb_tdb_init(const char *version)
-{
- LDB_MODULE_CHECK_VERSION(version);
- return ldb_register_backend("tdb", ltdb_connect, false);
-}
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index a03e13b..5b3682c 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -211,3 +211,7 @@ struct tdb_context *ltdb_wrap_open(TALLOC_CTX *mem_ctx,
int init_store(struct ltdb_private *ltdb, const char *name,
struct ldb_context *ldb, const char *options[],
struct ldb_module **_module);
+
+int ltdb_connect(struct ldb_context *ldb, const char *url,
+ unsigned int flags, const char *options[],
+ struct ldb_module **_module);
diff --git a/lib/ldb/ldb_tdb/ldb_tdb_init.c b/lib/ldb/ldb_tdb/ldb_tdb_init.c
new file mode 100644
index 0000000..b18c98a
--- /dev/null
+++ b/lib/ldb/ldb_tdb/ldb_tdb_init.c
@@ -0,0 +1,59 @@
+/*
+ ldb database library
+
+ Copyright (C) Andrew Tridgell 2004
+ Copyright (C) Stefan Metzmacher 2004
+ Copyright (C) Simo Sorce 2006-2008
+ Copyright (C) Matthias Dieter Wallnöfer 2009-2010
+
+ ** NOTE! The following LGPL license applies to the ldb
+ ** library. This does NOT imply that all of Samba is released
+ ** under the LGPL
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 3 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, see <http://www.gnu.org/licenses/>.
+*/
+
+/*
+ * Name: ldb_tdb
+ *
+ * Component: ldb tdb backend
+ *
+ * Description: core functions for tdb backend
+ *
+ * Author: Andrew Tridgell
+ * Author: Stefan Metzmacher
+ *
+ * Modifications:
+ *
+ * - description: make the module use asynchronous calls
+ * date: Feb 2006
+ * Author: Simo Sorce
+ *
+ * - description: make it possible to use event contexts
+ * date: Jan 2008
+ * Author: Simo Sorce
+ *
+ * - description: fix up memory leaks and small bugs
+ * date: Oct 2009
+ * Author: Matthias Dieter Wallnöfer
+ */
+
+#include "ldb_tdb.h"
+#include "ldb_private.h"
+
+int ldb_tdb_init(const char *version)
+{
+ LDB_MODULE_CHECK_VERSION(version);
+ return ldb_register_backend("tdb", ltdb_connect, false);
+}
diff --git a/lib/ldb/wscript b/lib/ldb/wscript
index 8ae5be3..23ca4f4 100644
--- a/lib/ldb/wscript
+++ b/lib/ldb/wscript
@@ -307,14 +307,20 @@ def build(bld):
bld.SAMBA_MODULE('ldb_tdb',
bld.SUBDIR('ldb_tdb',
- '''ldb_tdb.c ldb_search.c ldb_index.c
- ldb_cache.c ldb_tdb_wrap.c'''),
+ '''ldb_tdb_init.c'''),
init_function='ldb_tdb_init',
module_init_name='ldb_init_module',
internal_module=False,
- deps='tdb ldb',
+ deps='tdb ldb ldb_key_value',
subsystem='ldb')
+ bld.SAMBA_LIBRARY('ldb_key_value',
+ bld.SUBDIR('ldb_tdb',
+ '''ldb_tdb.c ldb_search.c ldb_index.c
+ ldb_cache.c ldb_tdb_wrap.c'''),
+ private_library=True,
+ deps='tdb ldb')
+
# have a separate subsystem for common/ldb.c, so it can rebuild
# for install with a different -DLDB_MODULESDIR=
bld.SAMBA_SUBSYSTEM('LIBLDB_MAIN',
--
1.9.1
From afeeb93fc6fac0bb547acb1ff138e7f41eca3a86 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Wed, 11 Jan 2017 17:10:19 +1300
Subject: [PATCH 21/34] ldb_mdb: Implement the lmdb backend for ldb
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_mdb/ldb_mdb.c | 739 +++++++++++++++++++++++++++++++++++++
lib/ldb/ldb_mdb/ldb_mdb.h | 57 +++
lib/ldb/ldb_tdb/ldb_tdb.h | 1 +
lib/ldb/wscript | 13 +
source4/ldap_server/ldap_backend.c | 3 +-
5 files changed, 812 insertions(+), 1 deletion(-)
create mode 100644 lib/ldb/ldb_mdb/ldb_mdb.c
create mode 100644 lib/ldb/ldb_mdb/ldb_mdb.h
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
new file mode 100644
index 0000000..e9308dd
--- /dev/null
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -0,0 +1,739 @@
+/*
+ ldb database library using mdb back end
+
+ Copyright (C) Jakub Hrozek 2014
+ Copyright (C) Catalyst.Net Ltd 2017
+
+ ** NOTE! The following LGPL license applies to the ldb
+ ** library. This does NOT imply that all of Samba is released
+ ** under the LGPL
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 3 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include "lmdb.h"
+#include "ldb_mdb.h"
+#include "../ldb_tdb/ldb_tdb.h"
+#include "include/dlinklist.h"
+
+#define MDB_URL_PREFIX "mdb://"
+#define MDB_URL_PREFIX_SIZE (sizeof(MDB_URL_PREFIX)-1)
+
+#define MEGABYTE (1024*1024)
+#define GIGABYTE (1024*1024*1024)
+
+int ldb_mdb_err_map(int lmdb_err)
+{
+ switch (lmdb_err) {
+ case MDB_SUCCESS:
+ return LDB_SUCCESS;
+ case EIO:
+ return LDB_ERR_OPERATIONS_ERROR;
+ case MDB_INCOMPATIBLE:
+ case MDB_CORRUPTED:
+ case MDB_INVALID:
+ return LDB_ERR_UNAVAILABLE;
+ case MDB_BAD_TXN:
+ case MDB_BAD_VALSIZE:
+#ifdef MDB_BAD_DBI
+ case MDB_BAD_DBI:
+#endif
+ case MDB_PANIC:
+ case EINVAL:
+ return LDB_ERR_PROTOCOL_ERROR;
+ case MDB_MAP_FULL:
+ case MDB_DBS_FULL:
+ case MDB_READERS_FULL:
+ case MDB_TLS_FULL:
+ case MDB_TXN_FULL:
+ case EAGAIN:
+ return LDB_ERR_BUSY;
+ case MDB_KEYEXIST:
+ return LDB_ERR_ENTRY_ALREADY_EXISTS;
+ case MDB_NOTFOUND:
+ case ENOENT:
+ return LDB_ERR_NO_SUCH_OBJECT;
+ case EACCES:
+ return LDB_ERR_INSUFFICIENT_ACCESS_RIGHTS;
+ default:
+ break;
+ }
+ return LDB_ERR_OTHER;
+}
+
+#define ldb_mdb_error(ldb, ecode) lmdb_error_at(ldb, ecode, __FILE__, __LINE__)
+static int lmdb_error_at(struct ldb_context *ldb,
+ int ecode,
+ const char *file,
+ int line)
+{
+ int ldb_err = ldb_mdb_err_map(ecode);
+ char *reason = mdb_strerror(ecode);
+ ldb_asprintf_errstring(ldb,
+ "(%d) - %s at %s:%d",
+ ecode,
+ reason,
+ file,
+ line);
+ return ldb_err;
+}
+
+static MDB_txn *lmdb_trans_get_tx(struct lmdb_trans *ltx)
+{
+ if (ltx == NULL) {
+ return NULL;
+ }
+
+ return ltx->tx;
+}
+
+static void trans_push(struct lmdb_private *lmdb, struct lmdb_trans *ltx)
+{
+ if (lmdb->txlist) {
+ talloc_steal(lmdb->txlist, ltx);
+ }
+
+ DLIST_ADD(lmdb->txlist, ltx);
+}
+
+static void trans_finished(struct lmdb_private *lmdb, struct lmdb_trans *ltx)
+{
+ DLIST_REMOVE(lmdb->txlist, ltx);
+ talloc_free(ltx);
+}
+
+
+static struct lmdb_trans *lmdb_private_trans_head(struct lmdb_private *lmdb)
+{
+ struct lmdb_trans *ltx;
+
+ ltx = lmdb->txlist;
+ return ltx;
+}
+
+static MDB_txn *get_current_txn(struct lmdb_private *lmdb)
+{
+ MDB_txn *txn;
+ if (lmdb->read_txn != NULL) {
+ return lmdb->read_txn;
+ }
+
+ txn = lmdb_trans_get_tx(lmdb_private_trans_head(lmdb));
+ if (txn == NULL) {
+ int ret;
+ ret = mdb_txn_begin(lmdb->env, NULL, MDB_RDONLY, &txn);
+ if (ret != 0) {
+ lmdb->error = ret;
+ ldb_asprintf_errstring(lmdb->ldb,
+ "%s failed: %s\n", __FUNCTION__,
+ mdb_strerror(ret));
+ }
+ lmdb->read_txn = txn;
+ }
+ return txn;
+}
+
+static int lmdb_store(struct ltdb_private *ltdb,
+ TDB_DATA key,
+ TDB_DATA data, int flags)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ MDB_val mdb_key;
+ MDB_val mdb_data;
+ int mdb_flags;
+ MDB_txn *txn = NULL;
+ MDB_dbi dbi = 0;
+
+ txn = lmdb_trans_get_tx(lmdb_private_trans_head(lmdb));
+ if (txn == NULL) {
+ ldb_debug(lmdb->ldb, LDB_DEBUG_FATAL, "No transaction");
+ lmdb->error = MDB_PANIC;
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ lmdb->error = mdb_dbi_open(txn, NULL, 0, &dbi);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ mdb_key.mv_size = key.dsize;
+ mdb_key.mv_data = key.dptr;
+
+ mdb_data.mv_size = data.dsize;
+ mdb_data.mv_data = data.dptr;
+
+ if (flags == TDB_INSERT) {
+ mdb_flags = MDB_NOOVERWRITE;
+ } else if ((flags == TDB_MODIFY)) {
+ /*
+ * Modifying a record, ensure that it exists.
+ * This mimics the TDB semantics
+ */
+ MDB_val value;
+ lmdb->error = mdb_get(txn, dbi, &mdb_key, &value);
+ if (lmdb->error != MDB_SUCCESS) {
+ if (ltdb->read_lock_count == 0 && lmdb->read_txn != NULL) {
+ mdb_txn_commit(lmdb->read_txn);
+ lmdb->read_txn = NULL;
+ }
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+ mdb_flags = 0;
+ } else {
+ mdb_flags = 0;
+ }
+
+ lmdb->error = mdb_put(txn, dbi, &mdb_key, &mdb_data, mdb_flags);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ return ldb_mdb_err_map(lmdb->error);
+}
+
+static int lmdb_exists(struct ltdb_private *ltdb, TDB_DATA key)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ MDB_val mdb_key;
+ MDB_val mdb_data;
+ MDB_txn *txn = NULL;
+ MDB_dbi dbi = 0;
+
+ txn = get_current_txn(lmdb);
+ if (txn == NULL) {
+ ldb_debug(lmdb->ldb, LDB_DEBUG_FATAL, "No transaction");
+ lmdb->error = MDB_PANIC;
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ lmdb->error = mdb_dbi_open(txn, NULL, 0, &dbi);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ mdb_key.mv_size = key.dsize;
+ mdb_key.mv_data = key.dptr;
+
+ lmdb->error = mdb_get(txn, dbi, &mdb_key, &mdb_data);
+ if (ltdb->read_lock_count == 0 && lmdb->read_txn != NULL) {
+ mdb_txn_commit(lmdb->read_txn);
+ lmdb->read_txn = NULL;
+ }
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+ return ldb_mdb_err_map(lmdb->error);
+}
+
+static int lmdb_delete(struct ltdb_private *ltdb, TDB_DATA key)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ MDB_val mdb_key;
+ MDB_txn *txn = NULL;
+ MDB_dbi dbi = 0;
+
+ txn = lmdb_trans_get_tx(lmdb_private_trans_head(lmdb));
+ if (txn == NULL) {
+ ldb_debug(lmdb->ldb, LDB_DEBUG_FATAL, "No transaction");
+ lmdb->error = MDB_PANIC;
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ lmdb->error = mdb_dbi_open(txn, NULL, 0, &dbi);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ mdb_key.mv_size = key.dsize;
+ mdb_key.mv_data = key.dptr;
+
+ lmdb->error = mdb_del(txn, dbi, &mdb_key, NULL);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+ return ldb_mdb_err_map(lmdb->error);
+}
+
+static int lmdb_traverse_fn(struct ltdb_private *ltdb,
+ ldb_kv_traverse_fn fn,
+ void *ctx)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ MDB_val mdb_key;
+ MDB_val mdb_data;
+ MDB_txn *txn = NULL;
+ MDB_dbi dbi = 0;
+ MDB_cursor *cursor = NULL;
+ int ret;
+
+ txn = get_current_txn(lmdb);
+ if (txn == NULL) {
+ ldb_debug(lmdb->ldb, LDB_DEBUG_FATAL, "No transaction");
+ lmdb->error = MDB_PANIC;
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ lmdb->error = mdb_dbi_open(txn, NULL, 0, &dbi);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ lmdb->error = mdb_cursor_open(txn, dbi, &cursor);
+ if (lmdb->error != MDB_SUCCESS) {
+ goto done;
+ }
+
+ while ((lmdb->error = mdb_cursor_get(
+ cursor, &mdb_key,
+ &mdb_data, MDB_NEXT)) == MDB_SUCCESS) {
+
+ struct ldb_val key = {
+ .length = mdb_key.mv_size,
+ .data = mdb_key.mv_data,
+ };
+ struct ldb_val data = {
+ .length = mdb_data.mv_size,
+ .data = mdb_data.mv_data,
+ };
+
+ ret = fn(ltdb, key, data, ctx);
+ if (ret != 0) {
+ goto done;
+ }
+ }
+ if (lmdb->error == MDB_NOTFOUND) {
+ lmdb->error = MDB_SUCCESS;
+ }
+done:
+ if (cursor != NULL) {
+ mdb_cursor_close(cursor);
+ }
+
+ if (ltdb->read_lock_count == 0 && lmdb->read_txn != NULL) {
+ mdb_txn_commit(lmdb->read_txn);
+ lmdb->read_txn = NULL;
+ }
+
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+ return ldb_mdb_err_map(lmdb->error);
+}
+
+static int lmdb_update_in_iterate(struct ltdb_private *ltdb,
+ TDB_DATA key,
+ TDB_DATA key2,
+ TDB_DATA data,
+ void *state)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ struct TDB_DATA copy;
+ int ret = LDB_SUCCESS;
+
+ /*
+ * Need to take a copy of the data as the delete operation alters the
+ * data, as it is in private lmdb memory.
+ */
+ copy.dsize = data.dsize;
+ copy.dptr = talloc_memdup(ltdb, data.dptr, data.dsize);
+ if (copy.dptr == NULL) {
+ lmdb->error = MDB_PANIC;
+ return ldb_oom(lmdb->ldb);
+ }
+
+ lmdb->error = lmdb_delete(ltdb, key);
+ if (lmdb->error != MDB_SUCCESS) {
+ ldb_debug(
+ lmdb->ldb,
+ LDB_DEBUG_ERROR,
+ "Failed to delete %*.*s "
+ "for rekey as %*.*s: %s",
+ (int)key.dsize, (int)key.dsize,
+ (const char *)key.dptr,
+ (int)key2.dsize, (int)key2.dsize,
+ (const char *)key.dptr,
+ mdb_strerror(lmdb->error));
+ ret = ldb_mdb_error(lmdb->ldb, lmdb->error);
+ goto done;
+ }
+ lmdb->error = lmdb_store(ltdb, key2, copy, 0);
+ if (lmdb->error != MDB_SUCCESS) {
+ ldb_debug(
+ lmdb->ldb,
+ LDB_DEBUG_ERROR,
+ "Failed to rekey %*.*s as %*.*s: %s",
+ (int)key.dsize, (int)key.dsize,
+ (const char *)key.dptr,
+ (int)key2.dsize, (int)key2.dsize,
+ (const char *)key.dptr,
+ mdb_strerror(lmdb->error));
+ ret = ldb_mdb_error(lmdb->ldb, lmdb->error);
+ goto done;
+ }
+
+done:
+ if (copy.dptr != NULL) {
+ TALLOC_FREE(copy.dptr);
+ copy.dsize = 0;
+ }
+
+ /*
+ * Explicity invalidate the data, as the delete has done this
+ */
+ data.dsize = 0;
+ data.dptr = NULL;
+
+ return ret;
+}
+/* Handles only a single record */
+static int lmdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
+ int (*parser)(TDB_DATA key, TDB_DATA data,
+ void *private_data),
+ void *ctx)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ MDB_val mdb_key;
+ MDB_val mdb_data;
+ MDB_txn *txn = NULL;
+ MDB_dbi dbi;
+ TDB_DATA data;
+
+ txn = get_current_txn(lmdb);
+ if (txn == NULL) {
+ ldb_debug(lmdb->ldb, LDB_DEBUG_FATAL, "No transaction");
+ lmdb->error = MDB_PANIC;
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ lmdb->error = mdb_dbi_open(txn, NULL, 0, &dbi);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ mdb_key.mv_size = key.dsize;
+ mdb_key.mv_data = key.dptr;
+
+ lmdb->error = mdb_get(txn, dbi, &mdb_key, &mdb_data);
+ if (lmdb->error != MDB_SUCCESS) {
+ /* TODO closing a handle should not even be necessary */
+ mdb_dbi_close(lmdb->env, dbi);
+ if (ltdb->read_lock_count == 0 && lmdb->read_txn != NULL) {
+ mdb_txn_commit(lmdb->read_txn);
+ lmdb->read_txn = NULL;
+ }
+ if (lmdb->error == MDB_NOTFOUND) {
+ return LDB_ERR_NO_SUCH_OBJECT;
+ }
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+ data.dptr = mdb_data.mv_data;
+ data.dsize = mdb_data.mv_size;
+
+ /* TODO closing a handle should not even be necessary */
+ mdb_dbi_close(lmdb->env, dbi);
+
+ /* We created a read transaction, commit it */
+ if (ltdb->read_lock_count == 0 && lmdb->read_txn != NULL) {
+ mdb_txn_commit(lmdb->read_txn);
+ lmdb->read_txn = NULL;
+ }
+ return parser(key, data, ctx);
+}
+
+
+static int lmdb_lock_read(struct ldb_module *module)
+{
+ void *data = ldb_module_get_private(module);
+ struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+
+ lmdb->error = MDB_SUCCESS;
+ if (ltdb->in_transaction == 0 &&
+ ltdb->read_lock_count == 0) {
+ lmdb->error = mdb_txn_begin(lmdb->env,
+ NULL,
+ MDB_RDONLY,
+ &lmdb->read_txn);
+ }
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ ltdb->read_lock_count++;
+ return ldb_mdb_err_map(lmdb->error);
+}
+
+static int lmdb_unlock_read(struct ldb_module *module)
+{
+ void *data = ldb_module_get_private(module);
+ struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
+
+ if (ltdb->in_transaction == 0 && ltdb->read_lock_count == 1) {
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ mdb_txn_commit(lmdb->read_txn);
+ lmdb->read_txn = NULL;
+ ltdb->read_lock_count--;
+ return LDB_SUCCESS;
+ }
+ ltdb->read_lock_count--;
+ return LDB_SUCCESS;
+}
+
+static int lmdb_transaction_start(struct ltdb_private *ltdb)
+{
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+ struct lmdb_trans *ltx;
+ struct lmdb_trans *ltx_head;
+ MDB_txn *tx_parent;
+
+ ltx = talloc_zero(lmdb, struct lmdb_trans);
+ if (ltx == NULL) {
+ return ldb_oom(lmdb->ldb);
+ }
+
+ ltx_head = lmdb_private_trans_head(lmdb);
+
+ tx_parent = lmdb_trans_get_tx(ltx_head);
+
+ lmdb->error = mdb_txn_begin(lmdb->env, tx_parent, 0, <x->tx);
+ if (lmdb->error != MDB_SUCCESS) {
+ return ldb_mdb_error(lmdb->ldb, lmdb->error);
+ }
+
+ trans_push(lmdb, ltx);
+
+ return ldb_mdb_err_map(lmdb->error);
+}
+
+static int lmdb_transaction_cancel(struct ltdb_private *ltdb)
+{
+ struct lmdb_trans *ltx;
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+
+ ltx = lmdb_private_trans_head(lmdb);
+ if (ltx == NULL) {
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
+ mdb_txn_abort(ltx->tx);
+ trans_finished(lmdb, ltx);
+ return LDB_SUCCESS;
+}
+
+static int lmdb_transaction_prepare_commit(struct ltdb_private *ltdb)
+{
+ /* No need to prepare a commit */
+ return LDB_SUCCESS;
+}
+
+static int lmdb_transaction_commit(struct ltdb_private *ltdb)
+{
+ struct lmdb_trans *ltx;
+ struct lmdb_private *lmdb = ltdb->lmdb_private;
+
+ ltx = lmdb_private_trans_head(lmdb);
+ if (ltx == NULL) {
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
+ lmdb->error = mdb_txn_commit(ltx->tx);
+ trans_finished(lmdb, ltx);
+
+ return lmdb->error;
+}
+
+static int lmdb_error(struct ltdb_private *ltdb)
+{
+ return ldb_mdb_err_map(ltdb->lmdb_private->error);
+}
+
+static const char *lmdb_errorstr(struct ltdb_private *ltdb)
+{
+ return mdb_strerror(ltdb->lmdb_private->error);
+}
+
+static const char * lmdb_name(struct ltdb_private *ltdb)
+{
+ return "lmdb";
+}
+
+static bool lmdb_changed(struct ltdb_private *ltdb)
+{
+ /*
+ * lmdb does no provide a quick way to determine if the database
+ * has changed. This function always returns true.
+ *
+ * Note that tdb uses a sequence number that allows this function
+ * to be implemented efficiently.
+ */
+ return true;
+}
+
+
+static struct kv_db_ops lmdb_key_value_ops = {
+ .store = lmdb_store,
+ .delete = lmdb_delete,
+ .exists = lmdb_exists,
+ .iterate = lmdb_traverse_fn,
+ .update_in_iterate = lmdb_update_in_iterate,
+ .fetch_and_parse = lmdb_parse_record,
+ .lock_read = lmdb_lock_read,
+ .unlock_read = lmdb_unlock_read,
+ .begin_write = lmdb_transaction_start,
+ .prepare_write = lmdb_transaction_prepare_commit,
+ .finish_write = lmdb_transaction_commit,
+ .abort_write = lmdb_transaction_cancel,
+ .error = lmdb_error,
+ .errorstr = lmdb_errorstr,
+ .name = lmdb_name,
+ .has_changed = lmdb_changed,
+};
+
+static const char *lmdb_get_path(const char *url)
+{
+ const char *path;
+
+ /* parse the url */
+ if (strchr(url, ':')) {
+ if (strncmp(url, MDB_URL_PREFIX, MDB_URL_PREFIX_SIZE) != 0) {
+ return NULL;
+ }
+ path = url + MDB_URL_PREFIX_SIZE;
+ } else {
+ path = url;
+ }
+
+ return path;
+}
+
+static int lmdb_pvt_destructor(struct lmdb_private *lmdb)
+{
+ struct lmdb_trans *ltx = NULL;
+
+ /*
+ * Close the read transaction if it's open
+ */
+ if (lmdb->read_txn != NULL) {
+ mdb_txn_abort(lmdb->read_txn);
+ }
+
+ if (lmdb->env == NULL) {
+ return 0;
+ }
+
+ /*
+ * Abort any currently active transactions
+ */
+ ltx = lmdb_private_trans_head(lmdb);
+ while (ltx != NULL) {
+ mdb_txn_abort(ltx->tx);
+ trans_finished(lmdb, ltx);
+ ltx = lmdb_private_trans_head(lmdb);
+ }
+
+ mdb_env_close(lmdb->env);
+ lmdb->env = NULL;
+
+ return 0;
+}
+
+static struct lmdb_private *lmdb_pvt_create(TALLOC_CTX *mem_ctx,
+ struct ldb_context *ldb,
+ const char *path)
+{
+ struct lmdb_private *lmdb;
+ int ret;
+
+ lmdb = talloc_zero(ldb, struct lmdb_private);
+ if (lmdb == NULL) {
+ return NULL;
+ }
+ lmdb->ldb = ldb;
+
+ ret = mdb_env_create(&lmdb->env);
+ if (ret != 0) {
+ ldb_asprintf_errstring(ldb,
+ "Could not create MDB environment %s: %s\n",
+ path, mdb_strerror(ret));
+ talloc_free(lmdb);
+ return NULL;
+ }
+
+ /* Close when lmdb is released */
+ talloc_set_destructor(lmdb, lmdb_pvt_destructor);
+
+ ret = mdb_env_set_mapsize(lmdb->env, 16LL * GIGABYTE);
+ if (ret != 0) {
+ talloc_free(lmdb);
+ return NULL;
+ }
+
+ mdb_env_set_maxreaders(lmdb->env, 100000);
+ /* MDB_NOSUBDIR implies there is a separate file called path and a
+ * separate lockfile called path-lock
+ */
+ ret = mdb_env_open(lmdb->env, path, MDB_NOSUBDIR|MDB_NOTLS, 0644);
+ if (ret != 0) {
+ ldb_asprintf_errstring(ldb,
+ "Could not open DB %s: %s\n",
+ path, mdb_strerror(ret));
+ talloc_free(lmdb);
+ return NULL;
+ }
+
+ return lmdb;
+
+}
+
+static int lmdb_connect(struct ldb_context *ldb, const char *url,
+ unsigned int flags, const char *options[],
+ struct ldb_module **_module)
+{
+ const char *path = NULL;
+ struct lmdb_private *lmdb = NULL;
+ struct ltdb_private *ltdb = NULL;
+
+ path = lmdb_get_path(url);
+ if (path == NULL) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR, "Invalid mdb URL '%s'", url);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
+ ltdb = talloc_zero(ldb, struct ltdb_private);
+ if (!ltdb) {
+ ldb_oom(ldb);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+ ltdb->kv_ops = &lmdb_key_value_ops;
+
+ lmdb = lmdb_pvt_create(ldb, ldb, path);
+ if (lmdb == NULL) {
+ ldb_asprintf_errstring(ldb, "Failed to connect to %s with lmdb", path);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
+ ltdb->lmdb_private = lmdb;
+ if (flags & LDB_FLG_RDONLY) {
+ ltdb->read_only = true;
+ }
+ return init_store(ltdb, "ldb_mdb backend", ldb, options, _module);
+}
+
+
+
+int ldb_mdb_init(const char *version)
+{
+ LDB_MODULE_CHECK_VERSION(version);
+ return ldb_register_backend("mdb", lmdb_connect, false);
+}
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.h b/lib/ldb/ldb_mdb/ldb_mdb.h
new file mode 100644
index 0000000..18d28e8
--- /dev/null
+++ b/lib/ldb/ldb_mdb/ldb_mdb.h
@@ -0,0 +1,57 @@
+/*
+ ldb database library using mdb back end - transaction operations
+
+ Copyright (C) Jakub Hrozek 2015
+ Copyright (C) Catalyst.Net Ltd 2017
+
+ ** NOTE! The following LGPL license applies to the ldb
+ ** library. This does NOT imply that all of Samba is released
+ ** under the LGPL
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 3 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, see <http://www.gnu.org/licenses/>.
+*/
+
+#ifndef _LDB_MDB_H_
+#define _LDB_MDB_H_
+
+#include <lmdb.h>
+
+#include "ldb_private.h"
+
+
+struct lmdb_private {
+ struct ldb_context *ldb;
+ MDB_env *env;
+
+ struct lmdb_trans *txlist;
+
+ struct ldb_mdb_metadata {
+ struct ldb_message *attributes;
+ unsigned seqnum;
+ } *meta;
+ int error;
+ MDB_txn *read_txn;
+
+};
+
+struct lmdb_trans {
+ struct lmdb_trans *next;
+ struct lmdb_trans *prev;
+
+ MDB_txn *tx;
+};
+
+int ldb_mdb_err_map(int lmdb_err);
+
+#endif /* _LDB_MDB_H_ */
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 5b3682c..49e0bfc 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -37,6 +37,7 @@ struct kv_db_ops {
struct ltdb_private {
struct kv_db_ops *kv_ops;
TDB_CONTEXT *tdb;
+ struct lmdb_private *lmdb_private;
unsigned int connect_flags;
unsigned long long sequence_number;
diff --git a/lib/ldb/wscript b/lib/ldb/wscript
index 23ca4f4..5c10ada 100644
--- a/lib/ldb/wscript
+++ b/lib/ldb/wscript
@@ -110,6 +110,10 @@ def configure(conf):
if not sys.platform.startswith("openbsd"):
conf.ADD_LDFLAGS('-Wl,-no-undefined', testflags=True)
+ # if lmdb.h is present, build the mdb back end
+ if conf.CHECK_FUNCS_IN('mdb_env_create', 'lmdb', headers='lmdb.h'):
+ conf.env.ENABLE_MDB_BACKEND = True
+
conf.DEFINE('HAVE_CONFIG_H', 1, add_to_cflags=True)
conf.SAMBA_CONFIG_H()
@@ -321,6 +325,15 @@ def build(bld):
private_library=True,
deps='tdb ldb')
+ bld.SAMBA_MODULE('ldb_mdb',
+ bld.SUBDIR('ldb_mdb',
+ '''ldb_mdb.c'''),
+ init_function='ldb_mdb_init',
+ module_init_name='ldb_init_module',
+ internal_module=False,
+ deps='lmdb ldb ldb_key_value',
+ subsystem='ldb')
+
# have a separate subsystem for common/ldb.c, so it can rebuild
# for install with a different -DLDB_MODULESDIR=
bld.SAMBA_SUBSYSTEM('LIBLDB_MAIN',
diff --git a/source4/ldap_server/ldap_backend.c b/source4/ldap_server/ldap_backend.c
index 95c7ee7..eb192c6 100644
--- a/source4/ldap_server/ldap_backend.c
+++ b/source4/ldap_server/ldap_backend.c
@@ -187,8 +187,9 @@ int ldapsrv_backend_Init(struct ldapsrv_connection *conn,
conn->connection->event.ctx,
conn->lp_ctx,
conn->session_info,
- conn->global_catalog ? LDB_FLG_RDONLY : 0,
+ (conn->global_catalog ? LDB_FLG_RDONLY : 0),
"sam.ldb", &conn->ldb, errstring);
+ /* TODO testing LMDB flags */
if (ret != LDB_SUCCESS) {
return ret;
}
--
1.9.1
From f54c8b8a412e011b19e281c22e080abf3c773371 Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Fri, 22 Dec 2017 13:28:47 +1300
Subject: [PATCH 22/34] ldbdump add support for lmdb databases
---
lib/ldb/tools/ldbdump.c | 117 +++++++++++++++++++++++++++++++++++++++++++++++-
lib/ldb/wscript | 6 ++-
2 files changed, 121 insertions(+), 2 deletions(-)
diff --git a/lib/ldb/tools/ldbdump.c b/lib/ldb/tools/ldbdump.c
index c399b59..4b1c2c2 100644
--- a/lib/ldb/tools/ldbdump.c
+++ b/lib/ldb/tools/ldbdump.c
@@ -27,6 +27,11 @@
#include <ldb.h>
#include <ldb_private.h>
+#ifdef HAVE_LMDB
+#include "lmdb.h"
+#endif /* ifdef HAVE_LMDB */
+
+
static struct ldb_context *ldb;
bool show_index = false;
bool validate_contents = false;
@@ -166,6 +171,107 @@ static int dump_tdb(const char *fname, struct ldb_dn *dn, bool emergency)
return tdb_traverse(tdb, traverse_fn, dn) == -1 ? 1 : 0;
}
+#ifdef HAVE_LMDB
+static int dump_lmdb(const char *fname, struct ldb_dn *dn, bool emergency)
+{
+ int ret;
+ struct MDB_env *env = NULL;
+ struct MDB_txn *txn = NULL;
+ MDB_dbi dbi;
+ struct MDB_cursor *cursor = NULL;
+ struct MDB_val key;
+ struct MDB_val data;
+
+ ret = mdb_env_create(&env);
+ if (ret != 0) {
+ fprintf(stderr,
+ "Could not create MDB environment: (%d) %s\n",
+ ret,
+ mdb_strerror(ret));
+ goto close_env;
+ }
+
+ ret = mdb_env_open(env, fname, MDB_NOSUBDIR|MDB_NOTLS|MDB_RDONLY, 0644);
+ if (ret != 0) {
+ fprintf(stderr,
+ "Could not open environment for %s: (%d) %s\n",
+ fname,
+ ret,
+ mdb_strerror(ret));
+ goto close_env;
+ }
+
+ ret = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn);
+ if (ret != 0) {
+ fprintf(stderr,
+ "Could not start transaction: (%d) %s\n",
+ ret,
+ mdb_strerror(ret));
+ goto close_env;
+ }
+
+ ret = mdb_dbi_open(txn, NULL, 0, &dbi);
+ if (ret != 0) {
+ fprintf(stderr,
+ "Could not open database: (%d) %s\n",
+ ret,
+ mdb_strerror(ret));
+ goto close_txn;
+ }
+
+ ret = mdb_cursor_open(txn, dbi, &cursor);
+ if (ret != 0) {
+ fprintf(stderr,
+ "Could not open cursor: (%d) %s\n",
+ ret,
+ mdb_strerror(ret));
+ goto close_txn;
+ }
+
+ ret = mdb_cursor_get(cursor, &key, &data, MDB_FIRST);
+ if (ret != 0 && ret != MDB_NOTFOUND) {
+ fprintf(stderr,
+ "Could not find first record: (%d) %s\n",
+ ret,
+ mdb_strerror(ret));
+ goto close_cursor;
+ }
+ while (ret != MDB_NOTFOUND) {
+ struct TDB_DATA tkey = {
+ .dptr = key.mv_data,
+ .dsize = key.mv_size
+ };
+ struct TDB_DATA tdata = {
+ .dptr = data.mv_data,
+ .dsize = data.mv_size
+ };
+ traverse_fn(NULL, tkey, tdata, dn);
+ ret = mdb_cursor_get(cursor, &key, &data, MDB_NEXT);
+ if (ret != 0 && ret != MDB_NOTFOUND) {
+ fprintf(stderr,
+ "Could not read next record: (%d) %s\n",
+ ret,
+ mdb_strerror(ret));
+ goto close_cursor;
+ }
+ }
+ ret = 0;
+
+close_cursor:
+ mdb_cursor_close(cursor);
+close_txn:
+ mdb_txn_commit(txn);
+close_env:
+ mdb_env_close(env);
+
+ if (ret != 0) {
+ return 1;
+ }
+ return 0;
+
+}
+#endif /* #ifdef HAVE_LMDB */
+
static void usage( void)
{
printf( "Usage: ldbdump [options] <filename>\n\n");
@@ -229,5 +335,14 @@ static void usage( void)
fname = argv[optind];
- return dump_tdb(fname, dn, emergency);
+ rc = dump_lmdb(fname, dn, emergency);
+ if (rc != 0) {
+ rc = dump_tdb(fname, dn, emergency);
+ if (rc != 0) {
+ fprintf(stderr, "Failed to open %s\n", fname);
+ return 1;
+ }
+ }
+ return 0;
+
}
diff --git a/lib/ldb/wscript b/lib/ldb/wscript
index 5c10ada..f750e34 100644
--- a/lib/ldb/wscript
+++ b/lib/ldb/wscript
@@ -111,8 +111,10 @@ def configure(conf):
conf.ADD_LDFLAGS('-Wl,-no-undefined', testflags=True)
# if lmdb.h is present, build the mdb back end
+ # and enable lmdb support in the tools.
if conf.CHECK_FUNCS_IN('mdb_env_create', 'lmdb', headers='lmdb.h'):
conf.env.ENABLE_MDB_BACKEND = True
+ conf.DEFINE('HAVE_LMDB', '1')
conf.DEFINE('HAVE_CONFIG_H', 1, add_to_cflags=True)
@@ -352,7 +354,9 @@ def build(bld):
install=False)
# ldbdump doesn't get installed
- bld.SAMBA_BINARY('ldbdump', 'tools/ldbdump.c', deps='ldb-cmdline ldb',
+ bld.SAMBA_BINARY('ldbdump',
+ 'tools/ldbdump.c',
+ deps='ldb-cmdline ldb lmdb',
install=False)
bld.SAMBA_LIBRARY('ldb-cmdline',
--
1.9.1
From b6a036e071feba70696542b346305c320aeffe46 Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Thu, 11 Jan 2018 12:35:02 +1300
Subject: [PATCH 23/34] ldb: Handle multiple ldb url schemes
Here we use ldb_relative_path more often and allow for the stripping of
ldb:// and mdb://.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
lib/ldb-samba/ldb_wrap.c | 6 ++++++
source4/dsdb/samdb/ldb_modules/partition_metadata.c | 20 +++++++++-----------
source4/dsdb/samdb/ldb_modules/schema_load.c | 19 +++++--------------
3 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/lib/ldb-samba/ldb_wrap.c b/lib/ldb-samba/ldb_wrap.c
index 9959b04..1c01a39 100644
--- a/lib/ldb-samba/ldb_wrap.c
+++ b/lib/ldb-samba/ldb_wrap.c
@@ -361,6 +361,12 @@ int samba_ldb_connect(struct ldb_context *ldb, struct loadparm_context *lp_ctx,
if (strncmp("tdb://", base_url, 6) == 0) {
base_url = base_url+6;
}
+ else if (strncmp("mdb://", base_url, 6) == 0) {
+ base_url = base_url+6;
+ }
+ else if (strncmp("ldb://", base_url, 6) == 0) {
+ base_url = base_url+6;
+ }
path = talloc_strdup(mem_ctx, base_url);
if (path == NULL) {
return NULL;
diff --git a/source4/dsdb/samdb/ldb_modules/partition_metadata.c b/source4/dsdb/samdb/ldb_modules/partition_metadata.c
index 3ba7905..7711ae3 100644
--- a/source4/dsdb/samdb/ldb_modules/partition_metadata.c
+++ b/source4/dsdb/samdb/ldb_modules/partition_metadata.c
@@ -18,6 +18,7 @@
*/
#include "dsdb/samdb/ldb_modules/partition.h"
+#include "lib/ldb-samba/ldb_wrap.h"
#include "system/filesys.h"
#define LDB_METADATA_SEQ_NUM "SEQ_NUM"
@@ -185,7 +186,6 @@ static int partition_metadata_open(struct ldb_module *module, bool create)
TALLOC_CTX *tmp_ctx;
struct partition_private_data *data;
struct loadparm_context *lp_ctx;
- const char *sam_name;
char *filename, *dirname;
int open_flags;
struct stat statbuf;
@@ -202,15 +202,11 @@ static int partition_metadata_open(struct ldb_module *module, bool create)
return ldb_module_oom(module);
}
- sam_name = (const char *)ldb_get_opaque(ldb, "ldb_url");
- if (!sam_name) {
- talloc_free(tmp_ctx);
- return ldb_operr(ldb);
- }
- if (strncmp("tdb://", sam_name, 6) == 0) {
- sam_name += 6;
- }
- filename = talloc_asprintf(tmp_ctx, "%s.d/metadata.tdb", sam_name);
+ /* the backend LDB is the DN (base64 encoded if not 'plain') followed by .ldb */
+ filename = ldb_relative_path(ldb,
+ tmp_ctx,
+ "sam.ldb.d/metadata.tdb");
+
if (!filename) {
talloc_free(tmp_ctx);
return ldb_oom(ldb);
@@ -222,7 +218,9 @@ static int partition_metadata_open(struct ldb_module *module, bool create)
/* While provisioning, sam.ldb.d directory may not exist,
* so create it. Ignore errors, if it already exists. */
- dirname = talloc_asprintf(tmp_ctx, "%s.d", sam_name);
+ dirname = ldb_relative_path(ldb,
+ tmp_ctx,
+ "sam.ldb.d");
if (!dirname) {
talloc_free(tmp_ctx);
return ldb_oom(ldb);
diff --git a/source4/dsdb/samdb/ldb_modules/schema_load.c b/source4/dsdb/samdb/ldb_modules/schema_load.c
index caaeb7b..229f06b 100644
--- a/source4/dsdb/samdb/ldb_modules/schema_load.c
+++ b/source4/dsdb/samdb/ldb_modules/schema_load.c
@@ -32,6 +32,7 @@
#include <tdb.h>
#include "lib/tdb_wrap/tdb_wrap.h"
#include "dsdb/samdb/ldb_modules/util.h"
+#include "lib/ldb-samba/ldb_wrap.h"
#include "system/filesys.h"
struct schema_load_private_data {
@@ -58,7 +59,6 @@ static int schema_metadata_open(struct ldb_module *module)
struct ldb_context *ldb = ldb_module_get_ctx(module);
TALLOC_CTX *tmp_ctx;
struct loadparm_context *lp_ctx;
- const char *sam_name;
char *filename;
int open_flags;
struct stat statbuf;
@@ -74,19 +74,10 @@ static int schema_metadata_open(struct ldb_module *module)
return ldb_module_oom(module);
}
- sam_name = (const char *)ldb_get_opaque(ldb, "ldb_url");
- if (!sam_name) {
- talloc_free(tmp_ctx);
- return ldb_operr(ldb);
- }
- if (strncmp("tdb://", sam_name, 6) == 0) {
- sam_name += 6;
- }
- filename = talloc_asprintf(tmp_ctx, "%s.d/metadata.tdb", sam_name);
- if (!filename) {
- talloc_free(tmp_ctx);
- return ldb_oom(ldb);
- }
+ /* the backend LDB is the DN (base64 encoded if not 'plain') followed by .ldb */
+ filename = ldb_relative_path(ldb,
+ tmp_ctx,
+ "sam.ldb.d/metadata.tdb");
open_flags = O_RDWR;
if (stat(filename, &statbuf) != 0) {
--
1.9.1
From 9ae0bf86c6242d34542dce18f6fbae0e3ed619a3 Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Thu, 11 Jan 2018 12:32:06 +1300
Subject: [PATCH 24/34] ldb: Introduce new ldb:// prefix to allow failover
If you try to open with tdb first you find that you can clobber a
database due to the default tdb behaviour.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_ldb/ldb_ldb.c | 56 ++++++++++++++++++++++++++++++
lib/ldb/ldb_mdb/ldb_mdb.c | 79 +++++++++++++++++++++++++-----------------
lib/ldb/ldb_mdb/ldb_mdb.h | 3 ++
lib/ldb/ldb_mdb/ldb_mdb_init.c | 31 +++++++++++++++++
lib/ldb/wscript | 19 ++++++++--
5 files changed, 154 insertions(+), 34 deletions(-)
create mode 100644 lib/ldb/ldb_ldb/ldb_ldb.c
create mode 100644 lib/ldb/ldb_mdb/ldb_mdb_init.c
diff --git a/lib/ldb/ldb_ldb/ldb_ldb.c b/lib/ldb/ldb_ldb/ldb_ldb.c
new file mode 100644
index 0000000..4d78e78
--- /dev/null
+++ b/lib/ldb/ldb_ldb/ldb_ldb.c
@@ -0,0 +1,56 @@
+
+#include "ldb_private.h"
+#include "../ldb_tdb/ldb_tdb.h"
+#include "../ldb_mdb/ldb_mdb.h"
+
+/*
+ connect to the database
+*/
+static int lldb_connect(struct ldb_context *ldb,
+ const char *url,
+ unsigned int flags,
+ const char *options[],
+ struct ldb_module **module)
+{
+ const char *path;
+ int ret;
+
+ /*
+ * Check and remove the url prefix
+ */
+ if (strchr(url, ':')) {
+ if (strncmp(url, "ldb://", 6) != 0) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR,
+ "Invalid ldb URL '%s'", url);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+ path = url+6;
+ } else {
+ path = url;
+ }
+
+ /*
+ * Don't create the database if it's not there
+ */
+ flags |= LDB_FLG_DONT_CREATE_DB;
+ /*
+ * Try opening the database as an lmdb
+ */
+ ret = lmdb_connect(ldb, path, flags, options, module);
+ if (ret == LDB_SUCCESS) {
+ return ret;
+ }
+ if (ret == LDB_ERR_UNAVAILABLE) {
+ /*
+ * Not mbd to try as tdb
+ */
+ ret = ltdb_connect(ldb, path, flags, options, module);
+ }
+ return ret;
+}
+
+int ldb_ldb_init(const char *version)
+{
+ LDB_MODULE_CHECK_VERSION(version);
+ return ldb_register_backend("ldb", lldb_connect, false);
+}
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
index e9308dd..71c2aa2 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.c
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -648,26 +648,30 @@ static int lmdb_pvt_destructor(struct lmdb_private *lmdb)
return 0;
}
-static struct lmdb_private *lmdb_pvt_create(TALLOC_CTX *mem_ctx,
- struct ldb_context *ldb,
- const char *path)
+static int lmdb_pvt_open(TALLOC_CTX *mem_ctx,
+ struct ldb_context *ldb,
+ const char *path,
+ unsigned int flags,
+ struct lmdb_private *lmdb)
{
- struct lmdb_private *lmdb;
int ret;
+ unsigned int mdb_flags;
- lmdb = talloc_zero(ldb, struct lmdb_private);
- if (lmdb == NULL) {
- return NULL;
+ if (flags & LDB_FLG_DONT_CREATE_DB) {
+ struct stat st;
+ if (stat(path, &st) != 0) {
+ return LDB_ERR_UNAVAILABLE;
+ }
}
- lmdb->ldb = ldb;
ret = mdb_env_create(&lmdb->env);
if (ret != 0) {
- ldb_asprintf_errstring(ldb,
- "Could not create MDB environment %s: %s\n",
- path, mdb_strerror(ret));
- talloc_free(lmdb);
- return NULL;
+ ldb_asprintf_errstring(
+ ldb,
+ "Could not create MDB environment %s: %s\n",
+ path,
+ mdb_strerror(ret));
+ return LDB_ERR_OPERATIONS_ERROR;
}
/* Close when lmdb is released */
@@ -675,34 +679,45 @@ static struct lmdb_private *lmdb_pvt_create(TALLOC_CTX *mem_ctx,
ret = mdb_env_set_mapsize(lmdb->env, 16LL * GIGABYTE);
if (ret != 0) {
- talloc_free(lmdb);
- return NULL;
+ ldb_asprintf_errstring(
+ ldb,
+ "Could not open MDB environment %s: %s\n",
+ path,
+ mdb_strerror(ret));
+ return ldb_mdb_err_map(ret);
}
mdb_env_set_maxreaders(lmdb->env, 100000);
/* MDB_NOSUBDIR implies there is a separate file called path and a
* separate lockfile called path-lock
*/
- ret = mdb_env_open(lmdb->env, path, MDB_NOSUBDIR|MDB_NOTLS, 0644);
+ mdb_flags = MDB_NOSUBDIR|MDB_NOTLS;
+ if (flags & LDB_FLG_RDONLY) {
+ mdb_flags |= MDB_RDONLY;
+ }
+ ret = mdb_env_open(lmdb->env, path, mdb_flags, 0644);
if (ret != 0) {
ldb_asprintf_errstring(ldb,
"Could not open DB %s: %s\n",
path, mdb_strerror(ret));
talloc_free(lmdb);
- return NULL;
+ return ldb_mdb_err_map(ret);
}
- return lmdb;
+ return LDB_SUCCESS;
}
-static int lmdb_connect(struct ldb_context *ldb, const char *url,
- unsigned int flags, const char *options[],
- struct ldb_module **_module)
+int lmdb_connect(struct ldb_context *ldb,
+ const char *url,
+ unsigned int flags,
+ const char *options[],
+ struct ldb_module **_module)
{
const char *path = NULL;
struct lmdb_private *lmdb = NULL;
struct ltdb_private *ltdb = NULL;
+ int ret;
path = lmdb_get_path(url);
if (path == NULL) {
@@ -715,12 +730,19 @@ static int lmdb_connect(struct ldb_context *ldb, const char *url,
ldb_oom(ldb);
return LDB_ERR_OPERATIONS_ERROR;
}
- ltdb->kv_ops = &lmdb_key_value_ops;
- lmdb = lmdb_pvt_create(ldb, ldb, path);
+ lmdb = talloc_zero(ldb, struct lmdb_private);
if (lmdb == NULL) {
- ldb_asprintf_errstring(ldb, "Failed to connect to %s with lmdb", path);
- return LDB_ERR_OPERATIONS_ERROR;
+ TALLOC_FREE(ltdb);
+ ldb_oom(ldb);
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+ lmdb->ldb = ldb;
+ ltdb->kv_ops = &lmdb_key_value_ops;
+
+ ret = lmdb_pvt_open(ldb, ldb, path, flags, lmdb);
+ if (ret != LDB_SUCCESS) {
+ return ret;
}
ltdb->lmdb_private = lmdb;
@@ -730,10 +752,3 @@ static int lmdb_connect(struct ldb_context *ldb, const char *url,
return init_store(ltdb, "ldb_mdb backend", ldb, options, _module);
}
-
-
-int ldb_mdb_init(const char *version)
-{
- LDB_MODULE_CHECK_VERSION(version);
- return ldb_register_backend("mdb", lmdb_connect, false);
-}
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.h b/lib/ldb/ldb_mdb/ldb_mdb.h
index 18d28e8..e62e353 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.h
+++ b/lib/ldb/ldb_mdb/ldb_mdb.h
@@ -53,5 +53,8 @@ struct lmdb_trans {
};
int ldb_mdb_err_map(int lmdb_err);
+int lmdb_connect(struct ldb_context *ldb, const char *url,
+ unsigned int flags, const char *options[],
+ struct ldb_module **_module);
#endif /* _LDB_MDB_H_ */
diff --git a/lib/ldb/ldb_mdb/ldb_mdb_init.c b/lib/ldb/ldb_mdb/ldb_mdb_init.c
new file mode 100644
index 0000000..339c3f2
--- /dev/null
+++ b/lib/ldb/ldb_mdb/ldb_mdb_init.c
@@ -0,0 +1,31 @@
+/*
+ ldb database library using mdb back end
+
+ Copyright (C) Jakub Hrozek 2014
+ Copyright (C) Catalyst.Net Ltd 2017
+
+ ** NOTE! The following LGPL license applies to the ldb
+ ** library. This does NOT imply that all of Samba is released
+ ** under the LGPL
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 3 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include "ldb_mdb.h"
+
+int ldb_mdb_init(const char *version)
+{
+ LDB_MODULE_CHECK_VERSION(version);
+ return ldb_register_backend("mdb", lmdb_connect, false);
+}
diff --git a/lib/ldb/wscript b/lib/ldb/wscript
index f750e34..e9579b8 100644
--- a/lib/ldb/wscript
+++ b/lib/ldb/wscript
@@ -329,11 +329,26 @@ def build(bld):
bld.SAMBA_MODULE('ldb_mdb',
bld.SUBDIR('ldb_mdb',
- '''ldb_mdb.c'''),
+ '''ldb_mdb_init.c'''),
init_function='ldb_mdb_init',
module_init_name='ldb_init_module',
internal_module=False,
- deps='lmdb ldb ldb_key_value',
+ deps='ldb ldb_key_value ldb_mdb_int',
+ subsystem='ldb')
+
+ bld.SAMBA_LIBRARY('ldb_mdb_int',
+ bld.SUBDIR('ldb_mdb',
+ '''ldb_mdb.c '''),
+ private_library=True,
+ deps='ldb lmdb ldb_key_value')
+
+ bld.SAMBA_MODULE('ldb_ldb',
+ bld.SUBDIR('ldb_ldb',
+ '''ldb_ldb.c'''),
+ init_function='ldb_ldb_init',
+ module_init_name='ldb_init_module',
+ internal_module=False,
+ deps='ldb ldb_key_value ldb_mdb_int',
subsystem='ldb')
# have a separate subsystem for common/ldb.c, so it can rebuild
--
1.9.1
From 59fe3aac8917994689e5243a1b3b8568371c28d9 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Fri, 5 Jan 2018 09:28:54 +1300
Subject: [PATCH 25/34] tests: Replace some references to tdb with ldb://
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
python/samba/tests/encrypted_secrets.py | 2 +-
source4/torture/drs/python/samba_tool_drs.py | 10 +++++-----
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/python/samba/tests/encrypted_secrets.py b/python/samba/tests/encrypted_secrets.py
index 3b6934f..114f3b9 100644
--- a/python/samba/tests/encrypted_secrets.py
+++ b/python/samba/tests/encrypted_secrets.py
@@ -54,7 +54,7 @@ class EncryptedSecretsTests(TestCase):
backend_subpath = os.path.join("sam.ldb.d",
backend_filename)
backend_path = self.lp.private_path(backend_subpath)
- backenddb = ldb.Ldb(backend_path)
+ backenddb = ldb.Ldb("ldb://" + backend_path, flags=ldb.FLG_DONT_CREATE_DB)
dn = "CN=Administrator,CN=Users,%s" % basedn
diff --git a/source4/torture/drs/python/samba_tool_drs.py b/source4/torture/drs/python/samba_tool_drs.py
index fcc681d..502c809 100644
--- a/source4/torture/drs/python/samba_tool_drs.py
+++ b/source4/torture/drs/python/samba_tool_drs.py
@@ -282,7 +282,7 @@ class SambaToolDrsTests(drs_base.DrsBaseTestCase):
self.dc1,
self.cmdline_creds,
self.tempdir))
- ldb_rootdse = self._get_rootDSE("tdb://" + os.path.join(self.tempdir, "private", "sam.ldb"), ldap_only=False)
+ ldb_rootdse = self._get_rootDSE("ldb://" + os.path.join(self.tempdir, "private", "sam.ldb"), ldap_only=False)
nc_name = ldb_rootdse["defaultNamingContext"]
ds_name = ldb_rootdse["dsServiceName"]
ldap_service_name = str(server_rootdse["ldapServiceName"][0])
@@ -291,7 +291,7 @@ class SambaToolDrsTests(drs_base.DrsBaseTestCase):
self.assertEqual(ds_name, server_ds_name)
self.assertEqual(ldap_service_name, server_ldap_service_name)
- samdb = samba.tests.connect_samdb("tdb://" + os.path.join(self.tempdir, "private", "sam.ldb"),
+ samdb = samba.tests.connect_samdb("ldb://" + os.path.join(self.tempdir, "private", "sam.ldb"),
ldap_only=False, lp=self.get_loadparm())
def get_krbtgt_pw():
krbtgt_pw = samdb.searchone("unicodePwd", "cn=krbtgt,CN=users,%s" % nc_name)
@@ -346,13 +346,13 @@ class SambaToolDrsTests(drs_base.DrsBaseTestCase):
self.dc1,
self.cmdline_creds,
self.tempdir))
- ldb_rootdse = self._get_rootDSE("tdb://" + os.path.join(self.tempdir, "private", "sam.ldb"), ldap_only=False)
+ ldb_rootdse = self._get_rootDSE("ldb://" + os.path.join(self.tempdir, "private", "sam.ldb"), ldap_only=False)
nc_name = ldb_rootdse["defaultNamingContext"]
config_nc_name = ldb_rootdse["configurationNamingContext"]
ds_name = ldb_rootdse["dsServiceName"]
ldap_service_name = str(server_rootdse["ldapServiceName"][0])
- samdb = samba.tests.connect_samdb("tdb://" + os.path.join(self.tempdir, "private", "sam.ldb"),
+ samdb = samba.tests.connect_samdb("ldb://" + os.path.join(self.tempdir, "private", "sam.ldb"),
ldap_only=False, lp=self.get_loadparm())
krbtgt_pw = samdb.searchone("unicodePwd", "cn=krbtgt,CN=users,%s" % nc_name)
self.assertIsNotNone(krbtgt_pw)
@@ -381,7 +381,7 @@ class SambaToolDrsTests(drs_base.DrsBaseTestCase):
self.assertRaises(samba.tests.BlackboxProcessError, demote_self)
# While we have this cloned, try demoting the other server on the clone
- out = self.check_output("samba-tool domain demote --remove-other-dead-server=%s -H %s/private/sam.ldb"
+ out = self.check_output("samba-tool domain demote --remove-other-dead-server=%s -H ldb://%s/private/sam.ldb"
% (self.dc2,
self.tempdir))
--
1.9.1
From c2070779246802377eaf1a72d36d1b0d8b7b784e Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Thu, 11 Jan 2018 12:40:01 +1300
Subject: [PATCH 26/34] tests/dlz_bind9: support for multiple db types
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
source4/torture/dns/dlz_bind9.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source4/torture/dns/dlz_bind9.c b/source4/torture/dns/dlz_bind9.c
index 2234e7a..42b104e 100644
--- a/source4/torture/dns/dlz_bind9.c
+++ b/source4/torture/dns/dlz_bind9.c
@@ -58,7 +58,7 @@ static char *test_dlz_bind9_binddns_dir(struct torture_context *tctx,
const char *file)
{
return talloc_asprintf(tctx,
- "%s/%s",
+ "ldb://%s/%s",
lpcfg_binddns_dir(tctx->lp_ctx),
file);
}
--
1.9.1
From e59b2973ace760e9063f0e52fdddb6d1c50a5271 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Mon, 8 Jan 2018 08:56:16 +1300
Subject: [PATCH 27/34] provision: Allow provisioning of a different database
backend
This sets the backendStore field in @PARTITION, depending on which
argument you set in the provision.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
python/samba/netcmd/domain.py | 13 ++++++++---
python/samba/provision/__init__.py | 41 ++++++++++++++++++++++++---------
python/samba/provision/backend.py | 2 +-
python/samba/provision/sambadns.py | 26 +++++++++++++++------
python/samba/samdb.py | 3 +++
source4/scripting/bin/samba_upgradedns | 3 +--
source4/setup/provision_partitions.ldif | 1 +
7 files changed, 65 insertions(+), 24 deletions(-)
diff --git a/python/samba/netcmd/domain.py b/python/samba/netcmd/domain.py
index 52c70ac..936b669 100644
--- a/python/samba/netcmd/domain.py
+++ b/python/samba/netcmd/domain.py
@@ -42,7 +42,7 @@ from samba.net import Net, LIBNET_JOIN_AUTOMATIC
import samba.ntacls
from samba.join import join_RODC, join_DC, join_subdomain
from samba.auth import system_session
-from samba.samdb import SamDB
+from samba.samdb import SamDB, get_default_backend_store
from samba.ndr import ndr_unpack, ndr_pack, ndr_print
from samba.dcerpc import drsuapi
from samba.dcerpc import drsblobs
@@ -257,6 +257,9 @@ class cmd_domain_provision(Command):
Option("--plaintext-secrets", action="store_true",
help="Store secret/sensitive values as plain text on disk" +
"(default is to encrypt secret/ensitive values)"),
+ Option("--backend-store", type="choice", metavar="BACKENDSTORE",
+ choices=["tdb", "mdb"],
+ help="Specify the database backend to be used TODO better words"),
]
openldap_options = [
@@ -327,7 +330,8 @@ class cmd_domain_provision(Command):
ldap_backend_forced_uri=None,
ldap_dryrun_mode=None,
base_schema=None,
- plaintext_secrets=False):
+ plaintext_secrets=False,
+ backend_store=None):
self.logger = self.get_logger("provision")
if quiet:
@@ -475,6 +479,8 @@ class cmd_domain_provision(Command):
domain_sid = security.dom_sid(domain_sid)
session = system_session()
+ if backend_store is None:
+ backend_store = get_default_backend_store()
try:
result = provision(self.logger,
session, smbconf=smbconf, targetdir=targetdir,
@@ -497,7 +503,8 @@ class cmd_domain_provision(Command):
ldap_backend_forced_uri=ldap_backend_forced_uri,
nosync=ldap_backend_nosync, ldap_dryrun_mode=ldap_dryrun_mode,
base_schema=base_schema,
- plaintext_secrets=plaintext_secrets)
+ plaintext_secrets=plaintext_secrets,
+ backend_store=backend_store)
except ProvisioningError as e:
raise CommandError("Provision failed", e)
diff --git a/python/samba/provision/__init__.py b/python/samba/provision/__init__.py
index 2a926bb..8096f8b 100644
--- a/python/samba/provision/__init__.py
+++ b/python/samba/provision/__init__.py
@@ -123,6 +123,7 @@ from samba.schema import Schema
from samba.samdb import SamDB
from samba.dbchecker import dbcheck
from samba.provision.kerberos import create_kdc_conf
+from samba.samdb import get_default_backend_store
DEFAULT_POLICY_GUID = "31B2F340-016D-11D2-945F-00C04FB984F9"
DEFAULT_DC_POLICY_GUID = "6AC1786C-016F-11D2-945F-00C04FB984F9"
@@ -797,7 +798,8 @@ def setup_name_mappings(idmap, sid, root_uid, nobody_uid,
def setup_samdb_partitions(samdb_path, logger, lp, session_info,
provision_backend, names, serverrole,
- erase=False, plaintext_secrets=False):
+ erase=False, plaintext_secrets=False,
+ backend_store=None):
"""Setup the partitions for the SAM database.
Alternatively, provision() may call this, and then populate the database.
@@ -830,11 +832,16 @@ def setup_samdb_partitions(samdb_path, logger, lp, session_info,
if not plaintext_secrets:
required_features = "requiredFeatures: encryptedSecrets"
+ if backend_store is None:
+ backend_store = get_default_backend_store()
+ backend_store_line = "backendStore: %s" % backend_store
+
samdb.transaction_start()
try:
logger.info("Setting up sam.ldb partitions and settings")
setup_add_ldif(samdb, setup_path("provision_partitions.ldif"), {
- "LDAP_BACKEND_LINE": ldap_backend_line
+ "LDAP_BACKEND_LINE": ldap_backend_line,
+ "BACKEND_STORE": backend_store_line
})
@@ -1228,7 +1235,7 @@ def create_default_gpo(sysvolpath, dnsdomain, policyguid, policyguid_dc):
def setup_samdb(path, session_info, provision_backend, lp, names,
logger, fill, serverrole, schema, am_rodc=False,
- plaintext_secrets=False):
+ plaintext_secrets=False, backend_store=None):
"""Setup a complete SAM Database.
:note: This will wipe the main SAM database file!
@@ -1237,7 +1244,8 @@ def setup_samdb(path, session_info, provision_backend, lp, names,
# Also wipes the database
setup_samdb_partitions(path, logger=logger, lp=lp,
provision_backend=provision_backend, session_info=session_info,
- names=names, serverrole=serverrole, plaintext_secrets=plaintext_secrets)
+ names=names, serverrole=serverrole, plaintext_secrets=plaintext_secrets,
+ backend_store=backend_store)
# Load the database, but don's load the global schema and don't connect
# quite yet
@@ -1275,7 +1283,8 @@ def setup_samdb(path, session_info, provision_backend, lp, names,
def fill_samdb(samdb, lp, names, logger, policyguid,
policyguid_dc, fill, adminpass, krbtgtpass, machinepass, dns_backend,
dnspass, invocationid, ntdsguid, serverrole, am_rodc=False,
- dom_for_fun_level=None, schema=None, next_rid=None, dc_rid=None):
+ dom_for_fun_level=None, schema=None, next_rid=None, dc_rid=None,
+ backend_store=None):
if next_rid is None:
next_rid = 1000
@@ -1813,7 +1822,8 @@ def provision_fill(samdb, secrets_ldb, logger, names, paths,
invocationid=None, machinepass=None, ntdsguid=None,
dns_backend=None, dnspass=None,
serverrole=None, dom_for_fun_level=None,
- am_rodc=False, lp=None, use_ntvfs=False, skip_sysvolacl=False):
+ am_rodc=False, lp=None, use_ntvfs=False,
+ skip_sysvolacl=False, backend_store=None):
# create/adapt the group policy GUIDs
# Default GUID for default policy are described at
# "How Core Group Policy Works"
@@ -1845,7 +1855,8 @@ def provision_fill(samdb, secrets_ldb, logger, names, paths,
dns_backend=dns_backend, dnspass=dnspass,
ntdsguid=ntdsguid, serverrole=serverrole,
dom_for_fun_level=dom_for_fun_level, am_rodc=am_rodc,
- next_rid=next_rid, dc_rid=dc_rid)
+ next_rid=next_rid, dc_rid=dc_rid,
+ backend_store=backend_store)
# Set up group policies (domain policy and domain controller
# policy)
@@ -1894,7 +1905,8 @@ def provision_fill(samdb, secrets_ldb, logger, names, paths,
setup_ad_dns(samdb, secrets_ldb, names, paths, lp, logger,
hostip=hostip, hostip6=hostip6, dns_backend=dns_backend,
dnspass=dnspass, os_level=dom_for_fun_level,
- targetdir=targetdir, fill_level=samdb_fill)
+ targetdir=targetdir, fill_level=samdb_fill,
+ backend_store=backend_store)
domainguid = samdb.searchone(basedn=samdb.get_default_basedn(),
attribute="objectGUID")
@@ -1999,6 +2011,9 @@ def directory_create_or_exists(path, mode=0o755):
else:
raise ProvisioningError("Failed to create directory %s: %s" % (path, e.strerror))
+# TODO
+# Decide and set the correct backend type, currently mdb for testing
+# but should probably be tdb for release.
def provision(logger, session_info, smbconf=None,
targetdir=None, samdb_fill=FILL_FULL, realm=None, rootdn=None,
domaindn=None, schemadn=None, configdn=None, serverdn=None,
@@ -2014,7 +2029,7 @@ def provision(logger, session_info, smbconf=None,
use_rfc2307=False, maxuid=None, maxgid=None, skip_sysvolacl=True,
ldap_backend_forced_uri=None, nosync=False, ldap_dryrun_mode=False,
ldap_backend_extra_port=None, base_schema=None,
- plaintext_secrets=False):
+ plaintext_secrets=False, backend_store=None):
"""Provision samba4
:note: caution, this wipes all existing data!
@@ -2031,6 +2046,8 @@ def provision(logger, session_info, smbconf=None,
if backend_type is None:
backend_type = "ldb"
+ if backend_store is None:
+ backend_store = get_default_backend_store()
if domainsid is None:
domainsid = security.random_sid()
@@ -2214,7 +2231,8 @@ def provision(logger, session_info, smbconf=None,
provision_backend, lp, names, logger=logger,
serverrole=serverrole,
schema=schema, fill=samdb_fill, am_rodc=am_rodc,
- plaintext_secrets=plaintext_secrets)
+ plaintext_secrets=plaintext_secrets,
+ backend_store=backend_store)
if serverrole == "active directory domain controller":
if paths.netlogon is None:
@@ -2245,7 +2263,8 @@ def provision(logger, session_info, smbconf=None,
dnspass=dnspass, serverrole=serverrole,
dom_for_fun_level=dom_for_fun_level, am_rodc=am_rodc,
lp=lp, use_ntvfs=use_ntvfs,
- skip_sysvolacl=skip_sysvolacl)
+ skip_sysvolacl=skip_sysvolacl,
+ backend_store=backend_store)
if not is_heimdal_built():
create_kdc_conf(paths.kdcconf, realm, domain, os.path.dirname(lp.get("log file")))
diff --git a/python/samba/provision/backend.py b/python/samba/provision/backend.py
index 2dbd4f7..d0c2cce 100644
--- a/python/samba/provision/backend.py
+++ b/python/samba/provision/backend.py
@@ -145,7 +145,7 @@ class ExistingBackend(ProvisionBackend):
def init(self):
# Check to see that this 'existing' LDAP backend in fact exists
- ldapi_db = Ldb(self.ldapi_uri)
+ ldapi_db = Ldb(self.ldapi_uri, flags=Ldb.FLG_DONT_CREATE_DB)
ldapi_db.search(base="", scope=SCOPE_BASE,
expression="(objectClass=OpenLDAProotDSE)")
diff --git a/python/samba/provision/sambadns.py b/python/samba/provision/sambadns.py
index 3a1df89..6140bf3 100644
--- a/python/samba/provision/sambadns.py
+++ b/python/samba/provision/sambadns.py
@@ -58,6 +58,7 @@ from samba.provision.common import (
FILL_DRS,
)
+from samba.samdb import get_default_backend_store
def get_domainguid(samdb, domaindn):
res = samdb.search(base=domaindn, scope=ldb.SCOPE_BASE, attrs=["objectGUID"])
@@ -787,12 +788,19 @@ def create_samdb_copy(samdb, logger, paths, names, domainsid, domainguid):
# Find the partitions and corresponding filenames
partfile = {}
- res = samdb.search(base="@PARTITION", scope=ldb.SCOPE_BASE, attrs=["partition"])
+ res = samdb.search(base="@PARTITION",
+ scope=ldb.SCOPE_BASE,
+ attrs=["partition", "backendStore"])
for tmp in res[0]["partition"]:
(nc, fname) = tmp.split(':')
partfile[nc.upper()] = fname
+ backend_store = get_default_backend_store()
+ if "backendStore" in res[0]:
+ backend_store = res[0]["backendStore"]
+
# Create empty domain partition
+
domaindn = names.domaindn.upper()
domainpart_file = os.path.join(dns_dir, partfile[domaindn])
try:
@@ -800,7 +808,8 @@ def create_samdb_copy(samdb, logger, paths, names, domainsid, domainguid):
file(domainpart_file, 'w').close()
# Fill the basedn and @OPTION records in domain partition
- dom_ldb = samba.Ldb(domainpart_file)
+ dom_url = "%s://%s" % (backend_store, domainpart_file)
+ dom_ldb = samba.Ldb(dom_url)
domainguid_line = "objectGUID: %s\n-" % domainguid
descr = b64encode(get_domain_descriptor(domainsid))
setup_add_ldif(dom_ldb, setup_path("provision_basedn.ldif"), {
@@ -875,7 +884,7 @@ def create_samdb_copy(samdb, logger, paths, names, domainsid, domainguid):
os.chown(dpath, -1, paths.bind_gid)
os.chmod(dpath, 0770)
for f in files:
- if f.endswith('.ldb') or f.endswith('.tdb'):
+ if f.endswith(('.ldb', '.tdb', 'ldb-lock')):
fpath = os.path.join(dirname, f)
os.chown(fpath, -1, paths.bind_gid)
os.chmod(fpath, 0660)
@@ -1069,7 +1078,7 @@ def fill_dns_data_partitions(samdb, domainsid, site, domaindn, forestdn,
def setup_ad_dns(samdb, secretsdb, names, paths, lp, logger,
dns_backend, os_level, dnspass=None, hostip=None, hostip6=None,
- targetdir=None, fill_level=FILL_FULL):
+ targetdir=None, fill_level=FILL_FULL, backend_store="mdb"):
"""Provision DNS information (assuming GC role)
:param samdb: LDB object connected to sam.ldb file
@@ -1164,12 +1173,14 @@ def setup_ad_dns(samdb, secretsdb, names, paths, lp, logger,
if dns_backend.startswith("BIND9_"):
setup_bind9_dns(samdb, secretsdb, names, paths, lp, logger,
dns_backend, os_level, site=site, dnspass=dnspass, hostip=hostip,
- hostip6=hostip6, targetdir=targetdir)
+ hostip6=hostip6, targetdir=targetdir,
+ backend_store=backend_store)
def setup_bind9_dns(samdb, secretsdb, names, paths, lp, logger,
dns_backend, os_level, site=None, dnspass=None, hostip=None,
- hostip6=None, targetdir=None, key_version_number=None):
+ hostip6=None, targetdir=None, key_version_number=None,
+ backend_store=None):
"""Provision DNS information (assuming BIND9 backend in DC role)
:param samdb: LDB object connected to sam.ldb file
@@ -1216,7 +1227,8 @@ def setup_bind9_dns(samdb, secretsdb, names, paths, lp, logger,
ntdsguid=names.ntdsguid)
if dns_backend == "BIND9_DLZ" and os_level >= DS_DOMAIN_FUNCTION_2003:
- create_samdb_copy(samdb, logger, paths, names, names.domainsid, domainguid)
+ create_samdb_copy(samdb, logger, paths,
+ names, names.domainsid, domainguid)
create_named_conf(paths, realm=names.realm,
dnsdomain=names.dnsdomain, dns_backend=dns_backend,
diff --git a/python/samba/samdb.py b/python/samba/samdb.py
index 89014a5..c51864b 100644
--- a/python/samba/samdb.py
+++ b/python/samba/samdb.py
@@ -36,6 +36,9 @@ from samba.compat import text_type
__docformat__ = "restructuredText"
+def get_default_backend_store():
+ return "tdb"
+
class SamDB(samba.Ldb):
"""The SAM database."""
diff --git a/source4/scripting/bin/samba_upgradedns b/source4/scripting/bin/samba_upgradedns
index db15b65..3a43577 100755
--- a/source4/scripting/bin/samba_upgradedns
+++ b/source4/scripting/bin/samba_upgradedns
@@ -533,8 +533,7 @@ if __name__ == '__main__':
create_dns_dir(logger, paths)
# Setup a copy of SAM for BIND9
- create_samdb_copy(ldbs.sam, logger, paths, names, domainsid,
- domainguid)
+ create_samdb_copy(ldbs.sam, logger, paths, names, domainsid, domainguid)
create_named_conf(paths, names.realm, dnsdomain, opts.dns_backend, logger)
diff --git a/source4/setup/provision_partitions.ldif b/source4/setup/provision_partitions.ldif
index 728f327..4cd5794 100644
--- a/source4/setup/provision_partitions.ldif
+++ b/source4/setup/provision_partitions.ldif
@@ -2,5 +2,6 @@ dn: @PARTITION
replicateEntries: @ATTRIBUTES
replicateEntries: @INDEXLIST
replicateEntries: @OPTIONS
+${BACKEND_STORE}
${LDAP_BACKEND_LINE}
--
1.9.1
From 2689df71039af54334e86ccb221b30f6b791873b Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Wed, 7 Feb 2018 11:10:34 +1300
Subject: [PATCH 28/34] dsdb: add lmdbLevelZero as a required feature.
---
python/samba/provision/__init__.py | 12 +++++++++++-
source4/dsdb/samdb/ldb_modules/samba_dsdb.c | 5 ++++-
source4/dsdb/samdb/samdb.h | 9 +++++++++
3 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/python/samba/provision/__init__.py b/python/samba/provision/__init__.py
index 8096f8b..df3d03c 100644
--- a/python/samba/provision/__init__.py
+++ b/python/samba/provision/__init__.py
@@ -828,7 +828,7 @@ def setup_samdb_partitions(samdb_path, logger, lp, session_info,
if provision_backend.type != "ldb":
ldap_backend_line = "ldapBackend: %s" % provision_backend.ldap_uri
- required_features = "# No required features"
+ required_features = None
if not plaintext_secrets:
required_features = "requiredFeatures: encryptedSecrets"
@@ -836,6 +836,16 @@ def setup_samdb_partitions(samdb_path, logger, lp, session_info,
backend_store = get_default_backend_store()
backend_store_line = "backendStore: %s" % backend_store
+ if backend_store == "mdb":
+ if required_features is not None:
+ required_features += "\n"
+ else:
+ required_features = ""
+ required_features += "requiredFeatures: lmdbLevelZero"
+
+ if required_features is None:
+ required_features = "# No required features"
+
samdb.transaction_start()
try:
logger.info("Setting up sam.ldb partitions and settings")
diff --git a/source4/dsdb/samdb/ldb_modules/samba_dsdb.c b/source4/dsdb/samdb/ldb_modules/samba_dsdb.c
index e0acb4e..382fc69 100644
--- a/source4/dsdb/samdb/ldb_modules/samba_dsdb.c
+++ b/source4/dsdb/samdb/ldb_modules/samba_dsdb.c
@@ -234,8 +234,11 @@ static bool check_required_features(struct ldb_message_element *el)
int k;
DATA_BLOB esf = data_blob_string_const(
SAMBA_ENCRYPTED_SECRETS_FEATURE);
+ DATA_BLOB lmdbl0 = data_blob_string_const(
+ SAMBA_LMDB_LEVEL_ZERO_FEATURE);
for (k = 0; k < el->num_values; k++) {
- if (data_blob_cmp(&esf, &el->values[k]) != 0) {
+ if ((data_blob_cmp(&esf, &el->values[k]) != 0) &&
+ (data_blob_cmp(&lmdbl0, &el->values[k]) != 0)) {
return false;
}
}
diff --git a/source4/dsdb/samdb/samdb.h b/source4/dsdb/samdb/samdb.h
index 6a4820c..71f1e9c 100644
--- a/source4/dsdb/samdb/samdb.h
+++ b/source4/dsdb/samdb/samdb.h
@@ -333,5 +333,14 @@ struct dsdb_extended_sec_desc_propagation_op {
#define SAMBA_SORTED_LINKS_FEATURE "sortedLinks"
#define SAMBA_ENCRYPTED_SECRETS_FEATURE "encryptedSecrets"
+/*
+ * lmdb level zero feature is an experimental release with basic support
+ * for lmdb database files, instead of tdb.
+ * - Keys are limited to 511 bytes long
+ * - Currently only the:
+ * partition data files
+ * are in lmdb format.
+ */
+#define SAMBA_LMDB_LEVEL_ZERO_FEATURE "lmdbLevelZero"
#endif /* __SAMDB_H__ */
--
1.9.1
From d6b7d99ceeafed02d69b885a55d9f722d40b0b7f Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Fri, 19 Jan 2018 09:28:14 +1300
Subject: [PATCH 29/34] ldb_mod_op_test: Build ldb_mdb_mod_op_test with a new
test
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
lib/ldb/tests/ldb_mod_op_test.c | 65 +++++++++++++++++++++++++++++++++++++++++
lib/ldb/wscript | 6 ++++
2 files changed, 71 insertions(+)
diff --git a/lib/ldb/tests/ldb_mod_op_test.c b/lib/ldb/tests/ldb_mod_op_test.c
index 2ab1c2e..f235eba 100644
--- a/lib/ldb/tests/ldb_mod_op_test.c
+++ b/lib/ldb/tests/ldb_mod_op_test.c
@@ -626,6 +626,46 @@ static void test_transactions(void **state)
assert_int_equal(res->count, 0);
}
+static void test_nested_transactions(void **state)
+{
+ int ret;
+ struct ldbtest_ctx *test_ctx = talloc_get_type_abort(*state,
+ struct ldbtest_ctx);
+ struct ldb_result *res;
+
+ /* start lev-0 transaction */
+ ret = ldb_transaction_start(test_ctx->ldb);
+ assert_int_equal(ret, 0);
+
+ add_keyval(test_ctx, "vegetable", "carrot");
+
+
+ /* start another lev-1 nested transaction */
+ ret = ldb_transaction_start(test_ctx->ldb);
+ assert_int_equal(ret, 0);
+
+ add_keyval(test_ctx, "fruit", "apple");
+
+ /* abort lev-1 nested transaction */
+ ret = ldb_transaction_cancel(test_ctx->ldb);
+ assert_int_equal(ret, 0);
+
+ /* commit lev-0 transaction */
+ ret = ldb_transaction_commit(test_ctx->ldb);
+ assert_int_equal(ret, 0);
+
+ res = get_keyval(test_ctx, "vegetable", "carrot");
+ assert_non_null(res);
+ assert_int_equal(res->count, 1);
+
+ /* This documents the current ldb behaviour, i.e. nested
+ * transactions are not supported. And the cancellation of the nested
+ * transaction has no effect.
+ */
+ res = get_keyval(test_ctx, "fruit", "apple");
+ assert_non_null(res);
+ assert_int_equal(res->count, 1);
+}
struct ldb_mod_test_ctx {
struct ldbtest_ctx *ldb_test_ctx;
const char *entry_dn;
@@ -3328,6 +3368,24 @@ static void test_ldb_add_to_index_unique_values_required(void **state)
assert_int_equal(ret, LDB_ERR_CONSTRAINT_VIOLATION);
talloc_free(tmp_ctx);
}
+
+
+static void test_ldb_talloc_destructor_transaction_cleanup(void **state)
+{
+ struct ldbtest_ctx *test_ctx = NULL;
+
+ test_ctx = talloc_get_type_abort(*state, struct ldbtest_ctx);
+ assert_non_null(test_ctx);
+
+ ldb_transaction_start(test_ctx->ldb);
+
+ /*
+ * Trigger the destructor
+ */
+ TALLOC_FREE(test_ctx->ldb);
+}
+
+
int main(int argc, const char **argv)
{
const struct CMUnitTest tests[] = {
@@ -3361,6 +3419,9 @@ int main(int argc, const char **argv)
cmocka_unit_test_setup_teardown(test_transactions,
ldbtest_setup,
ldbtest_teardown),
+ cmocka_unit_test_setup_teardown(test_nested_transactions,
+ ldbtest_setup,
+ ldbtest_teardown),
cmocka_unit_test_setup_teardown(test_ldb_modify_add_key,
ldb_modify_test_setup,
ldb_modify_test_teardown),
@@ -3467,6 +3528,10 @@ int main(int argc, const char **argv)
test_ldb_add_to_index_unique_values_required,
ldb_non_unique_index_test_setup,
ldb_non_unique_index_test_teardown),
+ cmocka_unit_test_setup_teardown(
+ test_ldb_talloc_destructor_transaction_cleanup,
+ ldbtest_setup,
+ ldbtest_teardown),
};
return cmocka_run_group_tests(tests, NULL, NULL);
diff --git a/lib/ldb/wscript b/lib/ldb/wscript
index e9579b8..ede0265b 100644
--- a/lib/ldb/wscript
+++ b/lib/ldb/wscript
@@ -385,6 +385,12 @@ def build(bld):
deps='cmocka ldb',
install=False)
+ bld.SAMBA_BINARY('ldb_mdb_mod_op_test',
+ source='tests/ldb_mod_op_test.c',
+ cflags='-DTEST_BE=\"mdb\"',
+ deps='cmocka ldb',
+ install=False)
+
bld.SAMBA_BINARY('ldb_msg_test',
source='tests/ldb_msg.c',
deps='cmocka ldb',
--
1.9.1
From e9e309c9afe9ecc018d6dd534216b52f863e9e1b Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Tue, 23 Jan 2018 11:03:16 +1300
Subject: [PATCH 30/34] ldb_mod_op_test: Make sure that closing the database
frees locks
Without the destructor firing, this test used to pass, but now we show
that we must be able to open a new ldb handle.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
lib/ldb/tests/ldb_mod_op_test.c | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/lib/ldb/tests/ldb_mod_op_test.c b/lib/ldb/tests/ldb_mod_op_test.c
index f235eba..c27c20f78 100644
--- a/lib/ldb/tests/ldb_mod_op_test.c
+++ b/lib/ldb/tests/ldb_mod_op_test.c
@@ -3383,6 +3383,35 @@ static void test_ldb_talloc_destructor_transaction_cleanup(void **state)
* Trigger the destructor
*/
TALLOC_FREE(test_ctx->ldb);
+
+ /*
+ * Now ensure that a new connection can be opened
+ */
+ {
+ TALLOC_CTX *tctx = talloc_new(test_ctx);
+ struct ldbtest_ctx *ctx = talloc_zero(tctx, struct ldbtest_ctx);
+ struct ldb_dn *basedn;
+ struct ldb_result *result = NULL;
+ int ret;
+
+ ldbtest_setup((void *)&ctx);
+
+ basedn = ldb_dn_new_fmt(tctx, ctx->ldb, "dc=test");
+ assert_non_null(basedn);
+
+ ret = ldb_search(ctx->ldb,
+ tctx,
+ &result,
+ basedn,
+ LDB_SCOPE_BASE,
+ NULL,
+ NULL);
+ assert_int_equal(ret, 0);
+ assert_non_null(result);
+ assert_int_equal(result->count, 0);
+
+ ldbtest_teardown((void *)&ctx);
+ }
}
--
1.9.1
From f0cea93b04a8e84ea533b1f63c907cc22b19b2a3 Mon Sep 17 00:00:00 2001
From: Gary Lockyer <gary at catalyst.net.nz>
Date: Tue, 23 Jan 2018 13:04:10 +1300
Subject: [PATCH 31/34] tests: Add ldb_mdb_mod_op_test to the tests
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
---
lib/ldb/wscript | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/ldb/wscript b/lib/ldb/wscript
index ede0265b..28bd757 100644
--- a/lib/ldb/wscript
+++ b/lib/ldb/wscript
@@ -424,6 +424,7 @@ def test(ctx):
cmocka_ret = 0
for test_exe in ['ldb_tdb_mod_op_test',
+ 'ldb_mdb_mod_op_test',
'ldb_msg_test']:
cmd = os.path.join(Utils.g_module.blddir, test_exe)
cmocka_ret = cmocka_ret or samba_utils.RUN_COMMAND(cmd)
--
1.9.1
From 78047c47b333275b5aca50ef8f27176460f7ebbc Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Fri, 16 Feb 2018 17:13:26 +1300
Subject: [PATCH 32/34] ldb: Change some prototypes to using ldb_val instead of
TDB_DATA
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_mdb/ldb_mdb.c | 64 ++++++++++++++++++++++-----------------------
lib/ldb/ldb_tdb/ldb_index.c | 8 +++---
lib/ldb/ldb_tdb/ldb_tdb.c | 56 ++++++++++++++++++++++++++++++++-------
lib/ldb/ldb_tdb/ldb_tdb.h | 10 +++----
4 files changed, 88 insertions(+), 50 deletions(-)
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
index 71c2aa2..3c3c1cc 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.c
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -145,8 +145,8 @@ static MDB_txn *get_current_txn(struct lmdb_private *lmdb)
}
static int lmdb_store(struct ltdb_private *ltdb,
- TDB_DATA key,
- TDB_DATA data, int flags)
+ struct ldb_val key,
+ struct ldb_val data, int flags)
{
struct lmdb_private *lmdb = ltdb->lmdb_private;
MDB_val mdb_key;
@@ -167,11 +167,11 @@ static int lmdb_store(struct ltdb_private *ltdb,
return ldb_mdb_error(lmdb->ldb, lmdb->error);
}
- mdb_key.mv_size = key.dsize;
- mdb_key.mv_data = key.dptr;
+ mdb_key.mv_size = key.length;
+ mdb_key.mv_data = key.data;
- mdb_data.mv_size = data.dsize;
- mdb_data.mv_data = data.dptr;
+ mdb_data.mv_size = data.length;
+ mdb_data.mv_data = data.data;
if (flags == TDB_INSERT) {
mdb_flags = MDB_NOOVERWRITE;
@@ -202,7 +202,7 @@ static int lmdb_store(struct ltdb_private *ltdb,
return ldb_mdb_err_map(lmdb->error);
}
-static int lmdb_exists(struct ltdb_private *ltdb, TDB_DATA key)
+static int lmdb_exists(struct ltdb_private *ltdb, struct ldb_val key)
{
struct lmdb_private *lmdb = ltdb->lmdb_private;
MDB_val mdb_key;
@@ -222,8 +222,8 @@ static int lmdb_exists(struct ltdb_private *ltdb, TDB_DATA key)
return ldb_mdb_error(lmdb->ldb, lmdb->error);
}
- mdb_key.mv_size = key.dsize;
- mdb_key.mv_data = key.dptr;
+ mdb_key.mv_size = key.length;
+ mdb_key.mv_data = key.data;
lmdb->error = mdb_get(txn, dbi, &mdb_key, &mdb_data);
if (ltdb->read_lock_count == 0 && lmdb->read_txn != NULL) {
@@ -236,7 +236,7 @@ static int lmdb_exists(struct ltdb_private *ltdb, TDB_DATA key)
return ldb_mdb_err_map(lmdb->error);
}
-static int lmdb_delete(struct ltdb_private *ltdb, TDB_DATA key)
+static int lmdb_delete(struct ltdb_private *ltdb, struct ldb_val key)
{
struct lmdb_private *lmdb = ltdb->lmdb_private;
MDB_val mdb_key;
@@ -255,8 +255,8 @@ static int lmdb_delete(struct ltdb_private *ltdb, TDB_DATA key)
return ldb_mdb_error(lmdb->ldb, lmdb->error);
}
- mdb_key.mv_size = key.dsize;
- mdb_key.mv_data = key.dptr;
+ mdb_key.mv_size = key.length;
+ mdb_key.mv_data = key.data;
lmdb->error = mdb_del(txn, dbi, &mdb_key, NULL);
if (lmdb->error != MDB_SUCCESS) {
@@ -332,22 +332,22 @@ done:
}
static int lmdb_update_in_iterate(struct ltdb_private *ltdb,
- TDB_DATA key,
- TDB_DATA key2,
- TDB_DATA data,
+ struct ldb_val key,
+ struct ldb_val key2,
+ struct ldb_val data,
void *state)
{
struct lmdb_private *lmdb = ltdb->lmdb_private;
- struct TDB_DATA copy;
+ struct ldb_val copy;
int ret = LDB_SUCCESS;
/*
* Need to take a copy of the data as the delete operation alters the
* data, as it is in private lmdb memory.
*/
- copy.dsize = data.dsize;
- copy.dptr = talloc_memdup(ltdb, data.dptr, data.dsize);
- if (copy.dptr == NULL) {
+ copy.length = data.length;
+ copy.data = talloc_memdup(ltdb, data.data, data.length);
+ if (copy.data == NULL) {
lmdb->error = MDB_PANIC;
return ldb_oom(lmdb->ldb);
}
@@ -359,10 +359,10 @@ static int lmdb_update_in_iterate(struct ltdb_private *ltdb,
LDB_DEBUG_ERROR,
"Failed to delete %*.*s "
"for rekey as %*.*s: %s",
- (int)key.dsize, (int)key.dsize,
- (const char *)key.dptr,
- (int)key2.dsize, (int)key2.dsize,
- (const char *)key.dptr,
+ (int)key.length, (int)key.length,
+ (const char *)key.data,
+ (int)key2.length, (int)key2.length,
+ (const char *)key.data,
mdb_strerror(lmdb->error));
ret = ldb_mdb_error(lmdb->ldb, lmdb->error);
goto done;
@@ -373,26 +373,26 @@ static int lmdb_update_in_iterate(struct ltdb_private *ltdb,
lmdb->ldb,
LDB_DEBUG_ERROR,
"Failed to rekey %*.*s as %*.*s: %s",
- (int)key.dsize, (int)key.dsize,
- (const char *)key.dptr,
- (int)key2.dsize, (int)key2.dsize,
- (const char *)key.dptr,
+ (int)key.length, (int)key.length,
+ (const char *)key.data,
+ (int)key2.length, (int)key2.length,
+ (const char *)key.data,
mdb_strerror(lmdb->error));
ret = ldb_mdb_error(lmdb->ldb, lmdb->error);
goto done;
}
done:
- if (copy.dptr != NULL) {
- TALLOC_FREE(copy.dptr);
- copy.dsize = 0;
+ if (copy.data != NULL) {
+ TALLOC_FREE(copy.data);
+ copy.length = 0;
}
/*
* Explicity invalidate the data, as the delete has done this
*/
- data.dsize = 0;
- data.dptr = NULL;
+ data.length = 0;
+ data.data = NULL;
return ret;
}
diff --git a/lib/ldb/ldb_tdb/ldb_index.c b/lib/ldb/ldb_tdb/ldb_index.c
index baf56d9..b7d96d7 100644
--- a/lib/ldb/ldb_tdb/ldb_index.c
+++ b/lib/ldb/ldb_tdb/ldb_index.c
@@ -2426,11 +2426,11 @@ static int re_key(struct ltdb_private *ltdb, struct ldb_val ldb_key, struct ldb_
}
if (key.dsize != key2.dsize ||
(memcmp(key.dptr, key2.dptr, key.dsize) != 0)) {
- TDB_DATA data = {
- .dptr = val.data,
- .dsize = val.length
+ struct ldb_val ldb_key2 = {
+ .data = key2.dptr,
+ .length = key2.dsize
};
- ltdb->kv_ops->update_in_iterate(ltdb, key, key2, data, ctx);
+ ltdb->kv_ops->update_in_iterate(ltdb, ldb_key, ldb_key2, val, ctx);
}
talloc_free(key2.dptr);
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 8993340..7c8eb49 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -1,4 +1,5 @@
/*
+ *
ldb database library
Copyright (C) Andrew Tridgell 2004
@@ -413,8 +414,17 @@ static int ltdb_modified(struct ldb_module *module, struct ldb_dn *dn)
return ret;
}
-static int ltdb_tdb_store(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags)
+static int ltdb_tdb_store(struct ltdb_private *ltdb, struct ldb_val ldb_key,
+ struct ldb_val ldb_data, int flags)
{
+ TDB_DATA key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
+ };
+ TDB_DATA data = {
+ .dptr = ldb_data.data,
+ .dsize = ldb_data.length
+ };
return tdb_store(ltdb->tdb, key, data, flags);
}
@@ -435,7 +445,8 @@ int ltdb_store(struct ldb_module *module, const struct ldb_message *msg, int flg
{
void *data = ldb_module_get_private(module);
struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
- TDB_DATA tdb_key, tdb_data;
+ TDB_DATA tdb_key;
+ struct ldb_val ldb_key;
struct ldb_val ldb_data;
int ret = LDB_SUCCESS;
TALLOC_CTX *tdb_key_ctx = talloc_new(module);
@@ -461,10 +472,10 @@ int ltdb_store(struct ldb_module *module, const struct ldb_message *msg, int flg
return LDB_ERR_OTHER;
}
- tdb_data.dptr = ldb_data.data;
- tdb_data.dsize = ldb_data.length;
+ ldb_key.data = tdb_key.dptr;
+ ldb_key.length = tdb_key.dsize;
- ret = ltdb->kv_ops->store(ltdb, tdb_key, tdb_data, flgs);
+ ret = ltdb->kv_ops->store(ltdb, ldb_key, ldb_data, flgs);
if (ret != 0) {
bool is_special = ldb_dn_is_special(msg->dn);
ret = ltdb->kv_ops->error(ltdb);
@@ -650,13 +661,21 @@ static int ltdb_add(struct ltdb_context *ctx)
return ret;
}
-static int ltdb_tdb_exists(struct ltdb_private *ltdb, TDB_DATA key)
+static int ltdb_tdb_exists(struct ltdb_private *ltdb, struct ldb_val ldb_key)
{
+ TDB_DATA key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
+ };
return tdb_exists(ltdb->tdb, key);
}
-static int ltdb_tdb_delete(struct ltdb_private *ltdb, TDB_DATA tdb_key)
+static int ltdb_tdb_delete(struct ltdb_private *ltdb, struct ldb_val ldb_key)
{
+ TDB_DATA tdb_key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
+ };
return tdb_delete(ltdb->tdb, tdb_key);
}
@@ -669,6 +688,7 @@ int ltdb_delete_noindex(struct ldb_module *module,
{
void *data = ldb_module_get_private(module);
struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
+ struct ldb_val ldb_key;
TDB_DATA tdb_key;
int ret;
TALLOC_CTX *tdb_key_ctx = talloc_new(module);
@@ -687,7 +707,10 @@ int ltdb_delete_noindex(struct ldb_module *module,
return LDB_ERR_OTHER;
}
- ret = ltdb->kv_ops->delete(ltdb, tdb_key);
+ ldb_key.data = tdb_key.dptr;
+ ldb_key.length = tdb_key.dsize;
+
+ ret = ltdb->kv_ops->delete(ltdb, ldb_key);
TALLOC_FREE(tdb_key_ctx);
if (ret != 0) {
@@ -1752,12 +1775,27 @@ static int ltdb_tdb_traverse_fn(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn
}
}
-static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA key2, TDB_DATA data, void *state)
+static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb,
+ struct ldb_val ldb_key,
+ struct ldb_val ldb_key2,
+ struct ldb_val ldb_data, void *state)
{
int tdb_ret;
struct ldb_context *ldb;
struct ltdb_reindex_context *ctx = (struct ltdb_reindex_context *)state;
struct ldb_module *module = ctx->module;
+ TDB_DATA key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
+ };
+ TDB_DATA key2 = {
+ .dptr = ldb_key2.data,
+ .dsize = ldb_key2.length
+ };
+ TDB_DATA data = {
+ .dptr = ldb_data.data,
+ .dsize = ldb_data.length
+ };
ldb = ldb_module_get_ctx(module);
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 49e0bfc..8c51e6f 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -10,12 +10,12 @@ typedef int (*ldb_kv_traverse_fn)(struct ltdb_private *ltdb,
void *ctx);
struct kv_db_ops {
- int (*store)(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags);
- int (*delete)(struct ltdb_private *ltdb, TDB_DATA key);
- int (*exists)(struct ltdb_private *ltdb, TDB_DATA key);
+ int (*store)(struct ltdb_private *ltdb, struct ldb_val key, struct ldb_val data, int flags);
+ int (*delete)(struct ltdb_private *ltdb, struct ldb_val key);
+ int (*exists)(struct ltdb_private *ltdb, struct ldb_val key);
int (*iterate)(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn, void *ctx);
- int (*update_in_iterate)(struct ltdb_private *ltdb, TDB_DATA key,
- TDB_DATA key2, TDB_DATA data, void *ctx);
+ int (*update_in_iterate)(struct ltdb_private *ltdb, struct ldb_val key,
+ struct ldb_val key2, struct ldb_val data, void *ctx);
int (*fetch_and_parse)(struct ltdb_private *ltdb, TDB_DATA key,
int (*parser)(TDB_DATA key, TDB_DATA data,
void *private_data),
--
1.9.1
From c3ea7f599322b8615708541f5a6ea1b59d214959 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Mon, 19 Feb 2018 12:37:20 +1300
Subject: [PATCH 33/34] ldb: Change remaining fetch prototypes to remove
TDB_DATA
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_mdb/ldb_mdb.c | 16 ++++++++--------
lib/ldb/ldb_tdb/ldb_search.c | 28 +++++++++++++++-------------
lib/ldb/ldb_tdb/ldb_tdb.c | 38 +++++++++++++++++++++++++++++++++++---
lib/ldb/ldb_tdb/ldb_tdb.h | 4 ++--
4 files changed, 60 insertions(+), 26 deletions(-)
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
index 3c3c1cc..fdb59c0 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.c
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -397,9 +397,9 @@ done:
return ret;
}
/* Handles only a single record */
-static int lmdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
- int (*parser)(TDB_DATA key, TDB_DATA data,
- void *private_data),
+static int lmdb_parse_record(struct ltdb_private *ltdb, struct ldb_val key,
+ int (*parser)(struct ldb_val key, struct ldb_val data,
+ void *private_data),
void *ctx)
{
struct lmdb_private *lmdb = ltdb->lmdb_private;
@@ -407,7 +407,7 @@ static int lmdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
MDB_val mdb_data;
MDB_txn *txn = NULL;
MDB_dbi dbi;
- TDB_DATA data;
+ struct ldb_val data;
txn = get_current_txn(lmdb);
if (txn == NULL) {
@@ -421,8 +421,8 @@ static int lmdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
return ldb_mdb_error(lmdb->ldb, lmdb->error);
}
- mdb_key.mv_size = key.dsize;
- mdb_key.mv_data = key.dptr;
+ mdb_key.mv_size = key.length;
+ mdb_key.mv_data = key.data;
lmdb->error = mdb_get(txn, dbi, &mdb_key, &mdb_data);
if (lmdb->error != MDB_SUCCESS) {
@@ -437,8 +437,8 @@ static int lmdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
}
return ldb_mdb_error(lmdb->ldb, lmdb->error);
}
- data.dptr = mdb_data.mv_data;
- data.dsize = mdb_data.mv_size;
+ data.data = mdb_data.mv_data;
+ data.length = mdb_data.mv_size;
/* TODO closing a handle should not even be necessary */
mdb_dbi_close(lmdb->env, dbi);
diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c
index 0d205b3..a7f09bf 100644
--- a/lib/ldb/ldb_tdb/ldb_search.c
+++ b/lib/ldb/ldb_tdb/ldb_search.c
@@ -180,17 +180,15 @@ struct ltdb_parse_data_unpack_ctx {
unsigned int unpack_flags;
};
-static int ltdb_parse_data_unpack(TDB_DATA key, TDB_DATA data,
+static int ltdb_parse_data_unpack(struct ldb_val key,
+ struct ldb_val data,
void *private_data)
{
struct ltdb_parse_data_unpack_ctx *ctx = private_data;
unsigned int nb_elements_in_db;
int ret;
struct ldb_context *ldb = ldb_module_get_ctx(ctx->module);
- struct ldb_val data_parse = {
- .data = data.dptr,
- .length = data.dsize
- };
+ struct ldb_val data_parse = data;
if (ctx->unpack_flags & LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC) {
/*
@@ -200,13 +198,13 @@ static int ltdb_parse_data_unpack(TDB_DATA key, TDB_DATA data,
* and the caller needs a stable result.
*/
data_parse.data = talloc_memdup(ctx->msg,
- data.dptr,
- data.dsize);
+ data.data,
+ data.length);
if (data_parse.data == NULL) {
ldb_debug(ldb, LDB_DEBUG_ERROR,
"Unable to allocate data(%d) for %*.*s\n",
- (int)data.dsize,
- (int)key.dsize, (int)key.dsize, key.dptr);
+ (int)data.length,
+ (int)key.length, (int)key.length, key.data);
return LDB_ERR_OPERATIONS_ERROR;
}
}
@@ -217,13 +215,13 @@ static int ltdb_parse_data_unpack(TDB_DATA key, TDB_DATA data,
ctx->unpack_flags,
&nb_elements_in_db);
if (ret == -1) {
- if (data_parse.data != data.dptr) {
+ if (data_parse.data != data.data) {
talloc_free(data_parse.data);
}
ldb_debug(ldb, LDB_DEBUG_ERROR, "Invalid data for index %*.*s\n",
- (int)key.dsize, (int)key.dsize, key.dptr);
- return LDB_ERR_OPERATIONS_ERROR;
+ (int)key.length, (int)key.length, key.data);
+ return LDB_ERR_OPERATIONS_ERROR;
}
return ret;
}
@@ -246,13 +244,17 @@ int ltdb_search_key(struct ldb_module *module, struct ltdb_private *ltdb,
.module = module,
.unpack_flags = unpack_flags
};
+ struct ldb_val ldb_key = {
+ .data = tdb_key.dptr,
+ .length = tdb_key.dsize
+ };
memset(msg, 0, sizeof(*msg));
msg->num_elements = 0;
msg->elements = NULL;
- ret = ltdb->kv_ops->fetch_and_parse(ltdb, tdb_key,
+ ret = ltdb->kv_ops->fetch_and_parse(ltdb, ldb_key,
ltdb_parse_data_unpack, &ctx);
if (ret == -1) {
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 7c8eb49..aed07e9 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -1745,6 +1745,9 @@ struct kv_ctx {
ldb_kv_traverse_fn kv_traverse_fn;
void *ctx;
struct ltdb_private *ltdb;
+ int (*parser)(struct ldb_val key,
+ struct ldb_val data,
+ void *private_data);
};
static int ldb_tdb_traverse_fn_wrapper(struct tdb_context *tdb, TDB_DATA tdb_key, TDB_DATA tdb_data, void *ctx)
@@ -1827,12 +1830,41 @@ static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb,
return tdb_ret;
}
-static int ltdb_tdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
- int (*parser)(TDB_DATA key, TDB_DATA data,
+static int ltdb_tdb_parse_record_wrapper(TDB_DATA tdb_key, TDB_DATA tdb_data,
+ void *ctx)
+{
+ struct kv_ctx *kv_ctx = ctx;
+ struct ldb_val key = {
+ .length = tdb_key.dsize,
+ .data = tdb_key.dptr,
+ };
+ struct ldb_val data = {
+ .length = tdb_data.dsize,
+ .data = tdb_data.dptr,
+ };
+
+ return kv_ctx->parser(key, data, kv_ctx->ctx);
+}
+
+static int ltdb_tdb_parse_record(struct ltdb_private *ltdb,
+ struct ldb_val ldb_key,
+ int (*parser)(struct ldb_val key,
+ struct ldb_val data,
void *private_data),
void *ctx)
{
- return tdb_parse_record(ltdb->tdb, key, parser, ctx);
+ struct kv_ctx kv_ctx = {
+ .parser = parser,
+ .ctx = ctx,
+ .ltdb = ltdb
+ };
+ TDB_DATA key = {
+ .dptr = ldb_key.data,
+ .dsize = ldb_key.length
+ };
+
+ return tdb_parse_record(ltdb->tdb, key, ltdb_tdb_parse_record_wrapper,
+ &kv_ctx);
}
static const char * ltdb_tdb_name(struct ltdb_private *ltdb)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 8c51e6f..b67c04c 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -16,8 +16,8 @@ struct kv_db_ops {
int (*iterate)(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn, void *ctx);
int (*update_in_iterate)(struct ltdb_private *ltdb, struct ldb_val key,
struct ldb_val key2, struct ldb_val data, void *ctx);
- int (*fetch_and_parse)(struct ltdb_private *ltdb, TDB_DATA key,
- int (*parser)(TDB_DATA key, TDB_DATA data,
+ int (*fetch_and_parse)(struct ltdb_private *ltdb, struct ldb_val key,
+ int (*parser)(struct ldb_val key, struct ldb_val data,
void *private_data),
void *ctx);
int (*lock_read)(struct ldb_module *);
--
1.9.1
From ea0e10a87266175154d66aeff824c3cde4914117 Mon Sep 17 00:00:00 2001
From: Garming Sam <garming at catalyst.net.nz>
Date: Mon, 19 Feb 2018 14:07:10 +1300
Subject: [PATCH 34/34] ldb_mdb: Set require private event contexts
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
---
lib/ldb/ldb_mdb/ldb_mdb.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
index fdb59c0..48a4ad0 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.c
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -719,6 +719,12 @@ int lmdb_connect(struct ldb_context *ldb,
struct ltdb_private *ltdb = NULL;
int ret;
+ /*
+ * We hold locks, so we must use a private event context
+ * on each returned handle
+ */
+ ldb_set_require_private_event_context(ldb);
+
path = lmdb_get_path(url);
if (path == NULL) {
ldb_debug(ldb, LDB_DEBUG_ERROR, "Invalid mdb URL '%s'", url);
--
1.9.1
More information about the samba-technical
mailing list