[SCM] Samba Shared Repository - branch master updated

Andrew Bartlett abartlet at samba.org
Fri Apr 6 03:13:02 UTC 2018


The branch, master has been updated
       via  946dfc7 upgradeprovision: detect and handle lmdb databases
       via  0935b25 ldb: Unwind transaction counter if start_transaction fails
       via  428c0a8 source3: initilize_password_db after a fork.
       via  a985683 dsdb: add lmdbLevelOne as a required feature.
       via  ba61c68 provision: Set @INDEXLIST first when building dummy sam.ldb
       via  34b12fb provision: allow provisioning of a different database backend
       via  301cd5e python: Add wrapper of mdb_copy that we can call from python
       via  5ca90e7 ldb tests: add cmocka tests of kv operations
       via  1fceb64 ldb_tdb: ltdb_tdb_delete require active transaction
       via  e001c5f ldb_tdb: ltdb_tdb_store require active transaction
       via  8d1b11a ldb_tdb: ltdb_tdb_parse_record map tdb error codes
       via  d206fcf ldb tests: ldb_mod_op_test use correct ldb to create dn
       via  0f7d153 ldb test: close pipes to stop forked tests failing on failure
       via  412cdb1 ldb index: Add tests for truncated base 64 index keys
       via  556466e ldb_tdb: A more robust check for if we can fit the index string in
       via  d161a6d ldb index: Fix truncation key length calculation
       via  eb4205f ldb: Allow GUID index mode to be tested on TDB
       via  68d7097 ldb: Ignore these tests in mdb test mode
       via  8c84854 upgradeprovision: Do not copy backup lmdb -lock files
       via  7bf8539 ldb: Change remaining fetch prototypes to remove TDB_DATA
       via  7885181 ldb: Change some prototypes to using ldb_val instead of TDB_DATA
       via  5dc7db5 samba-tool domain classicupgrade: Do not mix python-samdb transactions and passdb modifications
       via  f8b368c ldb: Fix missing NULL terminator in ldb_mod_op_test testsuite
      from  924f3f0 wafsamba: Add '-Werror=strict-overflow -Wstrict-overflow=2' to the developer build

https://git.samba.org/?p=samba.git;a=shortlog;h=master


- Log -----------------------------------------------------------------
commit 946dfc783cde9015cc7f288b69008ff22436f255
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Thu Mar 22 12:50:45 2018 +1300

    upgradeprovision: detect and handle lmdb databases
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>
    
    Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
    Autobuild-Date(master): Fri Apr  6 05:12:11 CEST 2018 on sn-devel-144

commit 0935b25bd29fe03dbfbede0b3d65fefef74c1784
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Fri Mar 23 11:23:39 2018 +1300

    ldb: Unwind transaction counter if start_transaction fails
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>

commit 428c0a81aecd578e77123ca27b835b2fc8a54e26
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 20 13:14:38 2018 +1300

    source3: initilize_password_db after a fork.
    
    This is required because we need a new pointer for LDB after the fork,
    and with LMDB we can not longer rely on tdb_reopen_all() to do that
    for us.
    
    This can not be done in reinit_after_fork() due to the dependency loop
    this would create.
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit a98568397d37a2dc9d20f20a5bc1f5b81c5f4b4d
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Wed Feb 7 11:10:34 2018 +1300

    dsdb: add lmdbLevelOne as a required feature.
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit ba61c684686933b1a69a1bb357428db9911b2048
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Fri Mar 23 10:58:11 2018 +1300

    provision: Set @INDEXLIST first when building dummy sam.ldb
    
    The new LMDB backed will not allow normal records to be added before the @INDEXLIST
    as this is what forces the GUID index mode.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit 34b12fbd8805d7f419ae5f41505e6ea430ad5168
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 20 14:38:19 2018 +1300

    provision: allow provisioning of a different database backend
    
    This sets the backendStore field in @PARTITION, depending on which
    argument you set in the provision.
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 301cd5e58a58e4e373c81425fc6394d70becbdf7
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Wed Apr 4 16:17:51 2018 +1200

    python: Add wrapper of mdb_copy that we can call from python
    
    This is like the use of tdbbackup for tdb files.
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 5ca90e758ade97fb5e335029c7a1768094e70564
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 20 12:15:12 2018 +1300

    ldb tests: add cmocka tests of kv operations
    
    Add tests for the behaviour the ldb layer expects the key value layer to
    provide.  This should make it easier to add another KV store
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 1fceb64dc2c7f994e800b00cc9fbd214ac42a201
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Thu Mar 15 11:37:06 2018 +1300

    ldb_tdb: ltdb_tdb_delete require active transaction
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit e001c5fb5f2de686cb86c9aae8741ae5776bde64
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Thu Mar 15 11:36:33 2018 +1300

    ldb_tdb: ltdb_tdb_store require active transaction
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 8d1b11aac9d43080c5efa1d84219af283f8c7d49
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Thu Mar 15 11:33:32 2018 +1300

    ldb_tdb: ltdb_tdb_parse_record map tdb error codes
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit d206fcf5c7b7f598993e385896c9d9e8db64c31a
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 20 11:20:35 2018 +1300

    ldb tests: ldb_mod_op_test use correct ldb to create dn
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 0f7d1534954b87a39c68d9ceb5d4b03c3cff1839
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 13 12:43:25 2018 +1300

    ldb test: close pipes to stop forked tests failing on failure
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 412cdb1714570bb688ba01625da829d48a0c07ad
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 13 08:45:28 2018 +1300

    ldb index: Add tests for truncated base 64 index keys
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 556466e7e3bed0edd8cd15e58dd4380f13c14364
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Wed Apr 4 14:00:57 2018 +1200

    ldb_tdb: A more robust check for if we can fit the index string in
    
    This avoids magic numbers and also is careful against overflow
    from a long attr_for_dn.
    
    This is done as a distinct commit to make the previous behaviour
    change more clear, and to show that this does not change the
    calculations, only improves the overflow check.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit d161a6dcfef09ad939df76c5c77eb8b7519a9326
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Tue Mar 13 08:10:54 2018 +1300

    ldb index: Fix truncation key length calculation
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit eb4205fc31830f4227667e245395d81d3b3654cb
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Fri Mar 23 11:10:25 2018 +1300

    ldb: Allow GUID index mode to be tested on TDB
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit 68d709741a868a3c02074934c4cc7355d6f9d19e
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Tue Mar 6 17:00:07 2018 +1300

    ldb: Ignore these tests in mdb test mode
    
    These are tests are specifically for when the GUID index is not in use
    which is always in with ldb_mdb.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit 8c848549eeff177d754624006b9fa629cf908369
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Wed Jan 17 14:02:09 2018 +1300

    upgradeprovision: Do not copy backup lmdb -lock files
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>

commit 7bf853950d551e519fe25a344eb6a742fc5dd299
Author: Garming Sam <garming at catalyst.net.nz>
Date:   Mon Feb 19 12:37:20 2018 +1300

    ldb: Change remaining fetch prototypes to remove TDB_DATA
    
    Signed-off-by: Garming Sam <garming at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit 78851816501aa7ea266d6447605910e3bc8f3b20
Author: Garming Sam <garming at catalyst.net.nz>
Date:   Fri Feb 16 17:13:26 2018 +1300

    ldb: Change some prototypes to using ldb_val instead of TDB_DATA
    
    Signed-off-by: Garming Sam <garming at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit 5dc7db578647e5381976af4d0d447d3685b779f9
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Fri Mar 23 13:05:55 2018 +1300

    samba-tool domain classicupgrade: Do not mix python-samdb transactions and passdb modifications
    
    This worked previously because we knew the same tdb was in use under the hood,
    but now that nested TDB transactions are banned this breaks, and it breaks for
    LMDB.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

commit f8b368c9f0c2a34b6d15303a9d6facd762e1a517
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Thu Mar 8 14:01:50 2018 +1300

    ldb: Fix missing NULL terminator in ldb_mod_op_test testsuite
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>

-----------------------------------------------------------------------

Summary of changes:
 lib/ldb/common/ldb.c                         |    1 +
 lib/ldb/ldb_tdb/ldb_index.c                  |   69 +-
 lib/ldb/ldb_tdb/ldb_search.c                 |   28 +-
 lib/ldb/ldb_tdb/ldb_tdb.c                    |  100 +-
 lib/ldb/ldb_tdb/ldb_tdb.h                    |   12 +-
 lib/ldb/tests/ldb_kv_ops_test.c              | 1396 ++++++++++++++++++++++++++
 lib/ldb/tests/ldb_mod_op_test.c              |  160 ++-
 lib/ldb/tests/python/index.py                |   44 +-
 lib/ldb/wscript                              |   19 +-
 python/samba/{tdb_util.py => mdb_util.py}    |   25 +-
 python/samba/netcmd/domain.py                |   14 +-
 python/samba/provision/__init__.py           |   50 +-
 python/samba/provision/sambadns.py           |   43 +-
 python/samba/samdb.py                        |    3 +
 python/samba/upgrade.py                      |   48 +-
 source3/rpc_server/lsasd.c                   |    3 +
 source3/smbd/process.c                       |    1 +
 source3/smbd/server_exit.c                   |    6 +-
 source3/winbindd/winbindd.c                  |    2 +
 source3/winbindd/winbindd_dual.c             |    2 +
 source4/dsdb/samdb/ldb_modules/samba_dsdb.c  |    5 +-
 source4/dsdb/samdb/samdb.h                   |    9 +
 source4/scripting/bin/samba_upgradeprovision |   29 +-
 source4/setup/provision_partitions.ldif      |    1 +
 24 files changed, 1928 insertions(+), 142 deletions(-)
 create mode 100644 lib/ldb/tests/ldb_kv_ops_test.c
 copy python/samba/{tdb_util.py => mdb_util.py} (55%)


Changeset truncated at 500 lines:

diff --git a/lib/ldb/common/ldb.c b/lib/ldb/common/ldb.c
index a4d9977..2249089 100644
--- a/lib/ldb/common/ldb.c
+++ b/lib/ldb/common/ldb.c
@@ -379,6 +379,7 @@ int ldb_transaction_start(struct ldb_context *ldb)
 				"ldb transaction start: %s (%d)",
 				ldb_strerror(status),
 				status);
+		ldb->transaction_active--;
 		}
 		if ((next_module && next_module->ldb->flags & LDB_FLG_ENABLE_TRACING)) {
 			ldb_debug(next_module->ldb, LDB_DEBUG_TRACE, "start ldb transaction error: %s",
diff --git a/lib/ldb/ldb_tdb/ldb_index.c b/lib/ldb/ldb_tdb/ldb_index.c
index bb534c3..59434f3 100644
--- a/lib/ldb/ldb_tdb/ldb_index.c
+++ b/lib/ldb/ldb_tdb/ldb_index.c
@@ -842,12 +842,18 @@ static struct ldb_dn *ltdb_index_key(struct ldb_context *ldb,
 	const char *attr_for_dn = NULL;
 	int r;
 	bool should_b64_encode;
-	unsigned max_key_length = ltdb_max_key_length(ltdb);
-	unsigned key_len = 0;
-	unsigned attr_len = 0;
-	unsigned indx_len = 0;
-	unsigned frmt_len = 0;
 
+	unsigned int max_key_length = ltdb_max_key_length(ltdb);
+	size_t key_len = 0;
+	size_t attr_len = 0;
+	const size_t indx_len = sizeof(LTDB_INDEX) - 1;
+	unsigned frmt_len = 0;
+	const size_t additional_key_length = 4;
+	unsigned int num_separators = 3; /* Estimate for overflow check */
+	const size_t min_data = 1;
+	const size_t min_key_length = additional_key_length
+		+ indx_len + num_separators + min_data;
+	
 	if (attr[0] == '@') {
 		attr_for_dn = attr;
 		v = *value;
@@ -884,7 +890,29 @@ static struct ldb_dn *ltdb_index_key(struct ldb_context *ldb,
 		}
 	}
 	attr_len = strlen(attr_for_dn);
-	indx_len = strlen(LTDB_INDEX);
+
+	/*
+	 * Check if there is any hope this will fit into the DB.
+	 * Overflow here is not actually critical the code below
+	 * checks again to make the printf and the DB does another
+	 * check for too long keys
+	 */
+	if (max_key_length - attr_len < min_key_length) {
+		ldb_asprintf_errstring(
+			ldb,
+			__location__ ": max_key_length "
+			"is too small (%u) < (%u)",
+			max_key_length,
+			(unsigned)(min_key_length + attr_len));
+		talloc_free(attr_folded);
+		return NULL;
+	}
+	
+	/*
+	 * ltdb_key_dn() makes something 4 bytes longer, it adds a leading
+	 * "DN=" and a trailing string terminator
+	 */
+	max_key_length -= additional_key_length;
 
 	/*
 	 * We do not base 64 encode a DN in a key, it has already been
@@ -909,16 +937,20 @@ static struct ldb_dn *ltdb_index_key(struct ldb_context *ldb,
 	}
 
 	if (should_b64_encode) {
-		unsigned vstr_len = 0;
+		size_t vstr_len = 0;
 		char *vstr = ldb_base64_encode(ldb, (char *)v.data, v.length);
 		if (!vstr) {
 			talloc_free(attr_folded);
 			return NULL;
 		}
 		vstr_len = strlen(vstr);
-		key_len = 3 + indx_len + attr_len + vstr_len;
+		/* 
+		 * Overflow here is not critical as we only use this
+		 * to choose the printf truncation
+		 */
+		key_len = num_separators + indx_len + attr_len + vstr_len;
 		if (key_len > max_key_length) {
-			unsigned excess = key_len - max_key_length;
+			size_t excess = key_len - max_key_length;
 			frmt_len = vstr_len - excess;
 			*truncation = KEY_TRUNCATED;
 			/*
@@ -943,9 +975,16 @@ static struct ldb_dn *ltdb_index_key(struct ldb_context *ldb,
 		}
 		talloc_free(vstr);
 	} else {
-		key_len = 2 + indx_len + attr_len + (int)v.length;
+		/* Only need two seperators */
+		num_separators = 2;
+		
+		/* 
+		 * Overflow here is not critical as we only use this
+		 * to choose the printf truncation
+		 */
+		key_len = num_separators + indx_len + attr_len + (int)v.length;
 		if (key_len > max_key_length) {
-			unsigned excess = key_len - max_key_length;
+			size_t excess = key_len - max_key_length;
 			frmt_len = v.length - excess;
 			*truncation = KEY_TRUNCATED;
 			/*
@@ -2783,11 +2822,11 @@ static int re_key(struct ltdb_private *ltdb, struct ldb_val ldb_key, struct ldb_
 	}
 	if (key.dsize != key2.dsize ||
 	    (memcmp(key.dptr, key2.dptr, key.dsize) != 0)) {
-		TDB_DATA data = {
-			.dptr = val.data,
-			.dsize = val.length
+		struct ldb_val ldb_key2 = {
+			.data = key2.dptr,
+			.length = key2.dsize
 		};
-		ltdb->kv_ops->update_in_iterate(ltdb, key, key2, data, ctx);
+		ltdb->kv_ops->update_in_iterate(ltdb, ldb_key, ldb_key2, val, ctx);
 	}
 	talloc_free(key2.dptr);
 
diff --git a/lib/ldb/ldb_tdb/ldb_search.c b/lib/ldb/ldb_tdb/ldb_search.c
index 78ef8b0..35583a1 100644
--- a/lib/ldb/ldb_tdb/ldb_search.c
+++ b/lib/ldb/ldb_tdb/ldb_search.c
@@ -180,17 +180,15 @@ struct ltdb_parse_data_unpack_ctx {
 	unsigned int unpack_flags;
 };
 
-static int ltdb_parse_data_unpack(TDB_DATA key, TDB_DATA data,
+static int ltdb_parse_data_unpack(struct ldb_val key,
+				  struct ldb_val data,
 				  void *private_data)
 {
 	struct ltdb_parse_data_unpack_ctx *ctx = private_data;
 	unsigned int nb_elements_in_db;
 	int ret;
 	struct ldb_context *ldb = ldb_module_get_ctx(ctx->module);
-	struct ldb_val data_parse = {
-		.data = data.dptr,
-		.length = data.dsize
-	};
+	struct ldb_val data_parse = data;
 
 	if (ctx->unpack_flags & LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC) {
 		/*
@@ -200,13 +198,13 @@ static int ltdb_parse_data_unpack(TDB_DATA key, TDB_DATA data,
 		 * and the caller needs a stable result.
 		 */
 		data_parse.data = talloc_memdup(ctx->msg,
-						data.dptr,
-						data.dsize);
+						data.data,
+						data.length);
 		if (data_parse.data == NULL) {
 			ldb_debug(ldb, LDB_DEBUG_ERROR,
 				  "Unable to allocate data(%d) for %*.*s\n",
-				  (int)data.dsize,
-				  (int)key.dsize, (int)key.dsize, key.dptr);
+				  (int)data.length,
+				  (int)key.length, (int)key.length, key.data);
 			return LDB_ERR_OPERATIONS_ERROR;
 		}
 	}
@@ -217,13 +215,13 @@ static int ltdb_parse_data_unpack(TDB_DATA key, TDB_DATA data,
 						   ctx->unpack_flags,
 						   &nb_elements_in_db);
 	if (ret == -1) {
-		if (data_parse.data != data.dptr) {
+		if (data_parse.data != data.data) {
 			talloc_free(data_parse.data);
 		}
 
 		ldb_debug(ldb, LDB_DEBUG_ERROR, "Invalid data for index %*.*s\n",
-			  (int)key.dsize, (int)key.dsize, key.dptr);
-		return LDB_ERR_OPERATIONS_ERROR;		
+			  (int)key.length, (int)key.length, key.data);
+		return LDB_ERR_OPERATIONS_ERROR;
 	}
 	return ret;
 }
@@ -246,13 +244,17 @@ int ltdb_search_key(struct ldb_module *module, struct ltdb_private *ltdb,
 		.module = module,
 		.unpack_flags = unpack_flags
 	};
+	struct ldb_val ldb_key = {
+		.data = tdb_key.dptr,
+		.length = tdb_key.dsize
+	};
 
 	memset(msg, 0, sizeof(*msg));
 
 	msg->num_elements = 0;
 	msg->elements = NULL;
 
-	ret = ltdb->kv_ops->fetch_and_parse(ltdb, tdb_key,
+	ret = ltdb->kv_ops->fetch_and_parse(ltdb, ldb_key,
 					    ltdb_parse_data_unpack, &ctx);
 
 	if (ret == -1) {
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 20796f2..1d06566 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -417,8 +417,21 @@ static int ltdb_modified(struct ldb_module *module, struct ldb_dn *dn)
 	return ret;
 }
 
-static int ltdb_tdb_store(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags)
+static int ltdb_tdb_store(struct ltdb_private *ltdb, struct ldb_val ldb_key,
+			  struct ldb_val ldb_data, int flags)
 {
+	TDB_DATA key = {
+		.dptr = ldb_key.data,
+		.dsize = ldb_key.length
+	};
+	TDB_DATA data = {
+		.dptr = ldb_data.data,
+		.dsize = ldb_data.length
+	};
+	bool transaction_active = tdb_transaction_active(ltdb->tdb);
+	if (transaction_active == false){
+		return LDB_ERR_PROTOCOL_ERROR;
+	}
 	return tdb_store(ltdb->tdb, key, data, flags);
 }
 
@@ -439,7 +452,8 @@ int ltdb_store(struct ldb_module *module, const struct ldb_message *msg, int flg
 {
 	void *data = ldb_module_get_private(module);
 	struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
-	TDB_DATA tdb_key, tdb_data;
+	TDB_DATA tdb_key;
+	struct ldb_val ldb_key;
 	struct ldb_val ldb_data;
 	int ret = LDB_SUCCESS;
 	TALLOC_CTX *tdb_key_ctx = talloc_new(module);
@@ -465,10 +479,10 @@ int ltdb_store(struct ldb_module *module, const struct ldb_message *msg, int flg
 		return LDB_ERR_OTHER;
 	}
 
-	tdb_data.dptr = ldb_data.data;
-	tdb_data.dsize = ldb_data.length;
+	ldb_key.data = tdb_key.dptr;
+	ldb_key.length = tdb_key.dsize;
 
-	ret = ltdb->kv_ops->store(ltdb, tdb_key, tdb_data, flgs);
+	ret = ltdb->kv_ops->store(ltdb, ldb_key, ldb_data, flgs);
 	if (ret != 0) {
 		bool is_special = ldb_dn_is_special(msg->dn);
 		ret = ltdb->kv_ops->error(ltdb);
@@ -654,8 +668,16 @@ static int ltdb_add(struct ltdb_context *ctx)
 	return ret;
 }
 
-static int ltdb_tdb_delete(struct ltdb_private *ltdb, TDB_DATA tdb_key)
+static int ltdb_tdb_delete(struct ltdb_private *ltdb, struct ldb_val ldb_key)
 {
+	TDB_DATA tdb_key = {
+		.dptr = ldb_key.data,
+		.dsize = ldb_key.length
+	};
+	bool transaction_active = tdb_transaction_active(ltdb->tdb);
+	if (transaction_active == false){
+		return LDB_ERR_PROTOCOL_ERROR;
+	}
 	return tdb_delete(ltdb->tdb, tdb_key);
 }
 
@@ -668,6 +690,7 @@ int ltdb_delete_noindex(struct ldb_module *module,
 {
 	void *data = ldb_module_get_private(module);
 	struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
+	struct ldb_val ldb_key;
 	TDB_DATA tdb_key;
 	int ret;
 	TALLOC_CTX *tdb_key_ctx = talloc_new(module);
@@ -686,7 +709,10 @@ int ltdb_delete_noindex(struct ldb_module *module,
 		return LDB_ERR_OTHER;
 	}
 
-	ret = ltdb->kv_ops->delete(ltdb, tdb_key);
+	ldb_key.data = tdb_key.dptr;
+	ldb_key.length = tdb_key.dsize;
+
+	ret = ltdb->kv_ops->delete(ltdb, ldb_key);
 	TALLOC_FREE(tdb_key_ctx);
 
 	if (ret != 0) {
@@ -1747,6 +1773,9 @@ struct kv_ctx {
 	ldb_kv_traverse_fn kv_traverse_fn;
 	void *ctx;
 	struct ltdb_private *ltdb;
+	int (*parser)(struct ldb_val key,
+		      struct ldb_val data,
+		      void *private_data);
 };
 
 static int ldb_tdb_traverse_fn_wrapper(struct tdb_context *tdb, TDB_DATA tdb_key, TDB_DATA tdb_data, void *ctx)
@@ -1777,12 +1806,27 @@ static int ltdb_tdb_traverse_fn(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn
 	}
 }
 
-static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA key2, TDB_DATA data, void *state)
+static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb,
+				      struct ldb_val ldb_key,
+				      struct ldb_val ldb_key2,
+				      struct ldb_val ldb_data, void *state)
 {
 	int tdb_ret;
 	struct ldb_context *ldb;
 	struct ltdb_reindex_context *ctx = (struct ltdb_reindex_context *)state;
 	struct ldb_module *module = ctx->module;
+	TDB_DATA key = {
+		.dptr = ldb_key.data,
+		.dsize = ldb_key.length
+	};
+	TDB_DATA key2 = {
+		.dptr = ldb_key2.data,
+		.dsize = ldb_key2.length
+	};
+	TDB_DATA data = {
+		.dptr = ldb_data.data,
+		.dsize = ldb_data.length
+	};
 
 	ldb = ldb_module_get_ctx(module);
 
@@ -1814,12 +1858,46 @@ static int ltdb_tdb_update_in_iterate(struct ltdb_private *ltdb, TDB_DATA key, T
 	return tdb_ret;
 }
 
-static int ltdb_tdb_parse_record(struct ltdb_private *ltdb, TDB_DATA key,
-				 int (*parser)(TDB_DATA key, TDB_DATA data,
+static int ltdb_tdb_parse_record_wrapper(TDB_DATA tdb_key, TDB_DATA tdb_data,
+					 void *ctx)
+{
+	struct kv_ctx *kv_ctx = ctx;
+	struct ldb_val key = {
+		.length = tdb_key.dsize,
+		.data = tdb_key.dptr,
+	};
+	struct ldb_val data = {
+		.length = tdb_data.dsize,
+		.data = tdb_data.dptr,
+	};
+
+	return kv_ctx->parser(key, data, kv_ctx->ctx);
+}
+
+static int ltdb_tdb_parse_record(struct ltdb_private *ltdb,
+				 struct ldb_val ldb_key,
+				 int (*parser)(struct ldb_val key,
+					       struct ldb_val data,
 					       void *private_data),
 				 void *ctx)
 {
-	return tdb_parse_record(ltdb->tdb, key, parser, ctx);
+	struct kv_ctx kv_ctx = {
+		.parser = parser,
+		.ctx = ctx,
+		.ltdb = ltdb
+	};
+	TDB_DATA key = {
+		.dptr = ldb_key.data,
+		.dsize = ldb_key.length
+	};
+	int ret;
+
+	ret = tdb_parse_record(ltdb->tdb, key, ltdb_tdb_parse_record_wrapper,
+			       &kv_ctx);
+	if (ret == 0) {
+		return LDB_SUCCESS;
+	}
+	return ltdb_err_map(tdb_error(ltdb->tdb));
 }
 
 static const char * ltdb_tdb_name(struct ltdb_private *ltdb)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.h b/lib/ldb/ldb_tdb/ldb_tdb.h
index 28dd209..f14666b 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.h
+++ b/lib/ldb/ldb_tdb/ldb_tdb.h
@@ -10,13 +10,13 @@ typedef int (*ldb_kv_traverse_fn)(struct ltdb_private *ltdb,
 				  void *ctx);
 
 struct kv_db_ops {
-	int (*store)(struct ltdb_private *ltdb, TDB_DATA key, TDB_DATA data, int flags);
-	int (*delete)(struct ltdb_private *ltdb, TDB_DATA key);
+	int (*store)(struct ltdb_private *ltdb, struct ldb_val key, struct ldb_val data, int flags);
+	int (*delete)(struct ltdb_private *ltdb, struct ldb_val key);
 	int (*iterate)(struct ltdb_private *ltdb, ldb_kv_traverse_fn fn, void *ctx);
-	int (*update_in_iterate)(struct ltdb_private *ltdb, TDB_DATA key,
-				 TDB_DATA key2, TDB_DATA data, void *ctx);
-	int (*fetch_and_parse)(struct ltdb_private *ltdb, TDB_DATA key,
-                               int (*parser)(TDB_DATA key, TDB_DATA data,
+	int (*update_in_iterate)(struct ltdb_private *ltdb, struct ldb_val key,
+				 struct ldb_val key2, struct ldb_val data, void *ctx);
+	int (*fetch_and_parse)(struct ltdb_private *ltdb, struct ldb_val key,
+                               int (*parser)(struct ldb_val key, struct ldb_val data,
                                              void *private_data),
                                void *ctx);
 	int (*lock_read)(struct ldb_module *);
diff --git a/lib/ldb/tests/ldb_kv_ops_test.c b/lib/ldb/tests/ldb_kv_ops_test.c
new file mode 100644
index 0000000..4cfb618
--- /dev/null
+++ b/lib/ldb/tests/ldb_kv_ops_test.c
@@ -0,0 +1,1396 @@
+/*
+ * Tests exercising the ldb key value operations.
+ *
+ *  Copyright (C) Andrew Bartlett <abartlet at samba.org> 2018
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+/*
+ * from cmocka.c:
+ * These headers or their equivalents should be included prior to
+ * including
+ * this header file.
+ *
+ * #include <stdarg.h>
+ * #include <stddef.h>
+ * #include <setjmp.h>
+ *
+ * This allows test applications to use custom definitions of C standard
+ * library functions and types.
+ *
+ */
+
+/*
+ * A KV module is expected to have the following behaviour
+ *
+ * - A transaction must be open to perform any read, write or delete operation
+ * - Writes and Deletes should not be visible until a transaction is commited
+ * - Nested transactions are not permitted
+ * - transactions can be rolled back and commited.
+ * - supports iteration over all records in the database
+ * - supports the update_in_iterate operation allowing entries to be
+ *   re-keyed.
+ */
+#include <stdarg.h>
+#include <stddef.h>
+#include <setjmp.h>
+#include <cmocka.h>
+
+#include <errno.h>
+#include <unistd.h>
+#include <talloc.h>
+
+#define TEVENT_DEPRECATED 1
+#include <tevent.h>
+
+#include <ldb.h>
+#include <ldb_module.h>
+#include <ldb_private.h>
+#include <string.h>
+#include <ctype.h>
+
+#include <sys/wait.h>
+
+#include "lmdb.h"
+#include "ldb_tdb/ldb_tdb.h"
+
+
+#define DEFAULT_BE  "tdb"
+
+#ifndef TEST_BE
+#define TEST_BE DEFAULT_BE
+#endif /* TEST_BE */
+
+#define NUM_RECS 1024
+
+
+struct test_ctx {
+	struct tevent_context *ev;
+	struct ldb_context *ldb;
+


-- 
Samba Shared Repository



More information about the samba-cvs mailing list