[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Thu Apr 4 07:46:02 UTC 2019
The branch, master has been updated
via 00874b61444 python join: Set index transaction cache size.
via 69408654b76 lib ldb key_value: Set index cache size on open
via 0952f98e415 lib ldb key_value: set the cache size for re-indexing
via 6129a05ca0c lib ldb key_value: Add get_size method
via 8f7bf13b968 lib ldb key_value: Pass index cache size
via 267cc71f5d5 lib ldb key_value: Remove index cache lazy initialisation
via 5ba049b599b flapping tests: Add samba3.smb2.notify
from 377d27359cc s3-libnet_join: allow fallback to NTLMSSP auth in libnet_join
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 00874b614449b094300a4e249ead22618031b1d8
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Apr 1 16:49:38 2019 +1300
python join: Set index transaction cache size.
The default value is too small for joining a large domain. So we specify a
size of 200,000 which is suitable for domains with up to 100,000 users.
At a later date this could be added as a parameter to the join, but
200,000 should be suitable for now.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Thu Apr 4 07:45:03 UTC 2019 on sn-devel-144
commit 69408654b7656189de0c37b0994c668a14e518a3
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Apr 1 16:33:52 2019 +1300
lib ldb key_value: Set index cache size on open
Set the default index cache from the passed option
"transaction_index_cache_size" on open. This allows the default cache
size to be overridden when processing large transactions i.e. joining a
large domain.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 0952f98e41558acdd2b9732a28f9469b2be4c757
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Apr 1 15:28:31 2019 +1300
lib ldb key_value: set the cache size for re-indexing
Set the index cache size to the number of records in the databse when
reindexing.
This significantly improves reindex performance. For a domain with
100,000 users the reindex times are reduced from 17 minutes to 45
seconds.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 6129a05ca0cc39a8043d58f793c4ec92e4c40182
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Apr 1 15:27:32 2019 +1300
lib ldb key_value: Add get_size method
Add the get_size method to the ldb_key_value layer, this will allow the
reindexing code to get an estimate of the number of records in the
database.
The lmdb backend returns an accurate count of the number of records in
the database withe the mdb_env_stat call.
The tdb backend does not provide a low cost method to determine the
number of records on the database. It does provide a tdb_summary call
however this this walks the entire database.
So for tdb we use the map size divided by 500, this over estimates the counts
for small domains, but the extra memory allocated for the cache should
not be significant.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 8f7bf13b96841b38cdc60765a5465be940e1477b
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Apr 1 14:10:10 2019 +1300
lib ldb key_value: Pass index cache size
Pass the index cache size to ldb_kv_index_transaction_start. This will
allow it to be set for reindex and join operations, where the current
defaults result in a significant performance penalty on large databases.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 267cc71f5d55f0c45cde8e60ac1529f6a39d776f
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Apr 1 13:12:20 2019 +1300
lib ldb key_value: Remove index cache lazy initialisation
Remove the lazy initialisation of the index cache. This make setting
the size of the cache for re-indexing easier, which will be done in
later commits.
Performance testing shows that the removal of lazy initialisation makes
no appreciable difference to performance.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 5ba049b599b5be2de1eabe548f1b770a5e89acdd
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Wed Apr 3 13:03:56 2019 +1300
flapping tests: Add samba3.smb2.notify
Add samba3.smb2.notify until Metze gets time to follow it up.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
-----------------------------------------------------------------------
Summary of changes:
lib/ldb/ldb_key_value/ldb_kv.c | 35 +++-
lib/ldb/ldb_key_value/ldb_kv.h | 20 +-
lib/ldb/ldb_key_value/ldb_kv_index.c | 34 ++-
lib/ldb/ldb_mdb/ldb_mdb.c | 22 ++
lib/ldb/ldb_tdb/ldb_tdb.c | 18 ++
lib/ldb/tests/ldb_key_value_test.c | 388 +++++++++++++++++++++++++++++++++++
lib/ldb/tests/ldb_kv_ops_test.c | 72 +++++++
lib/ldb/wscript | 10 +-
python/samba/join.py | 5 +
selftest/flapping.d/smb2_notify | 2 +
10 files changed, 592 insertions(+), 14 deletions(-)
create mode 100644 lib/ldb/tests/ldb_key_value_test.c
create mode 100644 selftest/flapping.d/smb2_notify
Changeset truncated at 500 lines:
diff --git a/lib/ldb/ldb_key_value/ldb_kv.c b/lib/ldb/ldb_key_value/ldb_kv.c
index d4f896736a2..bd35feed14b 100644
--- a/lib/ldb/ldb_key_value/ldb_kv.c
+++ b/lib/ldb/ldb_key_value/ldb_kv.c
@@ -1382,7 +1382,9 @@ static int ldb_kv_start_trans(struct ldb_module *module)
return ldb_kv->kv_ops->error(ldb_kv);
}
- ldb_kv_index_transaction_start(module);
+ ldb_kv_index_transaction_start(
+ module,
+ ldb_kv->index_transaction_cache_size);
ldb_kv->reindex_failed = false;
@@ -1946,5 +1948,36 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
}
}
+ /*
+ * Set the size of the transaction index cache.
+ * If the ldb option "transaction_index_cache_size" is set use that
+ * otherwise use DEFAULT_INDEX_CACHE_SIZE
+ */
+ ldb_kv->index_transaction_cache_size = DEFAULT_INDEX_CACHE_SIZE;
+ {
+ const char *size = ldb_options_find(
+ ldb,
+ options,
+ "transaction_index_cache_size");
+ if (size != NULL) {
+ size_t cache_size = 0;
+ errno = 0;
+
+ cache_size = strtoul( size, NULL, 0);
+ if (cache_size == 0 || errno == ERANGE) {
+ ldb_debug(
+ ldb,
+ LDB_DEBUG_WARNING,
+ "Invalid transaction_index_cache_size "
+ "value [%s], using default(%d)\n",
+ size,
+ DEFAULT_INDEX_CACHE_SIZE);
+ } else {
+ ldb_kv->index_transaction_cache_size =
+ cache_size;
+ }
+ }
+ }
+
return LDB_SUCCESS;
}
diff --git a/lib/ldb/ldb_key_value/ldb_kv.h b/lib/ldb/ldb_key_value/ldb_kv.h
index 5070a588c00..a4aa5ed9e62 100644
--- a/lib/ldb/ldb_key_value/ldb_kv.h
+++ b/lib/ldb/ldb_key_value/ldb_kv.h
@@ -4,6 +4,8 @@
#include "tdb.h"
#include "ldb_module.h"
+#ifndef __LDB_KV_H__
+#define __LDB_KV_H__
struct ldb_kv_private;
typedef int (*ldb_kv_traverse_fn)(struct ldb_kv_private *ldb_kv,
struct ldb_val key,
@@ -41,6 +43,7 @@ struct kv_db_ops {
const char *(*name)(struct ldb_kv_private *ldb_kv);
bool (*has_changed)(struct ldb_kv_private *ldb_kv);
bool (*transaction_active)(struct ldb_kv_private *ldb_kv);
+ size_t (*get_size)(struct ldb_kv_private *ldb_kv);
};
/* this private structure is used by the key value backends in the
@@ -100,6 +103,11 @@ struct ldb_kv_private {
* fork()ed child.
*/
pid_t pid;
+
+ /*
+ * The size to be used for the index transaction cache
+ */
+ size_t index_transaction_cache_size;
};
struct ldb_kv_context {
@@ -175,6 +183,13 @@ int ldb_kv_check_at_attributes_values(const struct ldb_val *value);
* The following definitions come from lib/ldb/ldb_key_value/ldb_kv_index.c
*/
+/*
+ * The default size of the in memory TDB used to cache index records
+ * The value chosen gives a prime modulo for the hash table and keeps the
+ * tdb memory overhead under 4 kB
+ */
+#define DEFAULT_INDEX_CACHE_SIZE 491
+
struct ldb_parse_tree;
int ldb_kv_search_indexed(struct ldb_kv_context *ctx, uint32_t *);
@@ -197,7 +212,9 @@ int ldb_kv_index_del_value(struct ldb_module *module,
struct ldb_message_element *el,
unsigned int v_idx);
int ldb_kv_reindex(struct ldb_module *module);
-int ldb_kv_index_transaction_start(struct ldb_module *module);
+int ldb_kv_index_transaction_start(
+ struct ldb_module *module,
+ size_t cache_size);
int ldb_kv_index_transaction_commit(struct ldb_module *module);
int ldb_kv_index_transaction_cancel(struct ldb_module *module);
int ldb_kv_key_dn_from_idx(struct ldb_module *module,
@@ -263,3 +280,4 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
struct ldb_context *ldb,
const char *options[],
struct ldb_module **_module);
+#endif /* __LDB_KV_H__ */
diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c
index 4dfcc2cb0ba..4101d4b986d 100644
--- a/lib/ldb/ldb_key_value/ldb_kv_index.c
+++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
@@ -199,7 +199,9 @@ static unsigned ldb_kv_max_key_length(struct ldb_kv_private *ldb_kv)
}
/* enable the idxptr mode when transactions start */
-int ldb_kv_index_transaction_start(struct ldb_module *module)
+int ldb_kv_index_transaction_start(
+ struct ldb_module *module,
+ size_t cache_size)
{
struct ldb_kv_private *ldb_kv = talloc_get_type(
ldb_module_get_private(module), struct ldb_kv_private);
@@ -208,6 +210,16 @@ int ldb_kv_index_transaction_start(struct ldb_module *module)
return ldb_oom(ldb_module_get_ctx(module));
}
+ ldb_kv->idxptr->itdb = tdb_open(
+ NULL,
+ cache_size,
+ TDB_INTERNAL,
+ O_RDWR,
+ 0);
+ if (ldb_kv->idxptr->itdb == NULL) {
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
return LDB_SUCCESS;
}
@@ -717,14 +729,6 @@ static int ldb_kv_dn_list_store(struct ldb_module *module,
return ldb_kv_dn_list_store_full(module, ldb_kv, dn, list);
}
- if (ldb_kv->idxptr->itdb == NULL) {
- ldb_kv->idxptr->itdb =
- tdb_open(NULL, 1000, TDB_INTERNAL, O_RDWR, 0);
- if (ldb_kv->idxptr->itdb == NULL) {
- return LDB_ERR_OPERATIONS_ERROR;
- }
- }
-
key.dptr = discard_const_p(unsigned char, ldb_dn_get_linearized(dn));
if (key.dptr == NULL) {
return LDB_ERR_OPERATIONS_ERROR;
@@ -3088,6 +3092,7 @@ int ldb_kv_reindex(struct ldb_module *module)
ldb_module_get_private(module), struct ldb_kv_private);
int ret;
struct ldb_kv_reindex_context ctx;
+ size_t index_cache_size = 0;
/*
* Only triggered after a modification, but make clear we do
@@ -3108,7 +3113,16 @@ int ldb_kv_reindex(struct ldb_module *module)
*/
ldb_kv_index_transaction_cancel(module);
- ret = ldb_kv_index_transaction_start(module);
+ /*
+ * Calculate the size of the index cache that we'll need for
+ * the re-index
+ */
+ index_cache_size = ldb_kv->kv_ops->get_size(ldb_kv);
+ if (index_cache_size < DEFAULT_INDEX_CACHE_SIZE) {
+ index_cache_size = DEFAULT_INDEX_CACHE_SIZE;
+ }
+
+ ret = ldb_kv_index_transaction_start(module, index_cache_size);
if (ret != LDB_SUCCESS) {
return ret;
}
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
index 646a67c554c..1e94d660862 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.c
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -576,6 +576,27 @@ static bool lmdb_changed(struct ldb_kv_private *ldb_kv)
return true;
}
+/*
+ * Get the number of records in the database.
+ *
+ * The mdb_env_stat call returns an accurate count, so we return the actual
+ * number of records in the database rather than an estimate.
+ */
+static size_t lmdb_get_size(struct ldb_kv_private *ldb_kv) {
+
+ struct MDB_stat stats = {0};
+ struct lmdb_private *lmdb = ldb_kv->lmdb_private;
+ int ret = 0;
+
+ ret = mdb_env_stat(lmdb->env, &stats);
+ if (ret != 0) {
+ return 0;
+ }
+ return stats.ms_entries;
+}
+
+
+
static struct kv_db_ops lmdb_key_value_ops = {
.store = lmdb_store,
.delete = lmdb_delete,
@@ -593,6 +614,7 @@ static struct kv_db_ops lmdb_key_value_ops = {
.name = lmdb_name,
.has_changed = lmdb_changed,
.transaction_active = lmdb_transaction_active,
+ .get_size = lmdb_get_size,
};
static const char *lmdb_get_path(const char *url)
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 812ddd3e389..9f16040659b 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -400,6 +400,23 @@ static bool ltdb_transaction_active(struct ldb_kv_private *ldb_kv)
return tdb_transaction_active(ldb_kv->tdb);
}
+/*
+ * Get an estimate of the number of records in a tdb database.
+ *
+ * This implementation will overestimate the number of records in a sparsely
+ * populated database. The size estimate is only used for allocating
+ * an in memory tdb to cache index records during a reindex, overestimating
+ * the contents is acceptable, and preferable to underestimating
+ */
+#define RECORD_SIZE 500
+static size_t ltdb_get_size(struct ldb_kv_private *ldb_kv)
+{
+ size_t map_size = tdb_map_size(ldb_kv->tdb);
+ size_t size = map_size / RECORD_SIZE;
+
+ return size;
+}
+
static const struct kv_db_ops key_value_ops = {
.store = ltdb_store,
.delete = ltdb_delete,
@@ -417,6 +434,7 @@ static const struct kv_db_ops key_value_ops = {
.name = ltdb_name,
.has_changed = ltdb_changed,
.transaction_active = ltdb_transaction_active,
+ .get_size = ltdb_get_size,
};
/*
diff --git a/lib/ldb/tests/ldb_key_value_test.c b/lib/ldb/tests/ldb_key_value_test.c
new file mode 100644
index 00000000000..3f31bb98350
--- /dev/null
+++ b/lib/ldb/tests/ldb_key_value_test.c
@@ -0,0 +1,388 @@
+/*
+ * Tests exercising the ldb key value operations.
+ *
+ * Copyright (C) Andrew Bartlett <abartlet at samba.org> 2019
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+/*
+ * from cmocka.c:
+ * These headers or their equivalents should be included prior to
+ * including
+ * this header file.
+ *
+ * #include <stdarg.h>
+ * #include <stddef.h>
+ * #include <setjmp.h>
+ *
+ * This allows test applications to use custom definitions of C standard
+ * library functions and types.
+ *
+ */
+
+/*
+ *
+ * Tests for the ldb key value layer
+ */
+#include <stdarg.h>
+#include <stddef.h>
+#include <setjmp.h>
+#include <cmocka.h>
+
+#include <errno.h>
+#include <unistd.h>
+#include <talloc.h>
+#include <tevent.h>
+#include <string.h>
+#include <ctype.h>
+
+#include <sys/wait.h>
+
+#include "ldb_key_value/ldb_kv.c"
+#include "ldb_key_value/ldb_kv_index.c"
+#include "ldb_key_value/ldb_kv_search.c"
+
+#define DEFAULT_BE "tdb"
+
+#ifndef TEST_BE
+#define TEST_BE DEFAULT_BE
+#endif /* TEST_BE */
+
+#define NUM_RECS 1024
+int ldb_kv_cache_reload(struct ldb_module *module) {
+ return LDB_SUCCESS;
+}
+int ldb_kv_cache_load(struct ldb_module *module) {
+ return LDB_SUCCESS;
+}
+int ldb_kv_check_at_attributes_values(const struct ldb_val *value) {
+ return LDB_SUCCESS;
+}
+int ldb_kv_increase_sequence_number(struct ldb_module *module) {
+ return LDB_SUCCESS;
+}
+
+struct test_ctx {
+};
+
+static int setup(void **state)
+{
+ struct test_ctx *test_ctx;
+
+ test_ctx = talloc_zero(NULL, struct test_ctx);
+ *state = test_ctx;
+ return 0;
+}
+
+static int teardown(void **state)
+{
+ struct test_ctx *test_ctx = talloc_get_type_abort(*state,
+ struct test_ctx);
+
+ talloc_free(test_ctx);
+ return 0;
+}
+
+/*
+ * Test that the index cache is opened by ldb_kv_index_transaction_start
+ * and correctly initialised with the passed index cache size.
+ */
+static void test_index_cache_init(void **state)
+{
+ struct test_ctx *test_ctx = talloc_get_type_abort(
+ *state,
+ struct test_ctx);
+ struct ldb_module *module = NULL;
+ struct ldb_kv_private *ldb_kv = NULL;
+ int ret = LDB_SUCCESS;
+
+ module = talloc_zero(test_ctx, struct ldb_module);
+ ldb_kv = talloc_zero(test_ctx, struct ldb_kv_private);
+ ldb_module_set_private(module, ldb_kv);
+
+ ret = ldb_kv_index_transaction_start(module, 191);
+ assert_int_equal(LDB_SUCCESS, ret);
+
+ assert_non_null(ldb_kv->idxptr);
+ assert_non_null(ldb_kv->idxptr->itdb);
+ assert_int_equal(191, tdb_hash_size(ldb_kv->idxptr->itdb));
+
+ TALLOC_FREE(ldb_kv);
+ TALLOC_FREE(module);
+}
+
+static int mock_begin_write(struct ldb_kv_private* ldb_kv) {
+ return LDB_SUCCESS;
+}
+static int mock_abort_write(struct ldb_kv_private* ldb_kv) {
+ return LDB_SUCCESS;
+}
+
+/*
+ * Test that the index cache is set to the default cache size at the start of
+ * a transaction.
+ */
+static void test_default_index_cache_size(void **state)
+{
+ struct test_ctx *test_ctx = talloc_get_type_abort(
+ *state,
+ struct test_ctx);
+ struct ldb_module *module = NULL;
+ struct ldb_kv_private *ldb_kv = NULL;
+ int ret = LDB_SUCCESS;
+ const struct kv_db_ops ops = {
+ .begin_write = mock_begin_write,
+ .abort_write = mock_abort_write
+ };
+
+ module = talloc_zero(test_ctx, struct ldb_module);
+ ldb_kv = talloc_zero(test_ctx, struct ldb_kv_private);
+ ldb_kv->pid = getpid();
+ ldb_kv->kv_ops = &ops;
+ ldb_kv->index_transaction_cache_size = DEFAULT_INDEX_CACHE_SIZE;
+ ldb_module_set_private(module, ldb_kv);
+
+ ret = ldb_kv_start_trans(module);
+ assert_int_equal(LDB_SUCCESS, ret);
+
+ assert_int_equal(
+ DEFAULT_INDEX_CACHE_SIZE,
+ tdb_hash_size(ldb_kv->idxptr->itdb));
+
+ ret = ldb_kv_del_trans(module);
+ assert_int_equal(LDB_SUCCESS, ret);
+
+ TALLOC_FREE(ldb_kv);
+ TALLOC_FREE(module);
+}
+
+static int db_size = 0;
+static size_t mock_get_size(struct ldb_kv_private *ldb_kv) {
+ return db_size;
+}
+
+static int mock_iterate(
+ struct ldb_kv_private *ldb_kv,
+ ldb_kv_traverse_fn fn,
+ void *ctx) {
+ return 1;
+}
+
+/*
+ * Test that the index cache is correctly sized by the re_index call
+ */
+static void test_reindex_cache_size(void **state)
+{
+ struct test_ctx *test_ctx = talloc_get_type_abort(
+ *state,
+ struct test_ctx);
+ struct ldb_module *module = NULL;
+ struct ldb_kv_private *ldb_kv = NULL;
+ int ret = LDB_SUCCESS;
+ const struct kv_db_ops ops = {
+ .iterate = mock_iterate,
+ .get_size = mock_get_size,
+ };
+
+ module = talloc_zero(test_ctx, struct ldb_module);
+ ldb_kv = talloc_zero(test_ctx, struct ldb_kv_private);
+ ldb_kv->kv_ops = &ops;
+ ldb_module_set_private(module, ldb_kv);
+
+ /*
+ * Use a value less than the DEFAULT_INDEX_CACHE_SIZE
+ * Should get the DEFAULT_INDEX_CACHE_SIZE
+ */
+ db_size = DEFAULT_INDEX_CACHE_SIZE - 1;
+ ret = ldb_kv_reindex(module);
+ assert_int_equal(LDB_SUCCESS, ret);
+
+ assert_int_equal(
+ DEFAULT_INDEX_CACHE_SIZE,
+ tdb_hash_size(ldb_kv->idxptr->itdb));
+
+ /*
+ * Use a value greater than the DEFAULT_INDEX_CACHE_SIZE
+ * Should get the value specified.
+ */
+ db_size = DEFAULT_INDEX_CACHE_SIZE + 1;
+ ret = ldb_kv_reindex(module);
+ assert_int_equal(LDB_SUCCESS, ret);
+
+ assert_int_equal(db_size, tdb_hash_size(ldb_kv->idxptr->itdb));
+
+ TALLOC_FREE(ldb_kv);
+ TALLOC_FREE(module);
+}
+
--
Samba Shared Repository
More information about the samba-cvs
mailing list