[SCM] Samba Shared Repository - branch v4-11-test updated
Karolin Seeger
kseeger at samba.org
Wed May 13 14:45:02 UTC 2020
The branch, v4-11-test has been updated
via 84362eef4cf vfs_shadow_copy2: implement case canonicalisation in shadow_copy2_get_real_filename()
via 3d60f8ac9fa s3/lib: add is_gmt_token()
via f8805f5db4d smbd: make get_real_filename_full_scan() public
via f23992c09a6 CI: add two tests for shadow_copy2 VFS module
via ecaeedb3183 ldb: Bump version to 2.0.11
via d65f5a1567e lib ldb: lmdb init var before calling mdb_reader_check
via 5fb94e3efc6 lib ldb: lmdb clear stale readers on write txn start
via 9d8ca2233cd ldb tests: Confirm lmdb free list handling
from b6a7b42f9a8 VERSION: Bump version up to 4.11.10...
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-11-test
- Log -----------------------------------------------------------------
commit 84362eef4cfd199e0ef3e019f6ed444d1a73ed0d
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 23 16:10:23 2020 +0200
vfs_shadow_copy2: implement case canonicalisation in shadow_copy2_get_real_filename()
unix_convert() can't do this for us in snapdirseverywhere mode, so we do it
ourselves.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14350
Signed-off-by: Ralph Boehme <slow at samba.org>
(Similar to commit a3d1ac2a597e2441d6855db566306298ae5db99b)
Autobuild-User(v4-11-test): Karolin Seeger <kseeger at samba.org>
Autobuild-Date(v4-11-test): Wed May 13 14:44:57 UTC 2020 on sn-devel-184
commit 3d60f8ac9fa2440fa559fd72f86adc53268254da
Author: Ralph Boehme <slow at samba.org>
Date: Mon Apr 27 14:38:28 2020 +0200
s3/lib: add is_gmt_token()
This is not present in master as master has been converted to use struct
smb_filename.twrp instead of @GMT string tokens as part of the path.
Signed-off-by: Ralph Boehme <slow at samba.org>
commit f8805f5db4d350ecc4b97c4ef4cc207eed18ae56
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 23 16:09:16 2020 +0200
smbd: make get_real_filename_full_scan() public
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14350
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(backported from commit aa5f19ddf1dec1ac4386441929bca94727f30ee6)
[Conflicts: source3/smbd/proto.h: more functions are missing in 4.12]
commit f23992c09a6fb4f5647e6805e1701000ca1dffa8
Author: Ralph Boehme <slow at samba.org>
Date: Tue Apr 21 13:06:03 2020 +0200
CI: add two tests for shadow_copy2 VFS module
Note that the test "fetch a previous version of a regular file via non-canonical
basepath" doesn't fail by "luck" because it runs into the "creating file"
optimisation in unix_convert().
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14350
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 6557777c86d72a185b3fe4061a8b5791fd748924)
commit ecaeedb3183494c6901e30778a02adcdf7eb5f2a
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Fri May 1 09:15:49 2020 +1200
ldb: Bump version to 2.0.11
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14330
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
commit d65f5a1567ed44490197f60dbdc95a1ebf7d1421
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Wed Apr 1 08:22:08 2020 +1300
lib ldb: lmdb init var before calling mdb_reader_check
Initilalise "stale" to zero before passing a pointer to it to
mdb_reader_check.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andreas Schneider <asn at samba.org>
Autobuild-User(master): Andreas Schneider <asn at cryptomilk.org>
Autobuild-Date(master): Tue Apr 7 12:29:00 UTC 2020 on sn-devel-184
(cherry picked from commit d1f4002b914efb31aa34a59e7c93d80f3174727c)
commit 5fb94e3efc6efbbee8ce52f6e0146d9571fc5187
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Mar 30 12:08:30 2020 +1300
lib ldb: lmdb clear stale readers on write txn start
In use process failures and Bind9 shut downs leave stale entries in the
lmdb reader table. This can result in lmdb filling it's database file, as
the free list can not be reclaimed due to the stale reader.
In this fix we call mdb_reader_check at the start of each transaction,
to free any stale readers. As the default maximum number of readers is
127, this should not impact on performance to any great extent.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=14330
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Tue Mar 31 01:26:07 UTC 2020 on sn-devel-184
(cherry picked from commit 89041a6d18a1d091ea713e6986cac5ca66c2b481)
commit 9d8ca2233cde625e0d64c427e417ae1d32da3714
Author: Gary Lockyer <gary at catalyst.net.nz>
Date: Mon Mar 16 15:18:12 2020 +1300
ldb tests: Confirm lmdb free list handling
Add cmocka tests to confirm lmdb's handling of the free list.
As a result of lmdb's MVCC (Multiversion Concurrency Control) long
running read transactions or stale readers (read transactions where the
process exited without ending the transaction) can cause the database to
run out of space.
Items in the free list are only reused when they would not be visible in
a read transaction. So long running read transactions prevent entries
in the free list being reused, and the database can run out of space.
Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit fc13304d1575ad6bc6e04cdb3eedf46d3c3678c7)
-----------------------------------------------------------------------
Summary of changes:
lib/ldb/ABI/{ldb-2.0.10.sigs => ldb-2.0.11.sigs} | 0
...ldb-util-1.1.10.sigs => pyldb-util-2.0.11.sigs} | 0
lib/ldb/ldb_mdb/ldb_mdb.c | 17 +
lib/ldb/tests/ldb_lmdb_free_list_test.c | 661 +++++++++++++++++++++
lib/ldb/wscript | 10 +-
source3/include/proto.h | 1 +
source3/lib/filename_util.c | 19 +
source3/modules/vfs_shadow_copy2.c | 91 ++-
source3/script/tests/test_shadow_copy.sh | 12 +
source3/smbd/filename.c | 10 +-
source3/smbd/proto.h | 6 +
11 files changed, 813 insertions(+), 14 deletions(-)
copy lib/ldb/ABI/{ldb-2.0.10.sigs => ldb-2.0.11.sigs} (100%)
copy lib/ldb/ABI/{pyldb-util-1.1.10.sigs => pyldb-util-2.0.11.sigs} (100%)
create mode 100644 lib/ldb/tests/ldb_lmdb_free_list_test.c
Changeset truncated at 500 lines:
diff --git a/lib/ldb/ABI/ldb-2.0.10.sigs b/lib/ldb/ABI/ldb-2.0.11.sigs
similarity index 100%
copy from lib/ldb/ABI/ldb-2.0.10.sigs
copy to lib/ldb/ABI/ldb-2.0.11.sigs
diff --git a/lib/ldb/ABI/pyldb-util-1.1.10.sigs b/lib/ldb/ABI/pyldb-util-2.0.11.sigs
similarity index 100%
copy from lib/ldb/ABI/pyldb-util-1.1.10.sigs
copy to lib/ldb/ABI/pyldb-util-2.0.11.sigs
diff --git a/lib/ldb/ldb_mdb/ldb_mdb.c b/lib/ldb/ldb_mdb/ldb_mdb.c
index 6c679c214b8..f0a418d07ee 100644
--- a/lib/ldb/ldb_mdb/ldb_mdb.c
+++ b/lib/ldb/ldb_mdb/ldb_mdb.c
@@ -641,6 +641,23 @@ static int lmdb_transaction_start(struct ldb_kv_private *ldb_kv)
return LDB_ERR_PROTOCOL_ERROR;
}
+ /*
+ * Clear out any stale readers
+ */
+ {
+ int stale = 0;
+ mdb_reader_check(lmdb->env, &stale);
+ if (stale > 0) {
+ ldb_debug(
+ lmdb->ldb,
+ LDB_DEBUG_ERROR,
+ "LMDB Stale readers, deleted (%d)",
+ stale);
+ }
+ }
+
+
+
ltx_head = lmdb_private_trans_head(lmdb);
tx_parent = lmdb_trans_get_tx(ltx_head);
diff --git a/lib/ldb/tests/ldb_lmdb_free_list_test.c b/lib/ldb/tests/ldb_lmdb_free_list_test.c
new file mode 100644
index 00000000000..9b295460730
--- /dev/null
+++ b/lib/ldb/tests/ldb_lmdb_free_list_test.c
@@ -0,0 +1,661 @@
+/*
+ * Copyright (C) Catalyst.Net Ltd 2020
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+/*
+ * Tests confirming lmdb's handling of the free space list in the presence
+ * of active and stale readers. A stale reader is a process that opens a
+ * read lock and then exits without releasing the lock.
+ *
+ * lmdb uses MVCC to maintain databased consistency, new copies of updated
+ * records are written to the database. The old entries are only
+ * reused when they are no longer referenced in a read transaction.
+ *
+ * The tests all update a single record multiple times
+ *
+ * If there is a read transaction or a stale reader lmdb will report
+ * out of space.
+ *
+ * If no read transaction and no stale reader, lmdb reclaims space from the
+ * free list.
+ */
+
+/*
+ * from cmocka.c:
+ * These headers or their equivalents should be included prior to
+ * including
+ * this header file.
+ *
+ * #include <stdarg.h>
+ * #include <stddef.h>
+ * #include <setjmp.h>
+ *
+ * This allows test applications to use custom definitions of C standard
+ * library functions and types.
+ *
+ */
+
+#include <stdarg.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <setjmp.h>
+#include <cmocka.h>
+
+#include <errno.h>
+#include <unistd.h>
+#include <talloc.h>
+#include <tevent.h>
+#include <ldb.h>
+#include <ldb_module.h>
+#include <ldb_private.h>
+#include <string.h>
+#include <ctype.h>
+
+#include <sys/wait.h>
+
+#include "ldb_tdb/ldb_tdb.h"
+#include "ldb_key_value/ldb_kv.h"
+
+#define DEFAULT_BE "mdb"
+
+#ifndef TEST_BE
+#define TEST_BE DEFAULT_BE
+#endif /* TEST_BE */
+
+const int RECORD_SIZE = 6144;
+const int ITERATIONS = 3;
+
+struct test_ctx {
+ struct tevent_context *ev;
+ struct ldb_context *ldb;
+
+ const char *dbfile;
+ const char *lockfile; /* lockfile is separate */
+
+ const char *dbpath;
+};
+
+static void unlink_old_db(struct test_ctx *test_ctx)
+{
+ int ret;
+
+ errno = 0;
+ ret = unlink(test_ctx->lockfile);
+ if (ret == -1 && errno != ENOENT) {
+ fail();
+ }
+
+ errno = 0;
+ ret = unlink(test_ctx->dbfile);
+ if (ret == -1 && errno != ENOENT) {
+ fail();
+ }
+}
+
+static int noconn_setup(void **state)
+{
+ struct test_ctx *test_ctx;
+
+ test_ctx = talloc_zero(NULL, struct test_ctx);
+ assert_non_null(test_ctx);
+
+ test_ctx->ev = tevent_context_init(test_ctx);
+ assert_non_null(test_ctx->ev);
+
+ test_ctx->ldb = ldb_init(test_ctx, test_ctx->ev);
+ assert_non_null(test_ctx->ldb);
+
+ test_ctx->dbfile = talloc_strdup(test_ctx, "lmdb_free_list_test.ldb");
+ assert_non_null(test_ctx->dbfile);
+
+ test_ctx->lockfile =
+ talloc_asprintf(test_ctx, "%s-lock", test_ctx->dbfile);
+ assert_non_null(test_ctx->lockfile);
+
+ test_ctx->dbpath =
+ talloc_asprintf(test_ctx, TEST_BE "://%s", test_ctx->dbfile);
+ assert_non_null(test_ctx->dbpath);
+
+ unlink_old_db(test_ctx);
+ *state = test_ctx;
+ return 0;
+}
+
+static int noconn_teardown(void **state)
+{
+ struct test_ctx *test_ctx =
+ talloc_get_type_abort(*state, struct test_ctx);
+
+ unlink_old_db(test_ctx);
+ talloc_free(test_ctx);
+ return 0;
+}
+
+static int setup(void **state)
+{
+ struct test_ctx *test_ctx;
+ int ret;
+ struct ldb_ldif *ldif;
+ const char *index_ldif = "dn: @INDEXLIST\n"
+ "@IDXGUID: objectUUID\n"
+ "@IDX_DN_GUID: GUID\n"
+ "\n";
+ /*
+ * Use a 64KiB DB for this test
+ */
+ const char *options[] = {"lmdb_env_size:65536", NULL};
+
+ noconn_setup((void **)&test_ctx);
+
+ ret = ldb_connect(test_ctx->ldb, test_ctx->dbpath, 0, options);
+ assert_int_equal(ret, 0);
+
+ while ((ldif = ldb_ldif_read_string(test_ctx->ldb, &index_ldif))) {
+ ret = ldb_add(test_ctx->ldb, ldif->msg);
+ assert_int_equal(ret, LDB_SUCCESS);
+ }
+ *state = test_ctx;
+ return 0;
+}
+
+static int teardown(void **state)
+{
+ struct test_ctx *test_ctx =
+ talloc_get_type_abort(*state, struct test_ctx);
+ noconn_teardown((void **)&test_ctx);
+ return 0;
+}
+
+static struct ldb_kv_private *get_ldb_kv(struct ldb_context *ldb)
+{
+ void *data = NULL;
+ struct ldb_kv_private *ldb_kv = NULL;
+
+ data = ldb_module_get_private(ldb->modules);
+ assert_non_null(data);
+
+ ldb_kv = talloc_get_type(data, struct ldb_kv_private);
+ assert_non_null(ldb_kv);
+
+ return ldb_kv;
+}
+
+static int parse(struct ldb_val key, struct ldb_val data, void *private_data)
+{
+ struct ldb_val *read = private_data;
+
+ /* Yes, we leak this. That is OK */
+ read->data = talloc_size(NULL, data.length);
+ assert_non_null(read->data);
+
+ memcpy(read->data, data.data, data.length);
+ read->length = data.length;
+ return LDB_SUCCESS;
+}
+
+/*
+ * This test has the same structure as the test_free_list_read_lock
+ * except the parent process does not keep the read lock open while the
+ * child process is performing an update.
+ */
+static void test_free_list_no_read_lock(void **state)
+{
+ int ret;
+ struct test_ctx *test_ctx =
+ talloc_get_type_abort(*state, struct test_ctx);
+ struct ldb_kv_private *ldb_kv = get_ldb_kv(test_ctx->ldb);
+ struct ldb_val key;
+ struct ldb_val val;
+
+ const char *KEY1 = "KEY01";
+
+ /*
+ * Pipes etc to co-ordinate the processes
+ */
+ int to_child[2];
+ int to_parent[2];
+ char buf[2];
+ pid_t pid;
+ size_t i;
+
+ TALLOC_CTX *tmp_ctx;
+ tmp_ctx = talloc_new(test_ctx);
+ assert_non_null(tmp_ctx);
+
+ ret = pipe(to_child);
+ assert_int_equal(ret, 0);
+ ret = pipe(to_parent);
+ assert_int_equal(ret, 0);
+ /*
+ * Now fork a new process
+ */
+
+ pid = fork();
+ if (pid == 0) {
+ /*
+ * Child process
+ */
+
+ struct ldb_context *ldb = NULL;
+ close(to_child[1]);
+ close(to_parent[0]);
+
+ /*
+ * Wait for the parent to get ready.
+ */
+ ret = read(to_child[0], buf, 2);
+ assert_int_equal(ret, 2);
+
+ ldb = ldb_init(test_ctx, test_ctx->ev);
+ assert_non_null(ldb);
+
+ ret = ldb_connect(ldb, test_ctx->dbpath, 0, NULL);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ ldb_kv = get_ldb_kv(ldb);
+ assert_non_null(ldb_kv);
+ /*
+ * Add a record to the database
+ */
+ key.data = (uint8_t *)talloc_strdup(tmp_ctx, KEY1);
+ key.length = strlen(KEY1) + 1;
+ val.data = talloc_zero_size(tmp_ctx, RECORD_SIZE);
+ assert_non_null(val.data);
+ memset(val.data, 'x', RECORD_SIZE);
+ val.length = RECORD_SIZE;
+ /*
+ * Do more iterations than when a read lock, stale reader
+ * active to confirm that the space is being re-used.
+ */
+ for (i = 0; i < ITERATIONS * 10; i++) {
+ ret = ldb_kv->kv_ops->begin_write(ldb_kv);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ ret = ldb_kv->kv_ops->store(ldb_kv, key, val, 0);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ ret = ldb_kv->kv_ops->finish_write(ldb_kv);
+ assert_int_equal(ret, LDB_SUCCESS);
+ }
+
+ /*
+ * Signal the parent that we've done the updates
+ */
+ ret = write(to_parent[1], "GO", 2);
+ assert_int_equal(ret, 2);
+ exit(0);
+ }
+
+ close(to_child[0]);
+ close(to_parent[1]);
+
+ /*
+ * Begin a read transaction
+ */
+ ret = ldb_kv->kv_ops->lock_read(test_ctx->ldb->modules);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ /*
+ * Now close it
+ */
+ ret = ldb_kv->kv_ops->unlock_read(test_ctx->ldb->modules);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ /*
+ * Signal the child process
+ */
+ ret = write(to_child[1], "GO", 2);
+ assert_int_equal(2, ret);
+
+ /*
+ * Wait for the child process to update the record
+ */
+ ret = read(to_parent[0], buf, 2);
+ assert_int_equal(2, ret);
+
+ /*
+ * Begin a read transaction
+ */
+ ret = ldb_kv->kv_ops->lock_read(test_ctx->ldb->modules);
+ assert_int_equal(ret, LDB_SUCCESS);
+ /*
+ * read the record
+ * and close the transaction
+ */
+ key.data = (uint8_t *)talloc_strdup(tmp_ctx, KEY1);
+ key.length = strlen(KEY1) + 1;
+
+ ret = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key, parse, &val);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ ret = ldb_kv->kv_ops->unlock_read(test_ctx->ldb->modules);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ close(to_child[1]);
+ close(to_parent[0]);
+ TALLOC_FREE(tmp_ctx);
+}
+
+/*
+ * This test has the same structure as the test_free_list_read_lock
+ * except the parent process keeps the read lock open while the
+ * child process is performing an update.
+ */
+static void test_free_list_read_lock(void **state)
+{
+ int ret;
+ struct test_ctx *test_ctx =
+ talloc_get_type_abort(*state, struct test_ctx);
+ struct ldb_kv_private *ldb_kv = get_ldb_kv(test_ctx->ldb);
+ struct ldb_val key;
+ struct ldb_val val;
+
+ const char *KEY1 = "KEY01";
+
+ /*
+ * Pipes etc to co-ordinate the processes
+ */
+ int to_child[2];
+ int to_parent[2];
+ char buf[2];
+ pid_t pid;
+ size_t i;
+
+ TALLOC_CTX *tmp_ctx;
+ tmp_ctx = talloc_new(test_ctx);
+ assert_non_null(tmp_ctx);
+
+ ret = pipe(to_child);
+ assert_int_equal(ret, 0);
+ ret = pipe(to_parent);
+ assert_int_equal(ret, 0);
+ /*
+ * Now fork a new process
+ */
+
+ pid = fork();
+ if (pid == 0) {
+ /*
+ * Child process
+ */
+
+ struct ldb_context *ldb = NULL;
+ close(to_child[1]);
+ close(to_parent[0]);
+
+ /*
+ * Wait for the transaction to start
+ */
+ ret = read(to_child[0], buf, 2);
+ assert_int_equal(ret, 2);
+
+ ldb = ldb_init(test_ctx, test_ctx->ev);
+ assert_non_null(ldb);
+
+ ret = ldb_connect(ldb, test_ctx->dbpath, 0, NULL);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ ldb_kv = get_ldb_kv(ldb);
+ assert_non_null(ldb_kv);
+ /*
+ * Add a record to the database
+ */
+ key.data = (uint8_t *)talloc_strdup(tmp_ctx, KEY1);
+ key.length = strlen(KEY1) + 1;
+ val.data = talloc_zero_size(tmp_ctx, RECORD_SIZE);
+ assert_non_null(val.data);
+ memset(val.data, 'x', RECORD_SIZE);
+ val.length = RECORD_SIZE;
+ for (i = 0; i < ITERATIONS; i++) {
+ ret = ldb_kv->kv_ops->begin_write(ldb_kv);
+ assert_int_equal(ret, 0);
+ ret = ldb_kv->kv_ops->store(ldb_kv, key, val, 0);
+ if (ret == LDB_ERR_BUSY && i > 0) {
+ int rc = ldb_kv->kv_ops->abort_write(ldb_kv);
+ assert_int_equal(rc, LDB_SUCCESS);
+ break;
+ }
+ assert_int_equal(ret, LDB_SUCCESS);
+ ret = ldb_kv->kv_ops->finish_write(ldb_kv);
+ assert_int_equal(ret, LDB_SUCCESS);
+ }
+ assert_int_equal(ret, LDB_ERR_BUSY);
+ assert_int_not_equal(i, 0);
+
+ /*
+ * Begin a read transaction
+ */
+ ret = ldb_kv->kv_ops->lock_read(ldb->modules);
+ assert_int_equal(ret, LDB_SUCCESS);
+ /*
+ * read the record
+ * and close the transaction
+ */
+ key.data = (uint8_t *)talloc_strdup(tmp_ctx, KEY1);
+ key.length = strlen(KEY1) + 1;
+
+ ret = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key, parse, &val);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ ret = ldb_kv->kv_ops->unlock_read(ldb->modules);
+ assert_int_equal(ret, LDB_SUCCESS);
+
+ /*
+ * Signal the the parent that we've done the update
--
Samba Shared Repository
More information about the samba-cvs
mailing list