[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Wed May 29 05:58:02 UTC 2019
The branch, master has been updated
via 8831b06d3d4 ldb: Release ldb 2.0.3
via 4a95410a154 downgradedatabase: blackbox: MDB backend
via 0942a65b63c downgradedatabase: adding special case for MDB
via 4eee09a2c17 dsdb: disable ORDERED_INTEGER with MDB pack format v1
via 6b4abb99521 ldb: pack_format_override option
via 68d99187cd5 downgradedatabase: blackbox: database repacked
via 8db1312b08e ldb: python test for repack
via 73763acf49c ldb: repack old format database if GUID indexing enabled
via d6ded22cb61 downgradedatabase: blackbox: check ordered integer removed
via 08b9d204b6e ldb: binding ordered indexes to GUID indexing
via 6005c8cbad7 ldb: only used a->syntax->index_format_fn if GUID indexing is enabled
via 74d15c9bf76 downgradedatabase: blackbox test
via 09f2a187b3d sambadowngradedatabase: Add "or later" to warning about using tools from Samba 4.8
via c0b679f6a3f sambaundoguididx: renamed to downgradedatabase
via 40ca8ed5a15 sambaundoguididx: fix for -s
via a3101b9704f ldb: Fix segfault parsing new pack formats
via 2de0aebed60 ldb: test for parse errors
from 73080f255e3 smbd: Adapt brl_pending_overlap to README.Coding
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 8831b06d3d4c1cb5b7732e9863228f8f3aea4e36
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Fri May 24 15:34:43 2019 +1200
ldb: Release ldb 2.0.3
* Default pack format changed to version 2 for GUID Indexed databases
(essentially Samba's sam.ldb). This means faster unpacking for records
with lots of attributes.
* Enforce link between all new database features, following toggling of
GUID indexing. If user toggles GUID indexing on/off, target pack format
is changed. Likewise the special ORDERED_INTEGER index format is used
(when GUID indexing on) or behaves like INTEGER (when GUID indexing off).
* Database repacked if packing format not as expected.
* In the case of MDB, since GUID indexing is mandatory, feature toggling
provided by pack_format_override LDB option.
* Check for errors from ldb_unpack_data() in ldb_tdb
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed May 29 05:56:59 UTC 2019 on sn-devel-184
commit 4a95410a1543f854743aac24c6c7dc5ffc423aae
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Fri May 24 14:54:03 2019 +1200
downgradedatabase: blackbox: MDB backend
When Samba uses TDB as a backend store, all new database features are
toggled on/off when GUID indexing is toggled. But, GUID indexing can't
be disabled on MDB, so the other features are toggled separately.
Consequently, the downgradedatabase script behaves differently depending
on the database backend. This patch adds testing for the MDB behaviour.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
commit 0942a65b63cc99f36d3eba99e9c9551e10c5782e
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Thu May 23 20:06:56 2019 +1200
downgradedatabase: adding special case for MDB
Though this script was initially written for undoing GUID indexing on
TDB databases, we're repurposing it to do a full downgrade of any
database. MDB databases can't be DN indexed, but they can have pack
format version 2 and ORDERED_INTEGER data types, which must be removed
during a downgrade.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
commit 4eee09a2c17d1276b1d0be9f26b23743eec485c2
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Thu May 23 20:13:05 2019 +1200
dsdb: disable ORDERED_INTEGER with MDB pack format v1
For TDB databases, the new ORDERED_INTEGER type is disabled along with
repacking at format version 1 if GUID indexing is disabled, so all the new
database features are toggled together. This scheme doesn't work with
MDB because GUID indexing is mandatory when using MDB. However, a
downgrade path is still required so in a previous commit we added
a pack_format_override option which allows a downgrade script to force
the database to use an earlier packing format. But, the new
ORDERED_INTEGER type would still be present in MDB databases so this
patch reads the pack_format_override opaque and converts ORDERED_INTEGER
types in @ATTRIBUTES to INTEGER and doesn't write any indexes of that
type to @INDEXLIST. The @INDEXLIST will be refreshed later, on the first
transaction.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
commit 6b4abb995215c732ff5c0bfaca2cecb7a374edff
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Thu May 23 19:49:39 2019 +1200
ldb: pack_format_override option
For TDB databases, toggling GUID indexing mode will also toggle
pack format version 2. This provides a convenient downgrade path for
Samba databases, but the process doesn't work for MDB databases because
GUID indexing cannot be disabled when the MDB backend is used. This patch
addresses that corner case by providing support for a pack_format_override
option which will force the database to use pack format version 2.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
commit 68d99187cd5f85baacbf4af262b26ae0b9682db5
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Thu May 23 14:42:35 2019 +1200
downgradedatabase: blackbox: database repacked
Now that the database is repacked when GUID indexing is toggled on TDB,
test that downgradedatabase repacks a TDB-backed database with V2 pack
format database with V1.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 8db1312b08ebaf8881cf633196f0f40b0badac9a
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Mon May 20 17:59:33 2019 +1200
ldb: python test for repack
Python test to confirm that after enabling or disabling GUID indexing,
the database is repacked on the next transaction with V1 if GUID
indexing was disabled, or V2 if it was enabled.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 73763acf49c24727e2deaaa061be8a288ee7b3d5
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Mon May 13 16:37:25 2019 +1200
ldb: repack old format database if GUID indexing enabled
VERY IMPORTANT PATCH
Now that we have a new packing format, we need to enable it by repacking
the database. We've decided to link all new database features together,
so once GUID indexing is enabled, the database will be repacked with
version 2 format. Repacking is done following the same iterate pattern as
reindexing.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit d6ded22cb617aeef75a415208b2ce56867b68047
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Thu May 23 14:35:22 2019 +1200
downgradedatabase: blackbox: check ordered integer removed
Test that when we undo GUID indexing on a TDB-backed database with
downgradedatabase, ORDERED_INTEGER is removed from @ATTRIBUTES
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 08b9d204b6eeda0aedea754f7e4b6f809d358cfe
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Wed May 22 14:07:19 2019 +1200
ldb: binding ordered indexes to GUID indexing
To reduce the number of potential combinations of database features in
ldb, we want to link all new database features since 4.7. GUID indexing,
ordered integers, and pack format changes will all upgrade together.
This patch makes ordered integers only function if GUID indexing is
enabled. If GUID indexing is disabled, ORDERED_INTEGER will not be
written to @ATTRIBUTES and a syntax's index_format_fn will never be
used.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 6005c8cbad77259fa3cb89ab21cb5f26b72413cd
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Tue May 28 13:00:43 2019 +1200
ldb: only used a->syntax->index_format_fn if GUID indexing is enabled
Only enable ORDERED_INTEGER and index_format_fn functionality if GUID
indexing is enabled. This is in line with the rest of ldb_kv which binds
the new ORDERED_INTEGER to GUID indexed databases, and allows a practical
way to create the old index format (by disabling the GUID index).
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
commit 74d15c9bf76f0a2fb5fa7b7b1d80971d10c4fe45
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Thu May 23 13:21:19 2019 +1200
downgradedatabase: blackbox test
This test confirms that running downgradedatabase causes all GUID keys to be
replaced with DN keys at the KV level
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 09f2a187b3d8c161e2c11588499b3256a9dbcc95
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed May 29 16:36:00 2019 +1200
sambadowngradedatabase: Add "or later" to warning about using tools from Samba 4.8
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit c0b679f6a3f21e262d03bf38ea63900d30c29bb5
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Fri May 24 14:37:50 2019 +1200
sambaundoguididx: renamed to downgradedatabase
In forthcoming commits we're going to repurpose this script to do an
entire downgrade of a database, disabling all new database features.
downgradedatabase is a more appropriate name.
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 40ca8ed5a152ae7c5ec039649c09a037a20a4143
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon May 20 16:29:10 2019 +1200
sambaundoguididx: fix for -s
Quick fix running this script with -s instead of -H. samdb_url() returns
a url with a protocol prefix, which causes issues further down in the
script.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit a3101b9704f554a350493553336cbbbd7d4ae02e
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed May 22 16:38:08 2019 +1200
ldb: Fix segfault parsing new pack formats
We need to check for the errors given by ldb_unpack() et al by preserving
the error code from kv_ctx->parser() called by tdb_parse_record() in
ltdb_parse_record().
Otherwise we will silently accept corrupt records and segfault later.
Likewise new pack formats will confuse the parser but not be
detected except by the incomplete struct ldb_message.
With this patch, the user will see a message like:
Invalid data for index DN=@BASEINFO
Failed to connect to 'st/ad_dc/private/sam.ldb' with backend 'tdb': Unable to load ltdb cache records for backend 'ldb_tdb backend'
Failed to connect to st/ad_dc/private/sam.ldb - Unable to load ltdb cache records for backend 'ldb_tdb backend'
This can be refined in the future by a specific check for
pack format versions in a higher caller, but this much is
needed regardless to detect corrupt records.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 2de0aebed60b8e83508f50e5391ede618ce0e595
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Tue May 28 17:22:10 2019 +1200
ldb: test for parse errors
Parse errors aren't passed up correctly by the tdb backend. This
patch modifies a test to expose the issue, next patch will fix it.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
-----------------------------------------------------------------------
Summary of changes:
lib/ldb/ABI/{ldb-2.0.2.sigs => ldb-2.0.3.sigs} | 0
...yldb-util-1.1.10.sigs => pyldb-util-2.0.3.sigs} | 0
lib/ldb/ldb_key_value/ldb_kv.c | 80 +++++++-
lib/ldb/ldb_key_value/ldb_kv.h | 9 +
lib/ldb/ldb_key_value/ldb_kv_cache.c | 52 +++---
lib/ldb/ldb_key_value/ldb_kv_index.c | 93 +++++++++-
lib/ldb/ldb_tdb/ldb_tdb.c | 8 +-
lib/ldb/pyldb.c | 2 +
lib/ldb/tests/guidindexpackv1.ldb | Bin 0 -> 1286144 bytes
lib/ldb/tests/ldb_kv_ops_test.c | 23 +++
lib/ldb/tests/python/repack.py | 204 +++++++++++++++++++++
lib/ldb/wscript | 6 +-
python/samba/tests/__init__.py | 9 +-
python/samba/tests/blackbox/downgradedatabase.py | 158 ++++++++++++++++
source4/dsdb/schema/schema_set.c | 97 +++++++++-
.../{sambaundoguididx => sambadowngradedatabase} | 37 +++-
source4/selftest/tests.py | 2 +
17 files changed, 728 insertions(+), 52 deletions(-)
copy lib/ldb/ABI/{ldb-2.0.2.sigs => ldb-2.0.3.sigs} (100%)
copy lib/ldb/ABI/{pyldb-util-1.1.10.sigs => pyldb-util-2.0.3.sigs} (100%)
create mode 100644 lib/ldb/tests/guidindexpackv1.ldb
create mode 100644 lib/ldb/tests/python/repack.py
create mode 100644 python/samba/tests/blackbox/downgradedatabase.py
rename source4/scripting/bin/{sambaundoguididx => sambadowngradedatabase} (62%)
Changeset truncated at 500 lines:
diff --git a/lib/ldb/ABI/ldb-2.0.2.sigs b/lib/ldb/ABI/ldb-2.0.3.sigs
similarity index 100%
copy from lib/ldb/ABI/ldb-2.0.2.sigs
copy to lib/ldb/ABI/ldb-2.0.3.sigs
diff --git a/lib/ldb/ABI/pyldb-util-1.1.10.sigs b/lib/ldb/ABI/pyldb-util-2.0.3.sigs
similarity index 100%
copy from lib/ldb/ABI/pyldb-util-1.1.10.sigs
copy to lib/ldb/ABI/pyldb-util-2.0.3.sigs
diff --git a/lib/ldb/ldb_key_value/ldb_kv.c b/lib/ldb/ldb_key_value/ldb_kv.c
index c8f7fd1396d..d9b7b0af46a 100644
--- a/lib/ldb/ldb_key_value/ldb_kv.c
+++ b/lib/ldb/ldb_key_value/ldb_kv.c
@@ -301,6 +301,34 @@ static int ldb_kv_check_special_dn(struct ldb_module *module,
return LDB_SUCCESS;
}
+/*
+ * Called after modifies and when starting a transaction. Checks target pack
+ * format version and current pack format version, which are set by cache_load,
+ * and repacks if necessary.
+ */
+static int ldb_kv_maybe_repack(struct ldb_kv_private *ldb_kv) {
+ /* Override option taken from ldb options */
+ if (ldb_kv->pack_format_override != 0) {
+ ldb_kv->target_pack_format_version =
+ ldb_kv->pack_format_override;
+ }
+
+ if (ldb_kv->pack_format_version !=
+ ldb_kv->target_pack_format_version) {
+ int r;
+ struct ldb_context *ldb = ldb_module_get_ctx(ldb_kv->module);
+ ldb_kv->pack_format_version =
+ ldb_kv->target_pack_format_version;
+ r = ldb_kv_repack(ldb_kv->module);
+ if (r != LDB_SUCCESS) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR,
+ "Database repack failed.");
+ }
+ return r;
+ }
+
+ return LDB_SUCCESS;
+}
/*
we've made a modification to a dn - possibly reindex and
@@ -1447,6 +1475,20 @@ static int ldb_kv_prepare_commit(struct ldb_module *module)
return ret;
}
+ /*
+ * If GUID indexing was toggled in this transaction, we repack at
+ * format version 2 if GUID indexing was enabled, or version 1 if
+ * it was disabled.
+ */
+ ret = ldb_kv_maybe_repack(ldb_kv);
+ if (ret != LDB_SUCCESS) {
+ ldb_kv_del_trans(module);
+ ldb_set_errstring(ldb_module_get_ctx(module),
+ "Failure during re-pack, so "
+ "transaction must be aborted.");
+ return ret;
+ }
+
if (ldb_kv->kv_ops->prepare_write(ldb_kv) != 0) {
ret = ldb_kv->kv_ops->error(ldb_kv);
ldb_debug_set(ldb_module_get_ctx(module),
@@ -1895,10 +1937,10 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
ldb_kv->sequence_number = 0;
- ldb_kv->pack_format_version = LDB_PACKING_FORMAT;
-
ldb_kv->pid = getpid();
+ ldb_kv->pack_format_override = 0;
+
ldb_kv->module = ldb_module_new(ldb, ldb, name, &ldb_kv_ops);
if (!ldb_kv->module) {
ldb_oom(ldb);
@@ -1935,6 +1977,40 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
}
}
+ /*
+ * Usually the presence of GUID indexing determines the pack format
+ * we use but in certain circumstances such as downgrading an
+ * MDB-backed database, we want to override the target pack format.
+ *
+ * We set/get opaques here because in the Samba partitions module,
+ * 'options' are not passed correctly so sub-databases can't see
+ * the options they need.
+ */
+ {
+ const char *pack_format_override =
+ ldb_options_find(ldb, options, "pack_format_override");
+ if (pack_format_override != NULL) {
+ int ret;
+ ldb_kv->pack_format_override =
+ strtoul(pack_format_override, NULL, 0);
+ ret = ldb_set_opaque(ldb,
+ "pack_format_override",
+ (void *)(intptr_t)ldb_kv->pack_format_override);
+ if (ret != LDB_SUCCESS) {
+ talloc_free(ldb_kv->module);
+ return ldb_module_operr(ldb_kv->module);
+ }
+ } else {
+ /*
+ * NULL -> 0 is fine, otherwise we get back
+ * the number we needed
+ */
+ ldb_kv->pack_format_override
+ = (intptr_t)ldb_get_opaque(ldb,
+ "pack_format_override");
+ }
+ }
+
/*
* Override full DB scans
*
diff --git a/lib/ldb/ldb_key_value/ldb_kv.h b/lib/ldb/ldb_key_value/ldb_kv.h
index 1186e86ec9f..8da970fd2f9 100644
--- a/lib/ldb/ldb_key_value/ldb_kv.h
+++ b/lib/ldb/ldb_key_value/ldb_kv.h
@@ -64,6 +64,8 @@ struct ldb_kv_private {
unsigned long long sequence_number;
uint32_t pack_format_version;
+ uint32_t target_pack_format_version;
+ uint32_t pack_format_override;
/* the low level tdb seqnum - used to avoid loading BASEINFO when
possible */
@@ -141,6 +143,12 @@ struct ldb_kv_reindex_context {
uint32_t count;
};
+struct ldb_kv_repack_context {
+ int error;
+ uint32_t count;
+ bool normal_record_seen;
+};
+
/* special record types */
#define LDB_KV_INDEX "@INDEX"
@@ -226,6 +234,7 @@ int ldb_kv_index_del_value(struct ldb_module *module,
struct ldb_message_element *el,
unsigned int v_idx);
int ldb_kv_reindex(struct ldb_module *module);
+int ldb_kv_repack(struct ldb_module *module);
int ldb_kv_index_transaction_start(
struct ldb_module *module,
size_t cache_size);
diff --git a/lib/ldb/ldb_key_value/ldb_kv_cache.c b/lib/ldb/ldb_key_value/ldb_kv_cache.c
index c5f661113fd..3d8a09123e0 100644
--- a/lib/ldb/ldb_key_value/ldb_kv_cache.c
+++ b/lib/ldb/ldb_key_value/ldb_kv_cache.c
@@ -418,7 +418,6 @@ int ldb_kv_cache_load(struct ldb_module *module)
const struct ldb_schema_attribute *a;
bool have_write_txn = false;
int r;
- uint32_t pack_format_version;
struct ldb_val key;
ldb = ldb_module_get_ctx(module);
@@ -453,29 +452,7 @@ int ldb_kv_cache_load(struct ldb_module *module)
/* Read packing format from first 4 bytes of @BASEINFO record */
r = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key,
get_pack_format_version,
- &pack_format_version);
-
- if (r != LDB_ERR_NO_SUCH_OBJECT) {
- if (r != LDB_SUCCESS) {
- goto failed_and_unlock;
- }
-
- /* Make sure the database has the right format */
- if (pack_format_version != ldb_kv->pack_format_version) {
- ldb_debug(ldb, LDB_DEBUG_ERROR,
- "Unexpected packing format. "
- "Expected: %#010x, Got: %#010x",
- pack_format_version,
- ldb_kv->pack_format_version);
- goto failed_and_unlock;
- }
- }
-
- /* Now fetch the whole @BASEINFO record */
- r = ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0);
- if (r != LDB_SUCCESS && r != LDB_ERR_NO_SUCH_OBJECT) {
- goto failed_and_unlock;
- }
+ &ldb_kv->pack_format_version);
/* possibly initialise the baseinfo */
if (r == LDB_ERR_NO_SUCH_OBJECT) {
@@ -492,15 +469,25 @@ int ldb_kv_cache_load(struct ldb_module *module)
have_write_txn = true;
+ /*
+ * We need to write but haven't figured out packing format yet.
+ * Just go with version 1 and we'll repack if we got it wrong.
+ */
+ ldb_kv->pack_format_version = LDB_PACKING_FORMAT;
+ ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT;
+
/* error handling for ltdb_baseinfo_init() is by
looking for the record again. */
ldb_kv_baseinfo_init(module);
- if (ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0) !=
- LDB_SUCCESS) {
- goto failed_and_unlock;
- }
+ } else if (r != LDB_SUCCESS) {
+ goto failed_and_unlock;
+ }
+ /* OK now we definitely have a @BASEINFO record so fetch it */
+ r = ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0);
+ if (r != LDB_SUCCESS) {
+ goto failed_and_unlock;
}
/* Ignore the result, and update the sequence number */
@@ -562,8 +549,15 @@ int ldb_kv_cache_load(struct ldb_module *module)
goto failed_and_unlock;
}
+ /*
+ * Initialise packing version and GUID index syntax, and force the
+ * two to travel together, ie a GUID indexed database must use V2
+ * packing format and a DN indexed database must use V1.
+ */
ldb_kv->GUID_index_syntax = NULL;
if (ldb_kv->cache->GUID_index_attribute != NULL) {
+ ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT_V2;
+
/*
* Now the attributes are loaded, set the guid_index_syntax.
* This can't fail, it will return a default at worst
@@ -571,6 +565,8 @@ int ldb_kv_cache_load(struct ldb_module *module)
a = ldb_schema_attribute_by_name(
ldb, ldb_kv->cache->GUID_index_attribute);
ldb_kv->GUID_index_syntax = a->syntax;
+ } else {
+ ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT;
}
done:
diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c
index f6b2620a875..5af248a1931 100644
--- a/lib/ldb/ldb_key_value/ldb_kv_index.c
+++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
@@ -924,7 +924,8 @@ static struct ldb_dn *ldb_kv_index_key(struct ldb_context *ldb,
v = *value;
} else {
ldb_attr_handler_t fn;
- if (a->syntax->index_format_fn) {
+ if (a->syntax->index_format_fn &&
+ ldb_kv->cache->GUID_index_attribute != NULL) {
fn = a->syntax->index_format_fn;
} else {
fn = a->syntax->canonicalise_fn;
@@ -3411,6 +3412,96 @@ static int re_index(struct ldb_kv_private *ldb_kv,
return 0;
}
+static int re_pack(struct ldb_kv_private *ldb_kv,
+ struct ldb_val key,
+ struct ldb_val val,
+ void *state)
+{
+ struct ldb_context *ldb;
+ struct ldb_message *msg;
+ struct ldb_module *module = ldb_kv->module;
+ struct ldb_kv_repack_context *ctx =
+ (struct ldb_kv_repack_context *)state;
+ int ret;
+
+ ldb = ldb_module_get_ctx(module);
+
+ msg = ldb_msg_new(module);
+ if (msg == NULL) {
+ return -1;
+ }
+
+ ret = ldb_unpack_data(ldb, &val, msg);
+ if (ret != 0) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack: unpack failed: %s\n",
+ ldb_dn_get_linearized(msg->dn));
+ ctx->error = ret;
+ talloc_free(msg);
+ return -1;
+ }
+
+ ret = ldb_kv_store(module, msg, TDB_MODIFY);
+ if (ret != LDB_SUCCESS) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack: store failed: %s\n",
+ ldb_dn_get_linearized(msg->dn));
+ ctx->error = ret;
+ talloc_free(msg);
+ return -1;
+ }
+
+ /*
+ * Warn the user that we're repacking the first time we see a normal
+ * record. This means we never warn if we're repacking a database with
+ * only @ records. This is because during database initialisation,
+ * we might repack as initial settings are written out, and we don't
+ * want to spam the log.
+ */
+ if ((!ctx->normal_record_seen) && (!ldb_dn_is_special(msg->dn))) {
+ ldb_debug(ldb, LDB_DEBUG_WARNING,
+ "Repacking database with format %#010x",
+ ldb_kv->pack_format_version);
+ ctx->normal_record_seen = true;
+ }
+
+ ctx->count++;
+ if (ctx->count % 10000 == 0) {
+ ldb_debug(ldb, LDB_DEBUG_WARNING,
+ "Repack: re-packed %u records so far",
+ ctx->count);
+ }
+
+ return 0;
+}
+
+int ldb_kv_repack(struct ldb_module *module)
+{
+ struct ldb_kv_private *ldb_kv = talloc_get_type(
+ ldb_module_get_private(module), struct ldb_kv_private);
+ struct ldb_context *ldb = ldb_module_get_ctx(module);
+ struct ldb_kv_repack_context ctx;
+ int ret;
+
+ ctx.count = 0;
+ ctx.error = LDB_SUCCESS;
+ ctx.normal_record_seen = false;
+
+ /* Iterate all database records and repack them in the new format */
+ ret = ldb_kv->kv_ops->iterate(ldb_kv, re_pack, &ctx);
+ if (ret < 0) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack traverse failed: %s",
+ ldb_errstring(ldb));
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+
+ if (ctx.error != LDB_SUCCESS) {
+ ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack failed: %s",
+ ldb_errstring(ldb));
+ return ctx.error;
+ }
+
+ return LDB_SUCCESS;
+}
+
/*
force a complete reindex of the database
*/
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index 2f0468fd40a..6e2cb633a4d 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -242,6 +242,7 @@ struct kv_ctx {
int (*parser)(struct ldb_val key,
struct ldb_val data,
void *private_data);
+ int parser_ret;
};
static int ltdb_traverse_fn_wrapper(struct tdb_context *tdb,
@@ -350,7 +351,8 @@ static int ltdb_parse_record_wrapper(TDB_DATA tdb_key,
.data = tdb_data.dptr,
};
- return kv_ctx->parser(key, data, kv_ctx->ctx);
+ kv_ctx->parser_ret = kv_ctx->parser(key, data, kv_ctx->ctx);
+ return kv_ctx->parser_ret;
}
static int ltdb_parse_record(struct ldb_kv_private *ldb_kv,
@@ -374,7 +376,9 @@ static int ltdb_parse_record(struct ldb_kv_private *ldb_kv,
ret = tdb_parse_record(
ldb_kv->tdb, key, ltdb_parse_record_wrapper, &kv_ctx);
- if (ret == 0) {
+ if (kv_ctx.parser_ret != LDB_SUCCESS) {
+ return kv_ctx.parser_ret;
+ } else if (ret == 0) {
return LDB_SUCCESS;
}
return ltdb_err_map(tdb_error(ldb_kv->tdb));
diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c
index cf7779128d4..940e43e3192 100644
--- a/lib/ldb/pyldb.c
+++ b/lib/ldb/pyldb.c
@@ -4430,6 +4430,8 @@ static PyObject* module_init(void)
ADD_LDB_INT(FLG_ENABLE_TRACING);
ADD_LDB_INT(FLG_DONT_CREATE_DB);
+ ADD_LDB_INT(PACKING_FORMAT);
+ ADD_LDB_INT(PACKING_FORMAT_V2);
/* Historical misspelling */
PyModule_AddIntConstant(m, "ERR_ALIAS_DEREFERINCING_PROBLEM", LDB_ERR_ALIAS_DEREFERENCING_PROBLEM);
diff --git a/lib/ldb/tests/guidindexpackv1.ldb b/lib/ldb/tests/guidindexpackv1.ldb
new file mode 100644
index 00000000000..4c79dfedddd
Binary files /dev/null and b/lib/ldb/tests/guidindexpackv1.ldb differ
diff --git a/lib/ldb/tests/ldb_kv_ops_test.c b/lib/ldb/tests/ldb_kv_ops_test.c
index c16297e136b..ab9d75bc69c 100644
--- a/lib/ldb/tests/ldb_kv_ops_test.c
+++ b/lib/ldb/tests/ldb_kv_ops_test.c
@@ -206,6 +206,17 @@ static int parse(struct ldb_val key,
return LDB_SUCCESS;
}
+/*
+ * Parse function that just returns the int we pass it.
+ */
+static int parse_return(struct ldb_val key,
+ struct ldb_val data,
+ void *private_data)
+{
+ int *rcode = private_data;
+ return *rcode;
+}
+
/*
* Test that data can be written to the kv store and be read back.
*/
@@ -228,6 +239,7 @@ static void test_add_get(void **state)
};
struct ldb_val read;
+ int rcode;
int flags = 0;
TALLOC_CTX *tmp_ctx;
@@ -265,6 +277,17 @@ static void test_add_get(void **state)
assert_int_equal(sizeof(value), read.length);
assert_memory_equal(value, read.data, sizeof(value));
+ /*
+ * Now check that the error code we return in the
+ * parse function is returned by fetch_and_parse.
+ */
+ for (rcode=0; rcode<50; rcode++) {
+ ret = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key,
+ parse_return,
+ &rcode);
+ assert_int_equal(ret, rcode);
+ }
+
ret = ldb_kv->kv_ops->unlock_read(test_ctx->ldb->modules);
assert_int_equal(ret, 0);
talloc_free(tmp_ctx);
diff --git a/lib/ldb/tests/python/repack.py b/lib/ldb/tests/python/repack.py
new file mode 100644
index 00000000000..0844cd24e58
--- /dev/null
+++ b/lib/ldb/tests/python/repack.py
@@ -0,0 +1,204 @@
+import os
+from unittest import TestCase
+import shutil
+from subprocess import check_output
+import ldb
+
+TDB_PREFIX = "tdb://"
+MDB_PREFIX = "mdb://"
+
+def tempdir():
+ import tempfile
+ try:
+ dir_prefix = os.path.join(os.environ["SELFTEST_PREFIX"], "tmp")
+ except KeyError:
+ dir_prefix = None
+ return tempfile.mkdtemp(dir=dir_prefix)
+
+
+# Check enabling and disabling GUID indexing works and that the database is
+# repacked at version 2 if GUID indexing is enabled, or version 1 if disabled.
+class GUIDIndexAndPackFormatTests(TestCase):
+ prefix = TDB_PREFIX
+
+ def setup_newdb(self):
+ self.testdir = tempdir()
+ self.filename = os.path.join(self.testdir,
+ "guidpackformattest.ldb")
+ url = self.prefix + self.filename
+ self.l = ldb.Ldb(url, options=["modules:"])
+
+ self.num_recs_added = 0
--
Samba Shared Repository
More information about the samba-cvs
mailing list