sambadowngradedatabase (Re: [SCM] Samba Shared Repository - branch master updated)
Stefan Metzmacher
metze at samba.org
Wed May 29 11:06:14 UTC 2019
Hi Andrew,
can you please prepare backports of the sambadowngradedatabase changes?
These are needed in the old branches.
Thanks!
metze
Am 29.05.19 um 07:58 schrieb Andrew Bartlett:
> The branch, master has been updated
> via 8831b06d3d4 ldb: Release ldb 2.0.3
> via 4a95410a154 downgradedatabase: blackbox: MDB backend
> via 0942a65b63c downgradedatabase: adding special case for MDB
> via 4eee09a2c17 dsdb: disable ORDERED_INTEGER with MDB pack format v1
> via 6b4abb99521 ldb: pack_format_override option
> via 68d99187cd5 downgradedatabase: blackbox: database repacked
> via 8db1312b08e ldb: python test for repack
> via 73763acf49c ldb: repack old format database if GUID indexing enabled
> via d6ded22cb61 downgradedatabase: blackbox: check ordered integer removed
> via 08b9d204b6e ldb: binding ordered indexes to GUID indexing
> via 6005c8cbad7 ldb: only used a->syntax->index_format_fn if GUID indexing is enabled
> via 74d15c9bf76 downgradedatabase: blackbox test
> via 09f2a187b3d sambadowngradedatabase: Add "or later" to warning about using tools from Samba 4.8
> via c0b679f6a3f sambaundoguididx: renamed to downgradedatabase
> via 40ca8ed5a15 sambaundoguididx: fix for -s
> via a3101b9704f ldb: Fix segfault parsing new pack formats
> via 2de0aebed60 ldb: test for parse errors
> from 73080f255e3 smbd: Adapt brl_pending_overlap to README.Coding
> https://git.samba.org/?p=samba.git;a=shortlog;h=master
>
>
> - Log -----------------------------------------------------------------
> commit 8831b06d3d4c1cb5b7732e9863228f8f3aea4e36
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Fri May 24 15:34:43 2019 +1200
>
> ldb: Release ldb 2.0.3
>
> * Default pack format changed to version 2 for GUID Indexed databases
> (essentially Samba's sam.ldb). This means faster unpacking for records
> with lots of attributes.
>
> * Enforce link between all new database features, following toggling of
> GUID indexing. If user toggles GUID indexing on/off, target pack format
> is changed. Likewise the special ORDERED_INTEGER index format is used
> (when GUID indexing on) or behaves like INTEGER (when GUID indexing off).
>
> * Database repacked if packing format not as expected.
>
> * In the case of MDB, since GUID indexing is mandatory, feature toggling
> provided by pack_format_override LDB option.
>
> * Check for errors from ldb_unpack_data() in ldb_tdb
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>
> Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
> Autobuild-Date(master): Wed May 29 05:56:59 UTC 2019 on sn-devel-184
>
> commit 4a95410a1543f854743aac24c6c7dc5ffc423aae
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Fri May 24 14:54:03 2019 +1200
>
> downgradedatabase: blackbox: MDB backend
>
> When Samba uses TDB as a backend store, all new database features are
> toggled on/off when GUID indexing is toggled. But, GUID indexing can't
> be disabled on MDB, so the other features are toggled separately.
> Consequently, the downgradedatabase script behaves differently depending
> on the database backend. This patch adds testing for the MDB behaviour.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>
> commit 0942a65b63cc99f36d3eba99e9c9551e10c5782e
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Thu May 23 20:06:56 2019 +1200
>
> downgradedatabase: adding special case for MDB
>
> Though this script was initially written for undoing GUID indexing on
> TDB databases, we're repurposing it to do a full downgrade of any
> database. MDB databases can't be DN indexed, but they can have pack
> format version 2 and ORDERED_INTEGER data types, which must be removed
> during a downgrade.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>
> commit 4eee09a2c17d1276b1d0be9f26b23743eec485c2
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Thu May 23 20:13:05 2019 +1200
>
> dsdb: disable ORDERED_INTEGER with MDB pack format v1
>
> For TDB databases, the new ORDERED_INTEGER type is disabled along with
> repacking at format version 1 if GUID indexing is disabled, so all the new
> database features are toggled together. This scheme doesn't work with
> MDB because GUID indexing is mandatory when using MDB. However, a
> downgrade path is still required so in a previous commit we added
> a pack_format_override option which allows a downgrade script to force
> the database to use an earlier packing format. But, the new
> ORDERED_INTEGER type would still be present in MDB databases so this
> patch reads the pack_format_override opaque and converts ORDERED_INTEGER
> types in @ATTRIBUTES to INTEGER and doesn't write any indexes of that
> type to @INDEXLIST. The @INDEXLIST will be refreshed later, on the first
> transaction.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>
> commit 6b4abb995215c732ff5c0bfaca2cecb7a374edff
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Thu May 23 19:49:39 2019 +1200
>
> ldb: pack_format_override option
>
> For TDB databases, toggling GUID indexing mode will also toggle
> pack format version 2. This provides a convenient downgrade path for
> Samba databases, but the process doesn't work for MDB databases because
> GUID indexing cannot be disabled when the MDB backend is used. This patch
> addresses that corner case by providing support for a pack_format_override
> option which will force the database to use pack format version 2.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>
> commit 68d99187cd5f85baacbf4af262b26ae0b9682db5
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Thu May 23 14:42:35 2019 +1200
>
> downgradedatabase: blackbox: database repacked
>
> Now that the database is repacked when GUID indexing is toggled on TDB,
> test that downgradedatabase repacks a TDB-backed database with V2 pack
> format database with V1.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 8db1312b08ebaf8881cf633196f0f40b0badac9a
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Mon May 20 17:59:33 2019 +1200
>
> ldb: python test for repack
>
> Python test to confirm that after enabling or disabling GUID indexing,
> the database is repacked on the next transaction with V1 if GUID
> indexing was disabled, or V2 if it was enabled.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 73763acf49c24727e2deaaa061be8a288ee7b3d5
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Mon May 13 16:37:25 2019 +1200
>
> ldb: repack old format database if GUID indexing enabled
>
> VERY IMPORTANT PATCH
> Now that we have a new packing format, we need to enable it by repacking
> the database. We've decided to link all new database features together,
> so once GUID indexing is enabled, the database will be repacked with
> version 2 format. Repacking is done following the same iterate pattern as
> reindexing.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit d6ded22cb617aeef75a415208b2ce56867b68047
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Thu May 23 14:35:22 2019 +1200
>
> downgradedatabase: blackbox: check ordered integer removed
>
> Test that when we undo GUID indexing on a TDB-backed database with
> downgradedatabase, ORDERED_INTEGER is removed from @ATTRIBUTES
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 08b9d204b6eeda0aedea754f7e4b6f809d358cfe
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Wed May 22 14:07:19 2019 +1200
>
> ldb: binding ordered indexes to GUID indexing
>
> To reduce the number of potential combinations of database features in
> ldb, we want to link all new database features since 4.7. GUID indexing,
> ordered integers, and pack format changes will all upgrade together.
> This patch makes ordered integers only function if GUID indexing is
> enabled. If GUID indexing is disabled, ORDERED_INTEGER will not be
> written to @ATTRIBUTES and a syntax's index_format_fn will never be
> used.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 6005c8cbad77259fa3cb89ab21cb5f26b72413cd
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Tue May 28 13:00:43 2019 +1200
>
> ldb: only used a->syntax->index_format_fn if GUID indexing is enabled
>
> Only enable ORDERED_INTEGER and index_format_fn functionality if GUID
> indexing is enabled. This is in line with the rest of ldb_kv which binds
> the new ORDERED_INTEGER to GUID indexed databases, and allows a practical
> way to create the old index format (by disabling the GUID index).
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>
> commit 74d15c9bf76f0a2fb5fa7b7b1d80971d10c4fe45
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Thu May 23 13:21:19 2019 +1200
>
> downgradedatabase: blackbox test
>
> This test confirms that running downgradedatabase causes all GUID keys to be
> replaced with DN keys at the KV level
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 09f2a187b3d8c161e2c11588499b3256a9dbcc95
> Author: Andrew Bartlett <abartlet at samba.org>
> Date: Wed May 29 16:36:00 2019 +1200
>
> sambadowngradedatabase: Add "or later" to warning about using tools from Samba 4.8
>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit c0b679f6a3f21e262d03bf38ea63900d30c29bb5
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Fri May 24 14:37:50 2019 +1200
>
> sambaundoguididx: renamed to downgradedatabase
>
> In forthcoming commits we're going to repurpose this script to do an
> entire downgrade of a database, disabling all new database features.
> downgradedatabase is a more appropriate name.
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 40ca8ed5a152ae7c5ec039649c09a037a20a4143
> Author: Andrew Bartlett <abartlet at samba.org>
> Date: Mon May 20 16:29:10 2019 +1200
>
> sambaundoguididx: fix for -s
>
> Quick fix running this script with -s instead of -H. samdb_url() returns
> a url with a protocol prefix, which causes issues further down in the
> script.
>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit a3101b9704f554a350493553336cbbbd7d4ae02e
> Author: Andrew Bartlett <abartlet at samba.org>
> Date: Wed May 22 16:38:08 2019 +1200
>
> ldb: Fix segfault parsing new pack formats
>
> We need to check for the errors given by ldb_unpack() et al by preserving
> the error code from kv_ctx->parser() called by tdb_parse_record() in
> ltdb_parse_record().
>
> Otherwise we will silently accept corrupt records and segfault later.
>
> Likewise new pack formats will confuse the parser but not be
> detected except by the incomplete struct ldb_message.
>
> With this patch, the user will see a message like:
>
> Invalid data for index DN=@BASEINFO
>
> Failed to connect to 'st/ad_dc/private/sam.ldb' with backend 'tdb': Unable to load ltdb cache records for backend 'ldb_tdb backend'
> Failed to connect to st/ad_dc/private/sam.ldb - Unable to load ltdb cache records for backend 'ldb_tdb backend'
>
> This can be refined in the future by a specific check for
> pack format versions in a higher caller, but this much is
> needed regardless to detect corrupt records.
>
> BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
>
> Signed-off-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> commit 2de0aebed60b8e83508f50e5391ede618ce0e595
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date: Tue May 28 17:22:10 2019 +1200
>
> ldb: test for parse errors
>
> Parse errors aren't passed up correctly by the tdb backend. This
> patch modifies a test to expose the issue, next patch will fix it.
>
> BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
>
> Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>
> -----------------------------------------------------------------------
>
> Summary of changes:
> lib/ldb/ABI/{ldb-2.0.2.sigs => ldb-2.0.3.sigs} | 0
> ...yldb-util-1.1.10.sigs => pyldb-util-2.0.3.sigs} | 0
> lib/ldb/ldb_key_value/ldb_kv.c | 80 +++++++-
> lib/ldb/ldb_key_value/ldb_kv.h | 9 +
> lib/ldb/ldb_key_value/ldb_kv_cache.c | 52 +++---
> lib/ldb/ldb_key_value/ldb_kv_index.c | 93 +++++++++-
> lib/ldb/ldb_tdb/ldb_tdb.c | 8 +-
> lib/ldb/pyldb.c | 2 +
> lib/ldb/tests/guidindexpackv1.ldb | Bin 0 -> 1286144 bytes
> lib/ldb/tests/ldb_kv_ops_test.c | 23 +++
> lib/ldb/tests/python/repack.py | 204 +++++++++++++++++++++
> lib/ldb/wscript | 6 +-
> python/samba/tests/__init__.py | 9 +-
> python/samba/tests/blackbox/downgradedatabase.py | 158 ++++++++++++++++
> source4/dsdb/schema/schema_set.c | 97 +++++++++-
> .../{sambaundoguididx => sambadowngradedatabase} | 37 +++-
> source4/selftest/tests.py | 2 +
> 17 files changed, 728 insertions(+), 52 deletions(-)
> copy lib/ldb/ABI/{ldb-2.0.2.sigs => ldb-2.0.3.sigs} (100%)
> copy lib/ldb/ABI/{pyldb-util-1.1.10.sigs => pyldb-util-2.0.3.sigs} (100%)
> create mode 100644 lib/ldb/tests/guidindexpackv1.ldb
> create mode 100644 lib/ldb/tests/python/repack.py
> create mode 100644 python/samba/tests/blackbox/downgradedatabase.py
> rename source4/scripting/bin/{sambaundoguididx => sambadowngradedatabase} (62%)
>
>
> Changeset truncated at 500 lines:
>
> diff --git a/lib/ldb/ABI/ldb-2.0.2.sigs b/lib/ldb/ABI/ldb-2.0.3.sigs
> similarity index 100%
> copy from lib/ldb/ABI/ldb-2.0.2.sigs
> copy to lib/ldb/ABI/ldb-2.0.3.sigs
> diff --git a/lib/ldb/ABI/pyldb-util-1.1.10.sigs b/lib/ldb/ABI/pyldb-util-2.0.3.sigs
> similarity index 100%
> copy from lib/ldb/ABI/pyldb-util-1.1.10.sigs
> copy to lib/ldb/ABI/pyldb-util-2.0.3.sigs
> diff --git a/lib/ldb/ldb_key_value/ldb_kv.c b/lib/ldb/ldb_key_value/ldb_kv.c
> index c8f7fd1396d..d9b7b0af46a 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv.c
> @@ -301,6 +301,34 @@ static int ldb_kv_check_special_dn(struct ldb_module *module,
> return LDB_SUCCESS;
> }
>
> +/*
> + * Called after modifies and when starting a transaction. Checks target pack
> + * format version and current pack format version, which are set by cache_load,
> + * and repacks if necessary.
> + */
> +static int ldb_kv_maybe_repack(struct ldb_kv_private *ldb_kv) {
> + /* Override option taken from ldb options */
> + if (ldb_kv->pack_format_override != 0) {
> + ldb_kv->target_pack_format_version =
> + ldb_kv->pack_format_override;
> + }
> +
> + if (ldb_kv->pack_format_version !=
> + ldb_kv->target_pack_format_version) {
> + int r;
> + struct ldb_context *ldb = ldb_module_get_ctx(ldb_kv->module);
> + ldb_kv->pack_format_version =
> + ldb_kv->target_pack_format_version;
> + r = ldb_kv_repack(ldb_kv->module);
> + if (r != LDB_SUCCESS) {
> + ldb_debug(ldb, LDB_DEBUG_ERROR,
> + "Database repack failed.");
> + }
> + return r;
> + }
> +
> + return LDB_SUCCESS;
> +}
>
> /*
> we've made a modification to a dn - possibly reindex and
> @@ -1447,6 +1475,20 @@ static int ldb_kv_prepare_commit(struct ldb_module *module)
> return ret;
> }
>
> + /*
> + * If GUID indexing was toggled in this transaction, we repack at
> + * format version 2 if GUID indexing was enabled, or version 1 if
> + * it was disabled.
> + */
> + ret = ldb_kv_maybe_repack(ldb_kv);
> + if (ret != LDB_SUCCESS) {
> + ldb_kv_del_trans(module);
> + ldb_set_errstring(ldb_module_get_ctx(module),
> + "Failure during re-pack, so "
> + "transaction must be aborted.");
> + return ret;
> + }
> +
> if (ldb_kv->kv_ops->prepare_write(ldb_kv) != 0) {
> ret = ldb_kv->kv_ops->error(ldb_kv);
> ldb_debug_set(ldb_module_get_ctx(module),
> @@ -1895,10 +1937,10 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
>
> ldb_kv->sequence_number = 0;
>
> - ldb_kv->pack_format_version = LDB_PACKING_FORMAT;
> -
> ldb_kv->pid = getpid();
>
> + ldb_kv->pack_format_override = 0;
> +
> ldb_kv->module = ldb_module_new(ldb, ldb, name, &ldb_kv_ops);
> if (!ldb_kv->module) {
> ldb_oom(ldb);
> @@ -1935,6 +1977,40 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
> }
> }
>
> + /*
> + * Usually the presence of GUID indexing determines the pack format
> + * we use but in certain circumstances such as downgrading an
> + * MDB-backed database, we want to override the target pack format.
> + *
> + * We set/get opaques here because in the Samba partitions module,
> + * 'options' are not passed correctly so sub-databases can't see
> + * the options they need.
> + */
> + {
> + const char *pack_format_override =
> + ldb_options_find(ldb, options, "pack_format_override");
> + if (pack_format_override != NULL) {
> + int ret;
> + ldb_kv->pack_format_override =
> + strtoul(pack_format_override, NULL, 0);
> + ret = ldb_set_opaque(ldb,
> + "pack_format_override",
> + (void *)(intptr_t)ldb_kv->pack_format_override);
> + if (ret != LDB_SUCCESS) {
> + talloc_free(ldb_kv->module);
> + return ldb_module_operr(ldb_kv->module);
> + }
> + } else {
> + /*
> + * NULL -> 0 is fine, otherwise we get back
> + * the number we needed
> + */
> + ldb_kv->pack_format_override
> + = (intptr_t)ldb_get_opaque(ldb,
> + "pack_format_override");
> + }
> + }
> +
> /*
> * Override full DB scans
> *
> diff --git a/lib/ldb/ldb_key_value/ldb_kv.h b/lib/ldb/ldb_key_value/ldb_kv.h
> index 1186e86ec9f..8da970fd2f9 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv.h
> +++ b/lib/ldb/ldb_key_value/ldb_kv.h
> @@ -64,6 +64,8 @@ struct ldb_kv_private {
>
> unsigned long long sequence_number;
> uint32_t pack_format_version;
> + uint32_t target_pack_format_version;
> + uint32_t pack_format_override;
>
> /* the low level tdb seqnum - used to avoid loading BASEINFO when
> possible */
> @@ -141,6 +143,12 @@ struct ldb_kv_reindex_context {
> uint32_t count;
> };
>
> +struct ldb_kv_repack_context {
> + int error;
> + uint32_t count;
> + bool normal_record_seen;
> +};
> +
>
> /* special record types */
> #define LDB_KV_INDEX "@INDEX"
> @@ -226,6 +234,7 @@ int ldb_kv_index_del_value(struct ldb_module *module,
> struct ldb_message_element *el,
> unsigned int v_idx);
> int ldb_kv_reindex(struct ldb_module *module);
> +int ldb_kv_repack(struct ldb_module *module);
> int ldb_kv_index_transaction_start(
> struct ldb_module *module,
> size_t cache_size);
> diff --git a/lib/ldb/ldb_key_value/ldb_kv_cache.c b/lib/ldb/ldb_key_value/ldb_kv_cache.c
> index c5f661113fd..3d8a09123e0 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv_cache.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv_cache.c
> @@ -418,7 +418,6 @@ int ldb_kv_cache_load(struct ldb_module *module)
> const struct ldb_schema_attribute *a;
> bool have_write_txn = false;
> int r;
> - uint32_t pack_format_version;
> struct ldb_val key;
>
> ldb = ldb_module_get_ctx(module);
> @@ -453,29 +452,7 @@ int ldb_kv_cache_load(struct ldb_module *module)
> /* Read packing format from first 4 bytes of @BASEINFO record */
> r = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key,
> get_pack_format_version,
> - &pack_format_version);
> -
> - if (r != LDB_ERR_NO_SUCH_OBJECT) {
> - if (r != LDB_SUCCESS) {
> - goto failed_and_unlock;
> - }
> -
> - /* Make sure the database has the right format */
> - if (pack_format_version != ldb_kv->pack_format_version) {
> - ldb_debug(ldb, LDB_DEBUG_ERROR,
> - "Unexpected packing format. "
> - "Expected: %#010x, Got: %#010x",
> - pack_format_version,
> - ldb_kv->pack_format_version);
> - goto failed_and_unlock;
> - }
> - }
> -
> - /* Now fetch the whole @BASEINFO record */
> - r = ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0);
> - if (r != LDB_SUCCESS && r != LDB_ERR_NO_SUCH_OBJECT) {
> - goto failed_and_unlock;
> - }
> + &ldb_kv->pack_format_version);
>
> /* possibly initialise the baseinfo */
> if (r == LDB_ERR_NO_SUCH_OBJECT) {
> @@ -492,15 +469,25 @@ int ldb_kv_cache_load(struct ldb_module *module)
>
> have_write_txn = true;
>
> + /*
> + * We need to write but haven't figured out packing format yet.
> + * Just go with version 1 and we'll repack if we got it wrong.
> + */
> + ldb_kv->pack_format_version = LDB_PACKING_FORMAT;
> + ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT;
> +
> /* error handling for ltdb_baseinfo_init() is by
> looking for the record again. */
> ldb_kv_baseinfo_init(module);
>
> - if (ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0) !=
> - LDB_SUCCESS) {
> - goto failed_and_unlock;
> - }
> + } else if (r != LDB_SUCCESS) {
> + goto failed_and_unlock;
> + }
>
> + /* OK now we definitely have a @BASEINFO record so fetch it */
> + r = ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0);
> + if (r != LDB_SUCCESS) {
> + goto failed_and_unlock;
> }
>
> /* Ignore the result, and update the sequence number */
> @@ -562,8 +549,15 @@ int ldb_kv_cache_load(struct ldb_module *module)
> goto failed_and_unlock;
> }
>
> + /*
> + * Initialise packing version and GUID index syntax, and force the
> + * two to travel together, ie a GUID indexed database must use V2
> + * packing format and a DN indexed database must use V1.
> + */
> ldb_kv->GUID_index_syntax = NULL;
> if (ldb_kv->cache->GUID_index_attribute != NULL) {
> + ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT_V2;
> +
> /*
> * Now the attributes are loaded, set the guid_index_syntax.
> * This can't fail, it will return a default at worst
> @@ -571,6 +565,8 @@ int ldb_kv_cache_load(struct ldb_module *module)
> a = ldb_schema_attribute_by_name(
> ldb, ldb_kv->cache->GUID_index_attribute);
> ldb_kv->GUID_index_syntax = a->syntax;
> + } else {
> + ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT;
> }
>
> done:
> diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c
> index f6b2620a875..5af248a1931 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv_index.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
> @@ -924,7 +924,8 @@ static struct ldb_dn *ldb_kv_index_key(struct ldb_context *ldb,
> v = *value;
> } else {
> ldb_attr_handler_t fn;
> - if (a->syntax->index_format_fn) {
> + if (a->syntax->index_format_fn &&
> + ldb_kv->cache->GUID_index_attribute != NULL) {
> fn = a->syntax->index_format_fn;
> } else {
> fn = a->syntax->canonicalise_fn;
> @@ -3411,6 +3412,96 @@ static int re_index(struct ldb_kv_private *ldb_kv,
> return 0;
> }
>
> +static int re_pack(struct ldb_kv_private *ldb_kv,
> + struct ldb_val key,
> + struct ldb_val val,
> + void *state)
> +{
> + struct ldb_context *ldb;
> + struct ldb_message *msg;
> + struct ldb_module *module = ldb_kv->module;
> + struct ldb_kv_repack_context *ctx =
> + (struct ldb_kv_repack_context *)state;
> + int ret;
> +
> + ldb = ldb_module_get_ctx(module);
> +
> + msg = ldb_msg_new(module);
> + if (msg == NULL) {
> + return -1;
> + }
> +
> + ret = ldb_unpack_data(ldb, &val, msg);
> + if (ret != 0) {
> + ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack: unpack failed: %s\n",
> + ldb_dn_get_linearized(msg->dn));
> + ctx->error = ret;
> + talloc_free(msg);
> + return -1;
> + }
> +
> + ret = ldb_kv_store(module, msg, TDB_MODIFY);
> + if (ret != LDB_SUCCESS) {
> + ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack: store failed: %s\n",
> + ldb_dn_get_linearized(msg->dn));
> + ctx->error = ret;
> + talloc_free(msg);
> + return -1;
> + }
> +
> + /*
> + * Warn the user that we're repacking the first time we see a normal
> + * record. This means we never warn if we're repacking a database with
> + * only @ records. This is because during database initialisation,
> + * we might repack as initial settings are written out, and we don't
> + * want to spam the log.
> + */
> + if ((!ctx->normal_record_seen) && (!ldb_dn_is_special(msg->dn))) {
> + ldb_debug(ldb, LDB_DEBUG_WARNING,
> + "Repacking database with format %#010x",
> + ldb_kv->pack_format_version);
> + ctx->normal_record_seen = true;
> + }
> +
> + ctx->count++;
> + if (ctx->count % 10000 == 0) {
> + ldb_debug(ldb, LDB_DEBUG_WARNING,
> + "Repack: re-packed %u records so far",
> + ctx->count);
> + }
> +
> + return 0;
> +}
> +
> +int ldb_kv_repack(struct ldb_module *module)
> +{
> + struct ldb_kv_private *ldb_kv = talloc_get_type(
> + ldb_module_get_private(module), struct ldb_kv_private);
> + struct ldb_context *ldb = ldb_module_get_ctx(module);
> + struct ldb_kv_repack_context ctx;
> + int ret;
> +
> + ctx.count = 0;
> + ctx.error = LDB_SUCCESS;
> + ctx.normal_record_seen = false;
> +
> + /* Iterate all database records and repack them in the new format */
> + ret = ldb_kv->kv_ops->iterate(ldb_kv, re_pack, &ctx);
> + if (ret < 0) {
> + ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack traverse failed: %s",
> + ldb_errstring(ldb));
> + return LDB_ERR_OPERATIONS_ERROR;
> + }
> +
> + if (ctx.error != LDB_SUCCESS) {
> + ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack failed: %s",
> + ldb_errstring(ldb));
> + return ctx.error;
> + }
> +
> + return LDB_SUCCESS;
> +}
> +
> /*
> force a complete reindex of the database
> */
> diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
> index 2f0468fd40a..6e2cb633a4d 100644
> --- a/lib/ldb/ldb_tdb/ldb_tdb.c
> +++ b/lib/ldb/ldb_tdb/ldb_tdb.c
> @@ -242,6 +242,7 @@ struct kv_ctx {
> int (*parser)(struct ldb_val key,
> struct ldb_val data,
> void *private_data);
> + int parser_ret;
> };
>
> static int ltdb_traverse_fn_wrapper(struct tdb_context *tdb,
> @@ -350,7 +351,8 @@ static int ltdb_parse_record_wrapper(TDB_DATA tdb_key,
> .data = tdb_data.dptr,
> };
>
> - return kv_ctx->parser(key, data, kv_ctx->ctx);
> + kv_ctx->parser_ret = kv_ctx->parser(key, data, kv_ctx->ctx);
> + return kv_ctx->parser_ret;
> }
>
> static int ltdb_parse_record(struct ldb_kv_private *ldb_kv,
> @@ -374,7 +376,9 @@ static int ltdb_parse_record(struct ldb_kv_private *ldb_kv,
>
> ret = tdb_parse_record(
> ldb_kv->tdb, key, ltdb_parse_record_wrapper, &kv_ctx);
> - if (ret == 0) {
> + if (kv_ctx.parser_ret != LDB_SUCCESS) {
> + return kv_ctx.parser_ret;
> + } else if (ret == 0) {
> return LDB_SUCCESS;
> }
> return ltdb_err_map(tdb_error(ldb_kv->tdb));
> diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c
> index cf7779128d4..940e43e3192 100644
> --- a/lib/ldb/pyldb.c
> +++ b/lib/ldb/pyldb.c
> @@ -4430,6 +4430,8 @@ static PyObject* module_init(void)
> ADD_LDB_INT(FLG_ENABLE_TRACING);
> ADD_LDB_INT(FLG_DONT_CREATE_DB);
>
> + ADD_LDB_INT(PACKING_FORMAT);
> + ADD_LDB_INT(PACKING_FORMAT_V2);
>
> /* Historical misspelling */
> PyModule_AddIntConstant(m, "ERR_ALIAS_DEREFERINCING_PROBLEM", LDB_ERR_ALIAS_DEREFERENCING_PROBLEM);
> diff --git a/lib/ldb/tests/guidindexpackv1.ldb b/lib/ldb/tests/guidindexpackv1.ldb
> new file mode 100644
> index 00000000000..4c79dfedddd
> Binary files /dev/null and b/lib/ldb/tests/guidindexpackv1.ldb differ
> diff --git a/lib/ldb/tests/ldb_kv_ops_test.c b/lib/ldb/tests/ldb_kv_ops_test.c
> index c16297e136b..ab9d75bc69c 100644
> --- a/lib/ldb/tests/ldb_kv_ops_test.c
> +++ b/lib/ldb/tests/ldb_kv_ops_test.c
> @@ -206,6 +206,17 @@ static int parse(struct ldb_val key,
> return LDB_SUCCESS;
> }
>
> +/*
> + * Parse function that just returns the int we pass it.
> + */
> +static int parse_return(struct ldb_val key,
> + struct ldb_val data,
> + void *private_data)
> +{
> + int *rcode = private_data;
> + return *rcode;
> +}
> +
> /*
> * Test that data can be written to the kv store and be read back.
> */
> @@ -228,6 +239,7 @@ static void test_add_get(void **state)
> };
>
> struct ldb_val read;
> + int rcode;
>
> int flags = 0;
> TALLOC_CTX *tmp_ctx;
> @@ -265,6 +277,17 @@ static void test_add_get(void **state)
> assert_int_equal(sizeof(value), read.length);
> assert_memory_equal(value, read.data, sizeof(value));
>
> + /*
> + * Now check that the error code we return in the
> + * parse function is returned by fetch_and_parse.
> + */
> + for (rcode=0; rcode<50; rcode++) {
> + ret = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key,
> + parse_return,
> + &rcode);
> + assert_int_equal(ret, rcode);
> + }
> +
> ret = ldb_kv->kv_ops->unlock_read(test_ctx->ldb->modules);
> assert_int_equal(ret, 0);
> talloc_free(tmp_ctx);
> diff --git a/lib/ldb/tests/python/repack.py b/lib/ldb/tests/python/repack.py
> new file mode 100644
> index 00000000000..0844cd24e58
> --- /dev/null
> +++ b/lib/ldb/tests/python/repack.py
> @@ -0,0 +1,204 @@
> +import os
> +from unittest import TestCase
> +import shutil
> +from subprocess import check_output
> +import ldb
> +
> +TDB_PREFIX = "tdb://"
> +MDB_PREFIX = "mdb://"
> +
> +def tempdir():
> + import tempfile
> + try:
> + dir_prefix = os.path.join(os.environ["SELFTEST_PREFIX"], "tmp")
> + except KeyError:
> + dir_prefix = None
> + return tempfile.mkdtemp(dir=dir_prefix)
> +
> +
> +# Check enabling and disabling GUID indexing works and that the database is
> +# repacked at version 2 if GUID indexing is enabled, or version 1 if disabled.
> +class GUIDIndexAndPackFormatTests(TestCase):
> + prefix = TDB_PREFIX
> +
> + def setup_newdb(self):
> + self.testdir = tempdir()
> + self.filename = os.path.join(self.testdir,
> + "guidpackformattest.ldb")
> + url = self.prefix + self.filename
> + self.l = ldb.Ldb(url, options=["modules:"])
> +
> + self.num_recs_added = 0
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190529/8f5f581e/signature.sig>
More information about the samba-technical
mailing list