sambadowngradedatabase (Re: [SCM] Samba Shared Repository - branch master updated)

Stefan Metzmacher metze at samba.org
Wed May 29 11:06:14 UTC 2019


Hi Andrew,

can you please prepare backports of the sambadowngradedatabase changes?
These are needed in the old branches.

Thanks!
metze

Am 29.05.19 um 07:58 schrieb Andrew Bartlett:
> The branch, master has been updated
>        via  8831b06d3d4 ldb: Release ldb 2.0.3
>        via  4a95410a154 downgradedatabase: blackbox: MDB backend
>        via  0942a65b63c downgradedatabase: adding special case for MDB
>        via  4eee09a2c17 dsdb: disable ORDERED_INTEGER with MDB pack format v1
>        via  6b4abb99521 ldb: pack_format_override option
>        via  68d99187cd5 downgradedatabase: blackbox: database repacked
>        via  8db1312b08e ldb: python test for repack
>        via  73763acf49c ldb: repack old format database if GUID indexing enabled
>        via  d6ded22cb61 downgradedatabase: blackbox: check ordered integer removed
>        via  08b9d204b6e ldb: binding ordered indexes to GUID indexing
>        via  6005c8cbad7 ldb: only used a->syntax->index_format_fn if GUID indexing is enabled
>        via  74d15c9bf76 downgradedatabase: blackbox test
>        via  09f2a187b3d sambadowngradedatabase: Add "or later" to warning about using tools from Samba 4.8
>        via  c0b679f6a3f sambaundoguididx: renamed to downgradedatabase
>        via  40ca8ed5a15 sambaundoguididx: fix for -s
>        via  a3101b9704f ldb: Fix segfault parsing new pack formats
>        via  2de0aebed60 ldb: test for parse errors
>       from  73080f255e3 smbd: Adapt brl_pending_overlap to README.Coding


> https://git.samba.org/?p=samba.git;a=shortlog;h=master
> 
> 
> - Log -----------------------------------------------------------------
> commit 8831b06d3d4c1cb5b7732e9863228f8f3aea4e36
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Fri May 24 15:34:43 2019 +1200
> 
>     ldb: Release ldb 2.0.3
>     
>     * Default pack format changed to version 2 for GUID Indexed databases
>       (essentially Samba's sam.ldb). This means faster unpacking for records
>       with lots of attributes.
>     
>     * Enforce link between all new database features, following toggling of
>       GUID indexing. If user toggles GUID indexing on/off, target pack format
>       is changed. Likewise the special ORDERED_INTEGER index format is used
>       (when GUID indexing on) or behaves like INTEGER (when GUID indexing off).
>     
>     * Database repacked if packing format not as expected.
>     
>     * In the case of MDB, since GUID indexing is mandatory, feature toggling
>       provided by pack_format_override LDB option.
>     
>     * Check for errors from ldb_unpack_data() in ldb_tdb
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>     Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
>     
>     Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
>     Autobuild-Date(master): Wed May 29 05:56:59 UTC 2019 on sn-devel-184
> 
> commit 4a95410a1543f854743aac24c6c7dc5ffc423aae
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Fri May 24 14:54:03 2019 +1200
> 
>     downgradedatabase: blackbox: MDB backend
>     
>     When Samba uses TDB as a backend store, all new database features are
>     toggled on/off when GUID indexing is toggled. But, GUID indexing can't
>     be disabled on MDB, so the other features are toggled separately.
>     Consequently, the downgradedatabase script behaves differently depending
>     on the database backend. This patch adds testing for the MDB behaviour.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>     Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
> 
> commit 0942a65b63cc99f36d3eba99e9c9551e10c5782e
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Thu May 23 20:06:56 2019 +1200
> 
>     downgradedatabase: adding special case for MDB
>     
>     Though this script was initially written for undoing GUID indexing on
>     TDB databases, we're repurposing it to do a full downgrade of any
>     database. MDB databases can't be DN indexed, but they can have pack
>     format version 2 and ORDERED_INTEGER data types, which must be removed
>     during a downgrade.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>     Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
> 
> commit 4eee09a2c17d1276b1d0be9f26b23743eec485c2
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Thu May 23 20:13:05 2019 +1200
> 
>     dsdb: disable ORDERED_INTEGER with MDB pack format v1
>     
>     For TDB databases, the new ORDERED_INTEGER type is disabled along with
>     repacking at format version 1 if GUID indexing is disabled, so all the new
>     database features are toggled together. This scheme doesn't work with
>     MDB because GUID indexing is mandatory when using MDB. However, a
>     downgrade path is still required so in a previous commit we added
>     a pack_format_override option which allows a downgrade script to force
>     the database to use an earlier packing format. But, the new
>     ORDERED_INTEGER type would still be present in MDB databases so this
>     patch reads the pack_format_override opaque and converts ORDERED_INTEGER
>     types in @ATTRIBUTES to INTEGER and doesn't write any indexes of that
>     type to @INDEXLIST. The @INDEXLIST will be refreshed later, on the first
>     transaction.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>     Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
> 
> commit 6b4abb995215c732ff5c0bfaca2cecb7a374edff
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Thu May 23 19:49:39 2019 +1200
> 
>     ldb: pack_format_override option
>     
>     For TDB databases, toggling GUID indexing mode will also toggle
>     pack format version 2. This provides a convenient downgrade path for
>     Samba databases, but the process doesn't work for MDB databases because
>     GUID indexing cannot be disabled when the MDB backend is used. This patch
>     addresses that corner case by providing support for a pack_format_override
>     option which will force the database to use pack format version 2.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>     Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
> 
> commit 68d99187cd5f85baacbf4af262b26ae0b9682db5
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Thu May 23 14:42:35 2019 +1200
> 
>     downgradedatabase: blackbox: database repacked
>     
>     Now that the database is repacked when GUID indexing is toggled on TDB,
>     test that downgradedatabase repacks a TDB-backed database with V2 pack
>     format database with V1.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 8db1312b08ebaf8881cf633196f0f40b0badac9a
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Mon May 20 17:59:33 2019 +1200
> 
>     ldb: python test for repack
>     
>     Python test to confirm that after enabling or disabling GUID indexing,
>     the database is repacked on the next transaction with V1 if GUID
>     indexing was disabled, or V2 if it was enabled.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 73763acf49c24727e2deaaa061be8a288ee7b3d5
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Mon May 13 16:37:25 2019 +1200
> 
>     ldb: repack old format database if GUID indexing enabled
>     
>     VERY IMPORTANT PATCH
>     Now that we have a new packing format, we need to enable it by repacking
>     the database. We've decided to link all new database features together,
>     so once GUID indexing is enabled, the database will be repacked with
>     version 2 format. Repacking is done following the same iterate pattern as
>     reindexing.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit d6ded22cb617aeef75a415208b2ce56867b68047
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Thu May 23 14:35:22 2019 +1200
> 
>     downgradedatabase: blackbox: check ordered integer removed
>     
>     Test that when we undo GUID indexing on a TDB-backed database with
>     downgradedatabase, ORDERED_INTEGER is removed from @ATTRIBUTES
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 08b9d204b6eeda0aedea754f7e4b6f809d358cfe
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Wed May 22 14:07:19 2019 +1200
> 
>     ldb: binding ordered indexes to GUID indexing
>     
>     To reduce the number of potential combinations of database features in
>     ldb, we want to link all new database features since 4.7. GUID indexing,
>     ordered integers, and pack format changes will all upgrade together.
>     This patch makes ordered integers only function if GUID indexing is
>     enabled. If GUID indexing is disabled, ORDERED_INTEGER will not be
>     written to @ATTRIBUTES and a syntax's index_format_fn will never be
>     used.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 6005c8cbad77259fa3cb89ab21cb5f26b72413cd
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Tue May 28 13:00:43 2019 +1200
> 
>     ldb: only used a->syntax->index_format_fn if GUID indexing is enabled
>     
>     Only enable ORDERED_INTEGER and index_format_fn functionality if GUID
>     indexing is enabled.  This is in line with the rest of ldb_kv which binds
>     the new ORDERED_INTEGER to GUID indexed databases, and allows a practical
>     way to create the old index format (by disabling the GUID index).
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
>     Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
> 
> commit 74d15c9bf76f0a2fb5fa7b7b1d80971d10c4fe45
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Thu May 23 13:21:19 2019 +1200
> 
>     downgradedatabase: blackbox test
>     
>     This test confirms that running downgradedatabase causes all GUID keys to be
>     replaced with DN keys at the KV level
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 09f2a187b3d8c161e2c11588499b3256a9dbcc95
> Author: Andrew Bartlett <abartlet at samba.org>
> Date:   Wed May 29 16:36:00 2019 +1200
> 
>     sambadowngradedatabase: Add "or later" to warning about using tools from Samba 4.8
>     
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit c0b679f6a3f21e262d03bf38ea63900d30c29bb5
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Fri May 24 14:37:50 2019 +1200
> 
>     sambaundoguididx: renamed to downgradedatabase
>     
>     In forthcoming commits we're going to repurpose this script to do an
>     entire downgrade of a database, disabling all new database features.
>     downgradedatabase is a more appropriate name.
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 40ca8ed5a152ae7c5ec039649c09a037a20a4143
> Author: Andrew Bartlett <abartlet at samba.org>
> Date:   Mon May 20 16:29:10 2019 +1200
> 
>     sambaundoguididx: fix for -s
>     
>     Quick fix running this script with -s instead of -H. samdb_url() returns
>     a url with a protocol prefix, which causes issues further down in the
>     script.
>     
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit a3101b9704f554a350493553336cbbbd7d4ae02e
> Author: Andrew Bartlett <abartlet at samba.org>
> Date:   Wed May 22 16:38:08 2019 +1200
> 
>     ldb: Fix segfault parsing new pack formats
>     
>     We need to check for the errors given by ldb_unpack() et al by preserving
>     the error code from kv_ctx->parser() called by tdb_parse_record() in
>     ltdb_parse_record().
>     
>     Otherwise we will silently accept corrupt records and segfault later.
>     
>     Likewise new pack formats will confuse the parser but not be
>     detected except by the incomplete struct ldb_message.
>     
>     With this patch, the user will see a message like:
>     
>      Invalid data for index  DN=@BASEINFO
>     
>      Failed to connect to 'st/ad_dc/private/sam.ldb' with backend 'tdb': Unable to load ltdb cache records for backend 'ldb_tdb backend'
>      Failed to connect to st/ad_dc/private/sam.ldb - Unable to load ltdb cache records for backend 'ldb_tdb backend'
>     
>     This can be refined in the future by a specific check for
>     pack format versions in a higher caller, but this much is
>     needed regardless to detect corrupt records.
>     
>     BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
>     
>     Signed-off-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> commit 2de0aebed60b8e83508f50e5391ede618ce0e595
> Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
> Date:   Tue May 28 17:22:10 2019 +1200
> 
>     ldb: test for parse errors
>     
>     Parse errors aren't passed up correctly by the tdb backend. This
>     patch modifies a test to expose the issue, next patch will fix it.
>     
>     BUG: https://bugzilla.samba.org/show_bug.cgi?id=13959
>     
>     Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>     Reviewed-by: Garming Sam <garming at catalyst.net.nz>
> 
> -----------------------------------------------------------------------
> 
> Summary of changes:
>  lib/ldb/ABI/{ldb-2.0.2.sigs => ldb-2.0.3.sigs}     |   0
>  ...yldb-util-1.1.10.sigs => pyldb-util-2.0.3.sigs} |   0
>  lib/ldb/ldb_key_value/ldb_kv.c                     |  80 +++++++-
>  lib/ldb/ldb_key_value/ldb_kv.h                     |   9 +
>  lib/ldb/ldb_key_value/ldb_kv_cache.c               |  52 +++---
>  lib/ldb/ldb_key_value/ldb_kv_index.c               |  93 +++++++++-
>  lib/ldb/ldb_tdb/ldb_tdb.c                          |   8 +-
>  lib/ldb/pyldb.c                                    |   2 +
>  lib/ldb/tests/guidindexpackv1.ldb                  | Bin 0 -> 1286144 bytes
>  lib/ldb/tests/ldb_kv_ops_test.c                    |  23 +++
>  lib/ldb/tests/python/repack.py                     | 204 +++++++++++++++++++++
>  lib/ldb/wscript                                    |   6 +-
>  python/samba/tests/__init__.py                     |   9 +-
>  python/samba/tests/blackbox/downgradedatabase.py   | 158 ++++++++++++++++
>  source4/dsdb/schema/schema_set.c                   |  97 +++++++++-
>  .../{sambaundoguididx => sambadowngradedatabase}   |  37 +++-
>  source4/selftest/tests.py                          |   2 +
>  17 files changed, 728 insertions(+), 52 deletions(-)
>  copy lib/ldb/ABI/{ldb-2.0.2.sigs => ldb-2.0.3.sigs} (100%)
>  copy lib/ldb/ABI/{pyldb-util-1.1.10.sigs => pyldb-util-2.0.3.sigs} (100%)
>  create mode 100644 lib/ldb/tests/guidindexpackv1.ldb
>  create mode 100644 lib/ldb/tests/python/repack.py
>  create mode 100644 python/samba/tests/blackbox/downgradedatabase.py
>  rename source4/scripting/bin/{sambaundoguididx => sambadowngradedatabase} (62%)
> 
> 
> Changeset truncated at 500 lines:
> 
> diff --git a/lib/ldb/ABI/ldb-2.0.2.sigs b/lib/ldb/ABI/ldb-2.0.3.sigs
> similarity index 100%
> copy from lib/ldb/ABI/ldb-2.0.2.sigs
> copy to lib/ldb/ABI/ldb-2.0.3.sigs
> diff --git a/lib/ldb/ABI/pyldb-util-1.1.10.sigs b/lib/ldb/ABI/pyldb-util-2.0.3.sigs
> similarity index 100%
> copy from lib/ldb/ABI/pyldb-util-1.1.10.sigs
> copy to lib/ldb/ABI/pyldb-util-2.0.3.sigs
> diff --git a/lib/ldb/ldb_key_value/ldb_kv.c b/lib/ldb/ldb_key_value/ldb_kv.c
> index c8f7fd1396d..d9b7b0af46a 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv.c
> @@ -301,6 +301,34 @@ static int ldb_kv_check_special_dn(struct ldb_module *module,
>  	return LDB_SUCCESS;
>  }
>  
> +/*
> + * Called after modifies and when starting a transaction. Checks target pack
> + * format version and current pack format version, which are set by cache_load,
> + * and repacks if necessary.
> + */
> +static int ldb_kv_maybe_repack(struct ldb_kv_private *ldb_kv) {
> +	/* Override option taken from ldb options */
> +	if (ldb_kv->pack_format_override != 0) {
> +		ldb_kv->target_pack_format_version =
> +			ldb_kv->pack_format_override;
> +	}
> +
> +	if (ldb_kv->pack_format_version !=
> +	    ldb_kv->target_pack_format_version) {
> +		int r;
> +		struct ldb_context *ldb = ldb_module_get_ctx(ldb_kv->module);
> +		ldb_kv->pack_format_version =
> +			ldb_kv->target_pack_format_version;
> +		r = ldb_kv_repack(ldb_kv->module);
> +		if (r != LDB_SUCCESS) {
> +			ldb_debug(ldb, LDB_DEBUG_ERROR,
> +				  "Database repack failed.");
> +		}
> +		return r;
> +	}
> +
> +	return LDB_SUCCESS;
> +}
>  
>  /*
>    we've made a modification to a dn - possibly reindex and
> @@ -1447,6 +1475,20 @@ static int ldb_kv_prepare_commit(struct ldb_module *module)
>  		return ret;
>  	}
>  
> +	/*
> +	 * If GUID indexing was toggled in this transaction, we repack at
> +	 * format version 2 if GUID indexing was enabled, or version 1 if
> +	 * it was disabled.
> +	 */
> +	ret = ldb_kv_maybe_repack(ldb_kv);
> +	if (ret != LDB_SUCCESS) {
> +		ldb_kv_del_trans(module);
> +		ldb_set_errstring(ldb_module_get_ctx(module),
> +				  "Failure during re-pack, so "
> +				  "transaction must be aborted.");
> +		return ret;
> +	}
> +
>  	if (ldb_kv->kv_ops->prepare_write(ldb_kv) != 0) {
>  		ret = ldb_kv->kv_ops->error(ldb_kv);
>  		ldb_debug_set(ldb_module_get_ctx(module),
> @@ -1895,10 +1937,10 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
>  
>  	ldb_kv->sequence_number = 0;
>  
> -	ldb_kv->pack_format_version = LDB_PACKING_FORMAT;
> -
>  	ldb_kv->pid = getpid();
>  
> +	ldb_kv->pack_format_override = 0;
> +
>  	ldb_kv->module = ldb_module_new(ldb, ldb, name, &ldb_kv_ops);
>  	if (!ldb_kv->module) {
>  		ldb_oom(ldb);
> @@ -1935,6 +1977,40 @@ int ldb_kv_init_store(struct ldb_kv_private *ldb_kv,
>  		}
>  	}
>  
> +	/*
> +	 * Usually the presence of GUID indexing determines the pack format
> +	 * we use but in certain circumstances such as downgrading an
> +	 * MDB-backed database, we want to override the target pack format.
> +	 *
> +	 * We set/get opaques here because in the Samba partitions module,
> +	 * 'options' are not passed correctly so sub-databases can't see
> +	 * the options they need.
> +	 */
> +	{
> +		const char *pack_format_override =
> +			ldb_options_find(ldb, options, "pack_format_override");
> +		if (pack_format_override != NULL) {
> +			int ret;
> +			ldb_kv->pack_format_override =
> +				strtoul(pack_format_override, NULL, 0);
> +			ret = ldb_set_opaque(ldb,
> +					     "pack_format_override",
> +			     (void *)(intptr_t)ldb_kv->pack_format_override);
> +			if (ret != LDB_SUCCESS) {
> +				talloc_free(ldb_kv->module);
> +				return ldb_module_operr(ldb_kv->module);
> +			}
> +		} else {
> +			/*
> +			 * NULL -> 0 is fine, otherwise we get back
> +			 * the number we needed
> +			 */
> +			ldb_kv->pack_format_override
> +				= (intptr_t)ldb_get_opaque(ldb,
> +						   "pack_format_override");
> +		}
> +	}
> +
>  	/*
>  	 * Override full DB scans
>  	 *
> diff --git a/lib/ldb/ldb_key_value/ldb_kv.h b/lib/ldb/ldb_key_value/ldb_kv.h
> index 1186e86ec9f..8da970fd2f9 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv.h
> +++ b/lib/ldb/ldb_key_value/ldb_kv.h
> @@ -64,6 +64,8 @@ struct ldb_kv_private {
>  
>  	unsigned long long sequence_number;
>  	uint32_t pack_format_version;
> +	uint32_t target_pack_format_version;
> +	uint32_t pack_format_override;
>  
>  	/* the low level tdb seqnum - used to avoid loading BASEINFO when
>  	   possible */
> @@ -141,6 +143,12 @@ struct ldb_kv_reindex_context {
>  	uint32_t count;
>  };
>  
> +struct ldb_kv_repack_context {
> +	int error;
> +	uint32_t count;
> +	bool normal_record_seen;
> +};
> +
>  
>  /* special record types */
>  #define LDB_KV_INDEX      "@INDEX"
> @@ -226,6 +234,7 @@ int ldb_kv_index_del_value(struct ldb_module *module,
>  			   struct ldb_message_element *el,
>  			   unsigned int v_idx);
>  int ldb_kv_reindex(struct ldb_module *module);
> +int ldb_kv_repack(struct ldb_module *module);
>  int ldb_kv_index_transaction_start(
>  	struct ldb_module *module,
>  	size_t cache_size);
> diff --git a/lib/ldb/ldb_key_value/ldb_kv_cache.c b/lib/ldb/ldb_key_value/ldb_kv_cache.c
> index c5f661113fd..3d8a09123e0 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv_cache.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv_cache.c
> @@ -418,7 +418,6 @@ int ldb_kv_cache_load(struct ldb_module *module)
>  	const struct ldb_schema_attribute *a;
>  	bool have_write_txn = false;
>  	int r;
> -	uint32_t pack_format_version;
>  	struct ldb_val key;
>  
>  	ldb = ldb_module_get_ctx(module);
> @@ -453,29 +452,7 @@ int ldb_kv_cache_load(struct ldb_module *module)
>  	/* Read packing format from first 4 bytes of @BASEINFO record */
>  	r = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key,
>  					    get_pack_format_version,
> -					    &pack_format_version);
> -
> -	if (r != LDB_ERR_NO_SUCH_OBJECT) {
> -		if (r != LDB_SUCCESS) {
> -			goto failed_and_unlock;
> -		}
> -
> -		/* Make sure the database has the right format */
> -		if (pack_format_version != ldb_kv->pack_format_version) {
> -			ldb_debug(ldb, LDB_DEBUG_ERROR,
> -				  "Unexpected packing format. "
> -				  "Expected: %#010x, Got: %#010x",
> -				  pack_format_version,
> -				  ldb_kv->pack_format_version);
> -			goto failed_and_unlock;
> -		}
> -	}
> -
> -	/* Now fetch the whole @BASEINFO record */
> -	r = ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0);
> -	if (r != LDB_SUCCESS && r != LDB_ERR_NO_SUCH_OBJECT) {
> -		goto failed_and_unlock;
> -	}
> +					    &ldb_kv->pack_format_version);
>  
>  	/* possibly initialise the baseinfo */
>  	if (r == LDB_ERR_NO_SUCH_OBJECT) {
> @@ -492,15 +469,25 @@ int ldb_kv_cache_load(struct ldb_module *module)
>  
>  		have_write_txn = true;
>  
> +		/*
> +		 * We need to write but haven't figured out packing format yet.
> +		 * Just go with version 1 and we'll repack if we got it wrong.
> +		 */
> +		ldb_kv->pack_format_version = LDB_PACKING_FORMAT;
> +		ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT;
> +
>  		/* error handling for ltdb_baseinfo_init() is by
>  		   looking for the record again. */
>  		ldb_kv_baseinfo_init(module);
>  
> -		if (ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0) !=
> -		    LDB_SUCCESS) {
> -			goto failed_and_unlock;
> -		}
> +	} else if (r != LDB_SUCCESS) {
> +		goto failed_and_unlock;
> +	}
>  
> +	/* OK now we definitely have a @BASEINFO record so fetch it */
> +	r = ldb_kv_search_dn1(module, baseinfo_dn, baseinfo, 0);
> +	if (r != LDB_SUCCESS) {
> +		goto failed_and_unlock;
>  	}
>  
>  	/* Ignore the result, and update the sequence number */
> @@ -562,8 +549,15 @@ int ldb_kv_cache_load(struct ldb_module *module)
>  		goto failed_and_unlock;
>  	}
>  
> +	/*
> +	 * Initialise packing version and GUID index syntax, and force the
> +	 * two to travel together, ie a GUID indexed database must use V2
> +	 * packing format and a DN indexed database must use V1.
> +	 */
>  	ldb_kv->GUID_index_syntax = NULL;
>  	if (ldb_kv->cache->GUID_index_attribute != NULL) {
> +		ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT_V2;
> +
>  		/*
>  		 * Now the attributes are loaded, set the guid_index_syntax.
>  		 * This can't fail, it will return a default at worst
> @@ -571,6 +565,8 @@ int ldb_kv_cache_load(struct ldb_module *module)
>  		a = ldb_schema_attribute_by_name(
>  		    ldb, ldb_kv->cache->GUID_index_attribute);
>  		ldb_kv->GUID_index_syntax = a->syntax;
> +	} else {
> +		ldb_kv->target_pack_format_version = LDB_PACKING_FORMAT;
>  	}
>  
>  done:
> diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c
> index f6b2620a875..5af248a1931 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv_index.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
> @@ -924,7 +924,8 @@ static struct ldb_dn *ldb_kv_index_key(struct ldb_context *ldb,
>  			v = *value;
>  		} else {
>  			ldb_attr_handler_t fn;
> -			if (a->syntax->index_format_fn) {
> +			if (a->syntax->index_format_fn &&
> +			    ldb_kv->cache->GUID_index_attribute != NULL) {
>  				fn = a->syntax->index_format_fn;
>  			} else {
>  				fn = a->syntax->canonicalise_fn;
> @@ -3411,6 +3412,96 @@ static int re_index(struct ldb_kv_private *ldb_kv,
>  	return 0;
>  }
>  
> +static int re_pack(struct ldb_kv_private *ldb_kv,
> +		   struct ldb_val key,
> +		   struct ldb_val val,
> +		   void *state)
> +{
> +	struct ldb_context *ldb;
> +	struct ldb_message *msg;
> +	struct ldb_module *module = ldb_kv->module;
> +	struct ldb_kv_repack_context *ctx =
> +	    (struct ldb_kv_repack_context *)state;
> +	int ret;
> +
> +	ldb = ldb_module_get_ctx(module);
> +
> +	msg = ldb_msg_new(module);
> +	if (msg == NULL) {
> +		return -1;
> +	}
> +
> +	ret = ldb_unpack_data(ldb, &val, msg);
> +	if (ret != 0) {
> +		ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack: unpack failed: %s\n",
> +			  ldb_dn_get_linearized(msg->dn));
> +		ctx->error = ret;
> +		talloc_free(msg);
> +		return -1;
> +	}
> +
> +	ret = ldb_kv_store(module, msg, TDB_MODIFY);
> +	if (ret != LDB_SUCCESS) {
> +		ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack: store failed: %s\n",
> +			  ldb_dn_get_linearized(msg->dn));
> +		ctx->error = ret;
> +		talloc_free(msg);
> +		return -1;
> +	}
> +
> +	/*
> +	 * Warn the user that we're repacking the first time we see a normal
> +	 * record. This means we never warn if we're repacking a database with
> +	 * only @ records. This is because during database initialisation,
> +	 * we might repack as initial settings are written out, and we don't
> +	 * want to spam the log.
> +	 */
> +	if ((!ctx->normal_record_seen) && (!ldb_dn_is_special(msg->dn))) {
> +		ldb_debug(ldb, LDB_DEBUG_WARNING,
> +			  "Repacking database with format %#010x",
> +			  ldb_kv->pack_format_version);
> +		ctx->normal_record_seen = true;
> +	}
> +
> +	ctx->count++;
> +	if (ctx->count % 10000 == 0) {
> +		ldb_debug(ldb, LDB_DEBUG_WARNING,
> +			  "Repack: re-packed %u records so far",
> +			  ctx->count);
> +	}
> +
> +	return 0;
> +}
> +
> +int ldb_kv_repack(struct ldb_module *module)
> +{
> +	struct ldb_kv_private *ldb_kv = talloc_get_type(
> +	    ldb_module_get_private(module), struct ldb_kv_private);
> +	struct ldb_context *ldb = ldb_module_get_ctx(module);
> +	struct ldb_kv_repack_context ctx;
> +	int ret;
> +
> +	ctx.count = 0;
> +	ctx.error = LDB_SUCCESS;
> +	ctx.normal_record_seen = false;
> +
> +	/* Iterate all database records and repack them in the new format */
> +	ret = ldb_kv->kv_ops->iterate(ldb_kv, re_pack, &ctx);
> +	if (ret < 0) {
> +		ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack traverse failed: %s",
> +			  ldb_errstring(ldb));
> +		return LDB_ERR_OPERATIONS_ERROR;
> +	}
> +
> +	if (ctx.error != LDB_SUCCESS) {
> +		ldb_debug(ldb, LDB_DEBUG_ERROR, "Repack failed: %s",
> +			  ldb_errstring(ldb));
> +		return ctx.error;
> +	}
> +
> +	return LDB_SUCCESS;
> +}
> +
>  /*
>    force a complete reindex of the database
>  */
> diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
> index 2f0468fd40a..6e2cb633a4d 100644
> --- a/lib/ldb/ldb_tdb/ldb_tdb.c
> +++ b/lib/ldb/ldb_tdb/ldb_tdb.c
> @@ -242,6 +242,7 @@ struct kv_ctx {
>  	int (*parser)(struct ldb_val key,
>  		      struct ldb_val data,
>  		      void *private_data);
> +	int parser_ret;
>  };
>  
>  static int ltdb_traverse_fn_wrapper(struct tdb_context *tdb,
> @@ -350,7 +351,8 @@ static int ltdb_parse_record_wrapper(TDB_DATA tdb_key,
>  		.data = tdb_data.dptr,
>  	};
>  
> -	return kv_ctx->parser(key, data, kv_ctx->ctx);
> +	kv_ctx->parser_ret = kv_ctx->parser(key, data, kv_ctx->ctx);
> +	return kv_ctx->parser_ret;
>  }
>  
>  static int ltdb_parse_record(struct ldb_kv_private *ldb_kv,
> @@ -374,7 +376,9 @@ static int ltdb_parse_record(struct ldb_kv_private *ldb_kv,
>  
>  	ret = tdb_parse_record(
>  	    ldb_kv->tdb, key, ltdb_parse_record_wrapper, &kv_ctx);
> -	if (ret == 0) {
> +	if (kv_ctx.parser_ret != LDB_SUCCESS) {
> +		return kv_ctx.parser_ret;
> +	} else if (ret == 0) {
>  		return LDB_SUCCESS;
>  	}
>  	return ltdb_err_map(tdb_error(ldb_kv->tdb));
> diff --git a/lib/ldb/pyldb.c b/lib/ldb/pyldb.c
> index cf7779128d4..940e43e3192 100644
> --- a/lib/ldb/pyldb.c
> +++ b/lib/ldb/pyldb.c
> @@ -4430,6 +4430,8 @@ static PyObject* module_init(void)
>  	ADD_LDB_INT(FLG_ENABLE_TRACING);
>  	ADD_LDB_INT(FLG_DONT_CREATE_DB);
>  
> +	ADD_LDB_INT(PACKING_FORMAT);
> +	ADD_LDB_INT(PACKING_FORMAT_V2);
>  
>  	/* Historical misspelling */
>  	PyModule_AddIntConstant(m, "ERR_ALIAS_DEREFERINCING_PROBLEM", LDB_ERR_ALIAS_DEREFERENCING_PROBLEM);
> diff --git a/lib/ldb/tests/guidindexpackv1.ldb b/lib/ldb/tests/guidindexpackv1.ldb
> new file mode 100644
> index 00000000000..4c79dfedddd
> Binary files /dev/null and b/lib/ldb/tests/guidindexpackv1.ldb differ
> diff --git a/lib/ldb/tests/ldb_kv_ops_test.c b/lib/ldb/tests/ldb_kv_ops_test.c
> index c16297e136b..ab9d75bc69c 100644
> --- a/lib/ldb/tests/ldb_kv_ops_test.c
> +++ b/lib/ldb/tests/ldb_kv_ops_test.c
> @@ -206,6 +206,17 @@ static int parse(struct ldb_val key,
>  	return LDB_SUCCESS;
>  }
>  
> +/*
> + * Parse function that just returns the int we pass it.
> + */
> +static int parse_return(struct ldb_val key,
> +		        struct ldb_val data,
> +		        void *private_data)
> +{
> +	int *rcode = private_data;
> +	return *rcode;
> +}
> +
>  /*
>   * Test that data can be written to the kv store and be read back.
>   */
> @@ -228,6 +239,7 @@ static void test_add_get(void **state)
>  	};
>  
>  	struct ldb_val read;
> +	int rcode;
>  
>  	int flags = 0;
>  	TALLOC_CTX *tmp_ctx;
> @@ -265,6 +277,17 @@ static void test_add_get(void **state)
>  	assert_int_equal(sizeof(value), read.length);
>  	assert_memory_equal(value, read.data, sizeof(value));
>  
> +	/*
> +	 * Now check that the error code we return in the
> +	 * parse function is returned by fetch_and_parse.
> +	 */
> +	for (rcode=0; rcode<50; rcode++) {
> +		ret = ldb_kv->kv_ops->fetch_and_parse(ldb_kv, key,
> +						      parse_return,
> +						      &rcode);
> +		assert_int_equal(ret, rcode);
> +	}
> +
>  	ret = ldb_kv->kv_ops->unlock_read(test_ctx->ldb->modules);
>  	assert_int_equal(ret, 0);
>  	talloc_free(tmp_ctx);
> diff --git a/lib/ldb/tests/python/repack.py b/lib/ldb/tests/python/repack.py
> new file mode 100644
> index 00000000000..0844cd24e58
> --- /dev/null
> +++ b/lib/ldb/tests/python/repack.py
> @@ -0,0 +1,204 @@
> +import os
> +from unittest import TestCase
> +import shutil
> +from subprocess import check_output
> +import ldb
> +
> +TDB_PREFIX = "tdb://"
> +MDB_PREFIX = "mdb://"
> +
> +def tempdir():
> +    import tempfile
> +    try:
> +        dir_prefix = os.path.join(os.environ["SELFTEST_PREFIX"], "tmp")
> +    except KeyError:
> +        dir_prefix = None
> +    return tempfile.mkdtemp(dir=dir_prefix)
> +
> +
> +# Check enabling and disabling GUID indexing works and that the database is
> +# repacked at version 2 if GUID indexing is enabled, or version 1 if disabled.
> +class GUIDIndexAndPackFormatTests(TestCase):
> +    prefix = TDB_PREFIX
> +
> +    def setup_newdb(self):
> +        self.testdir = tempdir()
> +        self.filename = os.path.join(self.testdir,
> +                                     "guidpackformattest.ldb")
> +        url = self.prefix + self.filename
> +        self.l = ldb.Ldb(url, options=["modules:"])
> +
> +        self.num_recs_added = 0
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-cvs/attachments/20190529/8f5f581e/signature-0001.sig>


More information about the samba-cvs mailing list