[SCM] Samba Shared Repository - branch master updated -
release-4-0-0alpha7-1939-ga6cc04a
Andrew Tridgell
tridge at samba.org
Mon Jun 1 03:14:08 GMT 2009
The branch, master has been updated
via a6cc04a20089e8fbcce138c271961c37ddcd6c34 (commit)
via a386173fa1c7c5bcc11ea9260d84b6c52c154b3d (commit)
via 7f16c29f1001b98fd1bdc66978a2c16f8e18d19d (commit)
via ba95882155db4f8c10725f47f70ae482d5343f9a (commit)
via 67b83d2489788f1899c253fdab554d0998f9c044 (commit)
via 08be1420ba52ef9bba90d0f811c7810841ee8568 (commit)
via 5ab03dbecc33320e23304b126f26bde3f6bc6c7d (commit)
via 47692f39b1b3b5c589bfb63a6968aaf9f9af70c4 (commit)
via 0a0d760704ad17f5f43ffc1782ad4d84f75de5a2 (commit)
via 52cfffb8cba92e272224febd3d8df68b93e29bc7 (commit)
via 4b4fec65db4e202afa13b2d15867f4d8a54d154e (commit)
via a9542ba21b4d4ef088ac2cf31f9f4074dc211aa8 (commit)
from 14c13620345dfd9f1e18761f103aa66138bf8907 (commit)
http://gitweb.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit a6cc04a20089e8fbcce138c271961c37ddcd6c34
Author: Andrew Tridgell <tridge at samba.org>
Date: Mon Jun 1 13:13:07 2009 +1000
overallocate all records by 25%
This greatly reduces the fragmentation of databases where records
tend to grow slowly by a small amount each time. The case where this
is most seen is the ldb index records. Adding this overallocation
reduced the size of the resulting database by more than 20x when
running a test that adds 10k users.
commit a386173fa1c7c5bcc11ea9260d84b6c52c154b3d
Author: Andrew Tridgell <tridge at samba.org>
Date: Mon Jun 1 13:11:39 2009 +1000
auto-repack in transactions that expand the tdb
The idea behind this is to recover from badly fragmented free
lists. Choosing the point where the file expands is fairly arbitrary,
but seems to work well.
commit 7f16c29f1001b98fd1bdc66978a2c16f8e18d19d
Author: Andrew Tridgell <tridge at samba.org>
Date: Mon Jun 1 12:58:48 2009 +1000
when comp_num is zero, the case folded DN is always ""
This fixes a bug where we would look at an uninitialised
dn->linearized
commit ba95882155db4f8c10725f47f70ae482d5343f9a
Merge: 67b83d2489788f1899c253fdab554d0998f9c044 14c13620345dfd9f1e18761f103aa66138bf8907
Author: Andrew Tridgell <tridge at samba.org>
Date: Mon Jun 1 10:39:30 2009 +1000
Merge branch 'master' of ssh://git.samba.org/data/git/samba
commit 67b83d2489788f1899c253fdab554d0998f9c044
Merge: 08be1420ba52ef9bba90d0f811c7810841ee8568 e63d9c29c99c5311c9f4a8dbe432ff4cea4fb924
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 22:12:00 2009 +1000
Merge branch 'master' of ssh://git.samba.org/data/git/samba
commit 08be1420ba52ef9bba90d0f811c7810841ee8568
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 17:35:12 2009 +1000
fixed tdbbackup to give tdb error messages
commit 5ab03dbecc33320e23304b126f26bde3f6bc6c7d
Merge: 47692f39b1b3b5c589bfb63a6968aaf9f9af70c4 bd1194810787901c5caa08961f97fecbcbd01978
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 16:12:34 2009 +1000
Merge branch 'master' of ssh://git.samba.org/data/git/samba
commit 47692f39b1b3b5c589bfb63a6968aaf9f9af70c4
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 16:11:45 2009 +1000
a useful debugging tool
When looking at performance problems with ldb it can be useful to see
which searches causes unindexed full searches. This makes it easy to
enable that.
commit 0a0d760704ad17f5f43ffc1782ad4d84f75de5a2
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 16:10:47 2009 +1000
fixed one-level indexing
one-level indexing was not always effective due to some broken logic
in the indexing code. This change means that if normal indexing fails,
we can still fall back on one-level indexing.
This reduces the number of full unindexed searches in s4 quite a lot
commit 52cfffb8cba92e272224febd3d8df68b93e29bc7
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 16:08:49 2009 +1000
enable one-level indexing in sam.ldb
commit 4b4fec65db4e202afa13b2d15867f4d8a54d154e
Author: Andrew Tridgell <tridge at samba.org>
Date: Thu May 28 16:08:28 2009 +1000
make TDB_NOSYNC affect all the fsync/msync calls in transactions
During a transaction commit tdb normally uses fsync/msync calls to
make it crash safe. This can be disabled using the TDB_NOSYNC flag,
but it wasn't disabling all the code paths that caused a fsync/msync.
commit a9542ba21b4d4ef088ac2cf31f9f4074dc211aa8
Author: Andrew Tridgell <tridge at samba.org>
Date: Tue May 26 14:47:11 2009 +1000
use domain_dn not ncname
fixed up from previous patch that removed the use of crossref records
-----------------------------------------------------------------------
Summary of changes:
lib/tdb/common/freelist.c | 3 ++
lib/tdb/common/transaction.c | 24 +++++++++++++++++----
lib/tdb/tools/tdbbackup.c | 38 +++++++++++++++++++++++++++------
source4/auth/sam.c | 5 +--
source4/dsdb/schema/schema_set.c | 5 ++++
source4/lib/ldb/common/ldb_dn.c | 10 +-------
source4/lib/ldb/ldb_tdb/ldb_index.c | 8 +-----
source4/lib/ldb/ldb_tdb/ldb_search.c | 7 ++++++
8 files changed, 71 insertions(+), 29 deletions(-)
Changeset truncated at 500 lines:
diff --git a/lib/tdb/common/freelist.c b/lib/tdb/common/freelist.c
index 2f2a4c3..3bc3965 100644
--- a/lib/tdb/common/freelist.c
+++ b/lib/tdb/common/freelist.c
@@ -284,6 +284,9 @@ tdb_off_t tdb_allocate(struct tdb_context *tdb, tdb_len_t length, struct list_st
if (tdb_lock(tdb, -1, F_WRLCK) == -1)
return 0;
+ /* over-allocate to reduce fragmentation */
+ length *= 1.25;
+
/* Extra bytes required for tailer */
length += sizeof(tdb_off_t);
length = TDB_ALIGN(length, TDB_ALIGNMENT);
diff --git a/lib/tdb/common/transaction.c b/lib/tdb/common/transaction.c
index cb723ed..e97fe67 100644
--- a/lib/tdb/common/transaction.c
+++ b/lib/tdb/common/transaction.c
@@ -122,6 +122,9 @@ struct tdb_transaction {
/* old file size before transaction */
tdb_len_t old_map_size;
+
+ /* we should re-pack on commit */
+ bool need_repack;
};
@@ -392,6 +395,8 @@ static int transaction_expand_file(struct tdb_context *tdb, tdb_off_t size,
return -1;
}
+ tdb->transaction->need_repack = true;
+
return 0;
}
@@ -522,6 +527,10 @@ fail:
*/
static int transaction_sync(struct tdb_context *tdb, tdb_off_t offset, tdb_len_t length)
{
+ if (tdb->flags & TDB_NOSYNC) {
+ return 0;
+ }
+
if (fsync(tdb->fd) != 0) {
tdb->ecode = TDB_ERR_IO;
TDB_LOG((tdb, TDB_DEBUG_FATAL, "tdb_transaction: fsync failed\n"));
@@ -961,6 +970,7 @@ int tdb_transaction_commit(struct tdb_context *tdb)
{
const struct tdb_methods *methods;
int i;
+ bool need_repack;
if (tdb->transaction == NULL) {
TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_transaction_commit: no transaction\n"));
@@ -1030,11 +1040,9 @@ int tdb_transaction_commit(struct tdb_context *tdb)
SAFE_FREE(tdb->transaction->blocks);
tdb->transaction->num_blocks = 0;
- if (!(tdb->flags & TDB_NOSYNC)) {
- /* ensure the new data is on disk */
- if (transaction_sync(tdb, 0, tdb->map_size) == -1) {
- return -1;
- }
+ /* ensure the new data is on disk */
+ if (transaction_sync(tdb, 0, tdb->map_size) == -1) {
+ return -1;
}
tdb_brlock(tdb, GLOBAL_LOCK, F_UNLCK, F_SETLKW, 0, 1);
@@ -1054,10 +1062,16 @@ int tdb_transaction_commit(struct tdb_context *tdb)
utime(tdb->name, NULL);
#endif
+ need_repack = tdb->transaction->need_repack;
+
/* use a transaction cancel to free memory and remove the
transaction locks */
tdb_transaction_cancel(tdb);
+ if (need_repack) {
+ return tdb_repack(tdb);
+ }
+
return 0;
}
diff --git a/lib/tdb/tools/tdbbackup.c b/lib/tdb/tools/tdbbackup.c
index 83c0e16..6aca8dd 100644
--- a/lib/tdb/tools/tdbbackup.c
+++ b/lib/tdb/tools/tdbbackup.c
@@ -53,6 +53,21 @@
static int failed;
+static struct tdb_logging_context log_ctx;
+
+#ifdef PRINTF_ATTRIBUTE
+static void tdb_log(struct tdb_context *tdb, enum tdb_debug_level level, const char *format, ...) PRINTF_ATTRIBUTE(3,4);
+#endif
+static void tdb_log(struct tdb_context *tdb, enum tdb_debug_level level, const char *format, ...)
+{
+ va_list ap;
+
+ va_start(ap, format);
+ vfprintf(stdout, format, ap);
+ va_end(ap);
+ fflush(stdout);
+}
+
static char *add_suffix(const char *name, const char *suffix)
{
char *ret;
@@ -107,7 +122,8 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size)
}
/* open the old tdb */
- tdb = tdb_open(old_name, 0, 0, O_RDWR, 0);
+ tdb = tdb_open_ex(old_name, 0, 0,
+ O_RDWR, 0, &log_ctx, NULL);
if (!tdb) {
printf("Failed to open %s\n", old_name);
free(tmp_name);
@@ -116,10 +132,11 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size)
/* create the new tdb */
unlink(tmp_name);
- tdb_new = tdb_open(tmp_name,
- hash_size ? hash_size : tdb_hash_size(tdb),
- TDB_DEFAULT, O_RDWR|O_CREAT|O_EXCL,
- st.st_mode & 0777);
+ tdb_new = tdb_open_ex(tmp_name,
+ hash_size ? hash_size : tdb_hash_size(tdb),
+ TDB_DEFAULT,
+ O_RDWR|O_CREAT|O_EXCL, st.st_mode & 0777,
+ &log_ctx, NULL);
if (!tdb_new) {
perror(tmp_name);
free(tmp_name);
@@ -170,7 +187,11 @@ static int backup_tdb(const char *old_name, const char *new_name, int hash_size)
/* close the new tdb and re-open read-only */
tdb_close(tdb_new);
- tdb_new = tdb_open(tmp_name, 0, TDB_DEFAULT, O_RDONLY, 0);
+ tdb_new = tdb_open_ex(tmp_name,
+ 0,
+ TDB_DEFAULT,
+ O_RDONLY, 0,
+ &log_ctx, NULL);
if (!tdb_new) {
fprintf(stderr,"failed to reopen %s\n", tmp_name);
unlink(tmp_name);
@@ -211,7 +232,8 @@ static int verify_tdb(const char *fname, const char *bak_name)
int count = -1;
/* open the tdb */
- tdb = tdb_open(fname, 0, 0, O_RDONLY, 0);
+ tdb = tdb_open_ex(fname, 0, 0,
+ O_RDONLY, 0, &log_ctx, NULL);
/* traverse the tdb, then close it */
if (tdb) {
@@ -264,6 +286,8 @@ static void usage(void)
int hashsize = 0;
const char *suffix = ".bak";
+ log_ctx.log_fn = tdb_log;
+
while ((c = getopt(argc, argv, "vhs:n:")) != -1) {
switch (c) {
case 'h':
diff --git a/source4/auth/sam.c b/source4/auth/sam.c
index ebdf193..c70c02c 100644
--- a/source4/auth/sam.c
+++ b/source4/auth/sam.c
@@ -269,7 +269,6 @@ _PUBLIC_ NTSTATUS authsam_make_server_info(TALLOC_CTX *mem_ctx, struct ldb_conte
struct dom_sid *account_sid;
struct dom_sid *primary_group_sid;
const char *str;
- struct ldb_dn *ncname;
int i;
uint_t rid;
TALLOC_CTX *tmp_ctx = talloc_new(mem_ctx);
@@ -358,10 +357,10 @@ _PUBLIC_ NTSTATUS authsam_make_server_info(TALLOC_CTX *mem_ctx, struct ldb_conte
server_info->allow_password_change
= samdb_result_allow_password_change(sam_ctx, mem_ctx,
- ncname, msg, "pwdLastSet");
+ domain_dn, msg, "pwdLastSet");
server_info->force_password_change
= samdb_result_force_password_change(sam_ctx, mem_ctx,
- ncname, msg);
+ domain_dn, msg);
server_info->logon_count = samdb_result_uint(msg, "logonCount", 0);
server_info->bad_password_count = samdb_result_uint(msg, "badPwdCount", 0);
diff --git a/source4/dsdb/schema/schema_set.c b/source4/dsdb/schema/schema_set.c
index dcaeb4f..b949935 100644
--- a/source4/dsdb/schema/schema_set.c
+++ b/source4/dsdb/schema/schema_set.c
@@ -86,6 +86,11 @@ static int dsdb_schema_set_attributes(struct ldb_context *ldb, struct dsdb_schem
goto op_error;
}
+ ret = ldb_msg_add_string(msg_idx, "@IDXONE", "1");
+ if (ret != LDB_SUCCESS) {
+ goto op_error;
+ }
+
for (attr = schema->attributes; attr; attr = attr->next) {
const char *syntax = attr->syntax->ldb_syntax;
diff --git a/source4/lib/ldb/common/ldb_dn.c b/source4/lib/ldb/common/ldb_dn.c
index 402d629..6f462dd 100644
--- a/source4/lib/ldb/common/ldb_dn.c
+++ b/source4/lib/ldb/common/ldb_dn.c
@@ -846,14 +846,8 @@ const char *ldb_dn_get_casefold(struct ldb_dn *dn)
}
if (dn->comp_num == 0) {
- if (dn->linearized && dn->linearized[0] == '\0') {
- /* hmm a NULL dn, should we faild casefolding ? */
- dn->casefold = talloc_strdup(dn, "");
- return dn->casefold;
- }
- /* A DN must be NULL, special, or have components */
- dn->invalid = true;
- return NULL;
+ dn->casefold = talloc_strdup(dn, "");
+ return dn->casefold;
}
/* calculate maximum possible length of DN */
diff --git a/source4/lib/ldb/ldb_tdb/ldb_index.c b/source4/lib/ldb/ldb_tdb/ldb_index.c
index c99c293..db0c315 100644
--- a/source4/lib/ldb/ldb_tdb/ldb_index.c
+++ b/source4/lib/ldb/ldb_tdb/ldb_index.c
@@ -1121,14 +1121,10 @@ int ltdb_search_indexed(struct ltdb_context *ac)
if (ac->scope != LDB_SCOPE_BASE && idxattr == 1) {
ret = ltdb_index_dn(ac->module, ac->tree, ltdb->cache->indexlist, dn_list);
-
- if (ret != LDB_SUCCESS && ret != LDB_ERR_NO_SUCH_OBJECT) {
- talloc_free(dn_list);
- return ret;
- }
}
- if (ac->scope == LDB_SCOPE_ONELEVEL && idxone == 1) {
+ if (ret == LDB_ERR_OPERATIONS_ERROR &&
+ ac->scope == LDB_SCOPE_ONELEVEL && idxone == 1) {
ret = ltdb_index_dn_one(ac->module, ac->base, dn_list);
}
diff --git a/source4/lib/ldb/ldb_tdb/ldb_search.c b/source4/lib/ldb/ldb_tdb/ldb_search.c
index d395c28..b307c5f 100644
--- a/source4/lib/ldb/ldb_tdb/ldb_search.c
+++ b/source4/lib/ldb/ldb_tdb/ldb_search.c
@@ -546,6 +546,13 @@ int ltdb_search(struct ltdb_context *ctx)
* callback error */
if ( ! ctx->request_terminated && ret != LDB_SUCCESS) {
/* Not indexed, so we need to do a full scan */
+#if 0
+ /* useful for debugging when slow performance
+ * is caused by unindexed searches */
+ char *expression = ldb_filter_from_tree(ctx, ctx->tree);
+ printf("FULL SEARCH: %s\n", expression);
+ talloc_free(expression);
+#endif
ret = ltdb_search_full(ctx);
if (ret != LDB_SUCCESS) {
ldb_set_errstring(ldb, "Indexed and full searches both failed!\n");
--
Samba Shared Repository
More information about the samba-cvs
mailing list