Samba4: LDB size limit and memory leak

[tridge at samba.org]

Hi again,

The following patch seems to solve the problem, at least for this
test. It combines the "repack on expand" patch I sent previously with
a tiny change that over-allocates records by 25% in the main tdb space
allocation code.

The reason this works is that it adds more extra space to larger
records. So when our index records get large, they get 25% extra space
allocated which leaves them room to grow. So the number of times a
record is relocated when it grows drops rapidly as the database grows.

Watching a series of ldbadd calls to add lots of users, I now see the
freelist length staying roughly static, and the database growing by
about 7k per user. I'm still doing a larger run with 20k users, so
perhaps it will degrade eventually, but I doubt it. 

Can you test on your system and see if its OK for you?

This won't solve the speed problems (user and group adds are still too
slow), but at least the db stays a reasonable size.

Cheers, Tridge


diff --git a/lib/tdb/common/freelist.c b/lib/tdb/common/freelist.c
index 2f2a4c3..3bc3965 100644
--- a/lib/tdb/common/freelist.c
+++ b/lib/tdb/common/freelist.c
@@ -284,6 +284,9 @@ tdb_off_t tdb_allocate(struct tdb_context *tdb, tdb_len_t length, struct list_st
 	if (tdb_lock(tdb, -1, F_WRLCK) == -1)
 		return 0;
 
+	/* over-allocate to reduce fragmentation */
+	length *= 1.25;
+
 	/* Extra bytes required for tailer */
 	length += sizeof(tdb_off_t);
 	length = TDB_ALIGN(length, TDB_ALIGNMENT);
diff --git a/lib/tdb/common/transaction.c b/lib/tdb/common/transaction.c
index f5c04a6..e97fe67 100644
--- a/lib/tdb/common/transaction.c
+++ b/lib/tdb/common/transaction.c
@@ -122,6 +122,9 @@ struct tdb_transaction {
 
 	/* old file size before transaction */
 	tdb_len_t old_map_size;
+
+	/* we should re-pack on commit */
+	bool need_repack;
 };
 
 
@@ -392,6 +395,8 @@ static int transaction_expand_file(struct tdb_context *tdb, tdb_off_t size,
 		return -1;
 	}
 
+	tdb->transaction->need_repack = true;
+
 	return 0;
 }
 
@@ -965,6 +970,7 @@ int tdb_transaction_commit(struct tdb_context *tdb)
 {	
 	const struct tdb_methods *methods;
 	int i;
+	bool need_repack;
 
 	if (tdb->transaction == NULL) {
 		TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_transaction_commit: no transaction\n"));
@@ -1056,10 +1062,16 @@ int tdb_transaction_commit(struct tdb_context *tdb)
 	utime(tdb->name, NULL);
 #endif
 
+	need_repack = tdb->transaction->need_repack;
+
 	/* use a transaction cancel to free memory and remove the
 	   transaction locks */
 	tdb_transaction_cancel(tdb);
 
+	if (need_repack) {
+		return tdb_repack(tdb);
+	}
+
 	return 0;
 }