Samba4: LDB size limit and memory leak

[tridge at samba.org]

Hi Marcel,

The following patch reduces the problem a bit by repacking the
database whenever it is expanded inside a transaction.

It still grows far more than it should though. Maybe we need to take a
fresh look at the ldb indexing approach. The killer is the indexes
like objectClass, which consists of a record with a key of
DN=@INDEX:OBJECTCLASS:USER that contains a list of every DN that is a
user object. That record grows by one entry for each new user, which
is the worst case for our free space allocation code in tdb. It also
means that for large N, adding N new users is O(N^2), as for each user
we need to re-write a record of size proportional to N. That is not
good.

Does anyone have any suggestions for a better indexing strategy?

Cheers, Tridge


--- a/lib/tdb/common/transaction.c
+++ b/lib/tdb/common/transaction.c
@@ -122,6 +122,9 @@ struct tdb_transaction {
 
 	/* old file size before transaction */
 	tdb_len_t old_map_size;
+
+	/* we should re-pack on commit */
+	bool need_repack;
 };
 
 
@@ -392,6 +395,8 @@ static int transaction_expand_file(struct tdb_context *tdb, tdb_off_t size,
 		return -1;
 	}
 
+	tdb->transaction->need_repack = true;
+
 	return 0;
 }
 
@@ -965,6 +970,7 @@ int tdb_transaction_commit(struct tdb_context *tdb)
 {	
 	const struct tdb_methods *methods;
 	int i;
+	bool need_repack;
 
 	if (tdb->transaction == NULL) {
 		TDB_LOG((tdb, TDB_DEBUG_ERROR, "tdb_transaction_commit: no transaction\n"));
@@ -1056,10 +1062,16 @@ int tdb_transaction_commit(struct tdb_context *tdb)
 	utime(tdb->name, NULL);
 #endif
 
+	need_repack = tdb->transaction->need_repack;
+
 	/* use a transaction cancel to free memory and remove the
 	   transaction locks */
 	tdb_transaction_cancel(tdb);
 
+	if (need_repack) {
+		return tdb_repack(tdb);
+	}
+
 	return 0;
 }