Progress on ldb_tdb bloat

[Andrew Bartlett]

Tridge and I have been working today on fixing the issues with ldb files
growing to massive sizes when new schema is added.  Getting Samba4 to
produce a 3gig tdb as you found is pretty impressive, but even a normal
provision shows this issue off pretty well.  We were able to test out a
few different behaviours with an ldbsearch/ldbadd, and have been able to
use this as a test case.

The problem is that when we re-index, we write out the index
incrementally as we traverse the ldb.  This ensures almost the worst
case for tdb's allocation strategy - it finds a space just big enough
and then when the next entry is added, it must find a new space to hold
the index. 

Already in the Samba GIT tree is a fix that repacks the ldb, ensuring
that the data is moved to the front of the file after a re-index.  This
should ensure that while the database can grow to 125MB under normal
circumstances, it should not continue to grow (as it will have a linear
120MB to work with while re-indexing next time). 

We are working on a scheme to perform the indexing in-memory during a
transaction, so that we write out the indexes just before the end of the
transaction, once we know their final size.  This should help a lot (at
the cost of holding all the indexes being touched in-memory). 

That said, it will still have a cost to re-index the database, so the
work you are looking to do, to ensure we do not re-index on schema
additions, will still be quite valuable. 

Thanks!

Andrew Bartlett
-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Red Hat Inc.                  http://redhat.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/samba-technical/attachments/20081216/c22376c4/attachment.bin