Progress on ldb_tdb bloat

[tridge at samba.org]

Hi Andrew,

 > We are working on a scheme to perform the indexing in-memory during a
 > transaction, so that we write out the indexes just before the end of the
 > transaction, once we know their final size.  This should help a lot (at
 > the cost of holding all the indexes being touched in-memory). 

I've now fixed up the bugs, and pushed that work to the main tree. The
end result is that schema.ldb after a provision is about 2.8 MB on my
box. Previously it had grown to 800 MB, which was getting quite
silly. The users.ldb on my box dropped from 56 MB to 0.9 MB.

I'll be interested to hear how this goes for other people. The change
is quite tricky internally, particularly as it (temporarily) stores
pointers into on-disk databases which is normally something that can't
be done safely.

I also think there is more room for improvement, especially in terms
of CPU usage. For example, to keep the patch as simple as possible we
hooked into the load/store layer in ldb_index.c, whereas it would be
more efficient to hook into the code that adds/remove an individual
index value.

I also expect that this change will reduce the runtime memory usage of
ldb quite a lot, as the mmap will be much less sparse, which will
reduce the number of instantiated pages a lot, and will also make for
better use of cache.

Cheers, Tridge