Progress on ldb_tdb bloat

[Andrew Bartlett]

On Wed, 2008-12-17 at 08:34 +1100, tridge at samba.org wrote:
> Hi Andrew,
> 
>  > We are working on a scheme to perform the indexing in-memory during a
>  > transaction, so that we write out the indexes just before the end of the
>  > transaction, once we know their final size.  This should help a lot (at
>  > the cost of holding all the indexes being touched in-memory). 
> 
> I've now fixed up the bugs, and pushed that work to the main tree. The
> end result is that schema.ldb after a provision is about 2.8 MB on my
> box. Previously it had grown to 800 MB, which was getting quite
> silly. The users.ldb on my box dropped from 56 MB to 0.9 MB.

This is very impressive.

> I'll be interested to hear how this goes for other people. The change
> is quite tricky internally, particularly as it (temporarily) stores
> pointers into on-disk databases which is normally something that can't
> be done safely.
> 
> I also think there is more room for improvement, especially in terms
> of CPU usage. For example, to keep the patch as simple as possible we
> hooked into the load/store layer in ldb_index.c, whereas it would be
> more efficient to hook into the code that adds/remove an individual
> index value.

Indeed.  Adding an index record could be as simple as prefixing another
pointer to the index list. 

> I also expect that this change will reduce the runtime memory usage of
> ldb quite a lot, as the mmap will be much less sparse, which will
> reduce the number of instantiated pages a lot, and will also make for
> better use of cache.

Indeed. 

Andrew Bartlett

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Red Hat Inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/samba-technical/attachments/20081217/9c64e237/attachment.bin