Proposed API change to tdb.

[Rusty Russell]

On Wed, 22 Dec 2010 06:02:42 am simo wrote:
> On Tue, 2010-12-21 at 11:25 -0800, Jeremy Allison wrote:
> > Hi Rusty & friends,
> > 
> > I'd like to propose an API addition to tdb.
> > 
> > Currently, when we expand a tdb file on running out
> > of freelist, we have a heuristic that we follow that
> > states:
> > 
> > "always make room for at least 100 more records, and at
> > least 25% more space."
> > 
> > and then rounded up to a multiple of a page size.
> > 
> > I am working with an OEM that is running Samba on
> > a memory contrained box, and they are storing some
> > of the tdb's in an in-memory filesystem, to prevent
> > disk spin-up and consequent power drain.
> > 
> > The problem with the above heuristic is it creates
> > tdb files that are far too large for their box and
> > prevents them storing them on the ramfs.

Nothing wrong with this, but AFAICT you're missing the root cause.

Neither the 25% nor the 100 record heuristics are the problem; it's the
fact that we disabled right merging of freed blocks for performance.  This
means our tdbs get fragmented.  As a heuristic, we repack the entire db when
we expand inside a transaction.

So smaller expansions == more regular repacking.  Perhaps try turning the
current #ifdef USE_RIGHT_MERGES into a customisable parameter, and turn
that on instead?

On a related note, I have a patch to tdb which adds a "tdb_report" function
which allows more visibility into the state of the tdb.  My current patch
is overkill (ASCII graphs!  Woot!) but I'll get a reduced version ready
for submission...

Thanks!
Rusty.