NTDB progress!

[Rusty Russell]

On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <obnox at samba.org> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
> 
> * To re-raise Volker's concern:
> 
>   You have been working on TDB2 for a long time. At least
>   a year or so. Now you seem to have overhauled the database
>   format and the code to some (if I am not mistaken) non-trivial
>   extent in a very short timeframe. That makes me worry too, that
>   this might not be very stable yet. And are the benchmarks now more
>   meaningful than the last ones.
> 
>   We are changing a core infrastructure component here.
>   The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year.  Which is
really crazy, since it's about four months of actual work.  The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug.  Gah!  So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
    backend) for beta2.  That's what I'm debugging now, though as you
    point out, s3/messaging and several s4 databases still need
    conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
    it (ie. foo.ntdb or foo.tdb).  Make any INTERNAL dbs use a dummy
    name if necessary (good practice anyway, since that name is good for
    log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
        if (!file_exists(ntdbname) && file_exists(tdbname)) {
                ... traverse and convert ...
                ... rename tdbname to tdbname.bak ...
                fail to open ntdb if something goes wrong here.
        }

(4) We can switch the callers based on an 'old tdb format' lpcfg.  This
    can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
    new ldb files will be created as ntdb unless 'old tdb format' is
    set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks!  Want to do s3/messaging and s4?

Thoughts?
Rusty.