NTDB progress!

[Michael Adam]

Andrew Bartlett wrote:
> On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> > Hi all,
> > 
> >         Sorry this has been delayed: two things happened.  Firstly,
> > other duties involved me going to Hong Kong for a week.  Secondly,
> > porting revealed an unacceptable slowdown for smaller databases going
> > from tdb to tdb2, so after much benchmarking, the format was simplfied
> > to be closer to the original tdb.  See benchmarks below taken from that
> > commit message; we still pay a slight penalty for 64 bit.
> > 
> > See my ntdb-wip head:
> > 
> >         https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip
> 
> With beta2 scheduled for Tuesday, I'm wondering where we are at with
> landing this code?
> 
> The only comment I've seen on the code itself is Volker's comment
> concerned about the change you mention here.  It is a reasonable
> concern, but I'm not sure what we can do about it, given we already
> decided that the options for not using TDB2 also were not acceptable. 

Did we? I don't really remember that in this clearness. I know that what
started as TDB2 would be desirable for some scalability reasons in the
samba3 file server. And we said we want to try to get it into
shape for the release, but I don't remember that we said 4.0 can't live
without it. Or do I misunderstand you here?

Rusty, I looked at the code to some extent, and I also have a couple of
questions and comments:

* To re-raise Volker's concern:

  You have been working on TDB2 for a long time. At least
  a year or so. Now you seem to have overhauled the database
  format and the code to some (if I am not mistaken) non-trivial
  extent in a very short timeframe. That makes me worry too, that
  this might not be very stable yet. And are the benchmarks now more
  meaningful than the last ones.

  We are changing a core infrastructure component here.
  The change will affect virturall all parts of samba.

* We discussed at sambaXP, that we would like to have the choice
  to use classical tdb or ntdb, ideally on a per-database basis.
  My proposal was to use the dbwrap layer to realize this:

  - Adding a new dbwrap_ntdb backend in parallel to dbwrap_tdb.
  - Changing those users in s3 that still use tdb directly (or
    with tdb_wrap) to use dbrwap_tdb.
  - changing s4 components to access tdb files via dbwrap_tdb
  - and then being able to switch between tdb and ntdb
    without touching the callers.

  I see you have added a dbwrap_ntdb backend and a
  dbrwap_local_open function to open a db with tdb or ntdb
  depending on the configuration.
  Using the dbwrap_open_local in db_open() also changes
  all typical users of tdb via dbrwap in s3. So far so good!

  But the other users of tdb are converted directly,
  like s3-messaging (local messaging). My understanding was
  that this should be switchable, too. It is of course a
  tedious task, but dbwrap should be a nice means to do this.

  Speaking of s4 users, I am a littel surprised that these
  are also not switchable (but have some fallback to tdb).
  I would have expected to see (if not conversion to dbwrap),
  then e.g. a ldb_ntdb backend in parallel to ldb_tdb.
  So this seems to be a point of no return.
  I may be too naive here or might have misunderstood your
  patches.

I think we should be able to switch this new feature on.
Especially with the very recent fundamental changes.
Note, I do not intend to request impossible or silly things
here, I am just saying what my idea was how to move forward.
I might also help out in converting things to dbwrap if that
would be useful. What do others think? For the s4-part, I would
also like to hear the opinion of (among others) Andrew and Metze.

Cheers - Michael

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 206 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20120613/4dee433b/attachment.pgp>