tdb and file-per-hash-chain

[tridge at samba.org]

James,

I can see how the file per hash chain will be better for some
situations than a plain tdb on a clustered filesystem, but I don't
think it will approach anything like what is needed to make a
clustered smbd really scalable. As soon as you get multiple nodes
hitting the same hash bucket you will be back with poor performance
again. That might be rare with netbench, but with loads that have
shared files it will be quite nasty.

I was talking to Brian Aker and Stewart Smith from MySQL at LCA'06
about this problem last week, and they pointed me at something called
'ndb' which is a a clustered database that apparently has a suitable
API to build a key-value pair database much like tdb. It's used to
build a clustered version of MySQL. Stewart told me the performance
numbers are in the ballpark of what we need for Samba. He sent me the
API docs, but I haven't had a chance to look through them yet.

Meanwhile, as Metze mentioned, it would certainly be worthwile to have
a look at the tdb work in Samba4. As part of the transactions work I
abstracted about some of the IO routines, so a transaction can
override the read/write functions. It isn't broken out by hash chain
like you have done, but at least some of the abstraction is done.

Cheers, Tridge