[PATCH] GUID index for LDB

[Andrew Bartlett]

On Tue, 2017-09-12 at 02:57 +0200, Stefan Metzmacher wrote:
> Hi Andrew,
> 
> > > So we're upgrading the database on first use with the new code?
> > > 
> > > My fear with this is that a simple package upgrade will make
> > > a dc with a large database unusable for quite some time.
> > > 
> > > Can you please check the cost of an upgrade for databases with
> > > 1.) 5000 users, 5000 computers and 5000 groups
> > > 2.) 20000 users, 20000 computers and 20000 groups
> > > 3.) with the numbers of the largest known customer size
> > 
> > With 100k users in 1-4 groups each, it takes 3min30sec on my
> > desktop.
> > 
> > Oddly, the undo operation (just wrote a downgrade script in case it
> > is
> > needed) takes 11mins. 
> > 
> > The patches I've got to only re-index once per change to the index
> > helps a lot, naturally :-). 
> 
> Ok, that's relatively fast and I guess for most current installations
> it would under 30 seconds, which would be fine I guess.

I agree. 

> Can you also test this with factor 10 and 100 smaller, so 10000 and
> 1000
> users, to be sure.

1000 users is practically instant, the schema DB is larger. 

I had a 27000 user DB handy (no groups), and it re-indexed in 38 second
including the time to dump the DB to the console (oops :-). 

This isn't going to be a big deal for our current use cases, but will
open us up to much larger opportunities. 

> > > I guess rewriting the whole database consumes quite some cpu
> > > and also memory. A server may run out of memory while doing this
> > > as we need more than twice the size of all sam.ldb* databases
> > > together.
> > 
> > We only hold the index in memory, the rest could get paged out, but
> > I
> > agree that running on a server with 2x DB size in free memory would
> > avoid thrashing. 
> 
> Don't we also hold the transaction preparation area in memory before
> we write the recovery area and write the changes to the file?

I'm not sure, I assumed it was in that record, but I'm not going to
open up that can of worms anyway.  Samba 4.7 and above needs memory, as
noted in the release notes.

> > I agree a DB without a big transaction area already and perfectly
> > packed seems the worst possible case.  I gave up measuring the undo
> > run, but the upgrade case only took 4mins, probably because the
> > index
> > records get to re-use their place in the DB.  (The new GUID index
> > is
> > smaller, indeed this seems to be the primary advantage). 
> > 
> > The (100k user) database does grow from 800MB to 2700MB. 
> 
> That's more than factor 3 and may mean /var gets full during this :-(
> 
> It would be really good to have this optimized somehow,
> maybe using more than one transaction.
> 
> But that's really tricky, it would be ideal if we could
> somehow truncate the file size with tdb_repack().
> 
> I would be good to make the worse case to use 2 of the original size
> (because of the recovery area) and also truncate the recovery area
> on a successful commit, if it's at the end of the file (in case the
> file got just expanded to hold the recovery area.
> 
> If we notice that the recovery area will be more than 25% of the
> database size we should do an implicit repack and make sure the
> recovery area is at the end and can be truncated after the
> commit finished. Which the transaction lock held we should
> be able to at least truncate back to the original file size,
> as no other caller had a chance to remap the file while the
> transaction lock is held.

I'm very sorry, but I don't have the time to rework tdb for an extreme
case well beyond any known or foreseeable installation.  It is all in a
transaction, if it fails it will just fail.

I have a downgrade script in any case.  

In the long term is is quite likely the main DB will be moved to LMDB,
as the folks wanting 100k users don't like being anywhere near that 4GB
limit, so re-working TDB for this is pointless.

Sadly we didn't accept TDB2 for Samba.

> > Process resident size was around 2.3GB (from memory). 
> > 
> > tdbtool output from all the DBs in sam.ldb.d attached. 
> > 
> > Given all this, and the new output being printed, I think it is
> > reasonable to do this on first use for Samba 4.8, with adequate
> > warning
> > in the WHATSNEW.txt
> 
> What do you mean with "the new output being printed"?

In my current branch I've added some incremental output during the re-
index.

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 862 bytes
Desc: This is a digitally signed message part
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170912/7aa38a2b/signature.sig>