[PATCH] GUID index for LDB

[Stefan Metzmacher]

Hi Andrew,

>> So we're upgrading the database on first use with the new code?
>>
>> My fear with this is that a simple package upgrade will make
>> a dc with a large database unusable for quite some time.
>>
>> Can you please check the cost of an upgrade for databases with
>> 1.) 5000 users, 5000 computers and 5000 groups
>> 2.) 20000 users, 20000 computers and 20000 groups
>> 3.) with the numbers of the largest known customer size
> 
> With 100k users in 1-4 groups each, it takes 3min30sec on my desktop.
> 
> Oddly, the undo operation (just wrote a downgrade script in case it is
> needed) takes 11mins. 
> 
> The patches I've got to only re-index once per change to the index
> helps a lot, naturally :-). 

Ok, that's relatively fast and I guess for most current installations
it would under 30 seconds, which would be fine I guess.

Can you also test this with factor 10 and 100 smaller, so 10000 and 1000
users, to be sure.

>> I guess rewriting the whole database consumes quite some cpu
>> and also memory. A server may run out of memory while doing this
>> as we need more than twice the size of all sam.ldb* databases together.
> 
> We only hold the index in memory, the rest could get paged out, but I
> agree that running on a server with 2x DB size in free memory would
> avoid thrashing. 

Don't we also hold the transaction preparation area in memory before
we write the recovery area and write the changes to the file?

> I agree a DB without a big transaction area already and perfectly
> packed seems the worst possible case.  I gave up measuring the undo
> run, but the upgrade case only took 4mins, probably because the index
> records get to re-use their place in the DB.  (The new GUID index is
> smaller, indeed this seems to be the primary advantage). 
> 
> The (100k user) database does grow from 800MB to 2700MB. 

That's more than factor 3 and may mean /var gets full during this :-(

It would be really good to have this optimized somehow,
maybe using more than one transaction.

But that's really tricky, it would be ideal if we could
somehow truncate the file size with tdb_repack().

I would be good to make the worse case to use 2 of the original size
(because of the recovery area) and also truncate the recovery area
on a successful commit, if it's at the end of the file (in case the
file got just expanded to hold the recovery area.

If we notice that the recovery area will be more than 25% of the
database size we should do an implicit repack and make sure the
recovery area is at the end and can be truncated after the
commit finished. Which the transaction lock held we should
be able to at least truncate back to the original file size,
as no other caller had a chance to remap the file while the
transaction lock is held.

> Process resident size was around 2.3GB (from memory). 
> 
> tdbtool output from all the DBs in sam.ldb.d attached. 
> 
> Given all this, and the new output being printed, I think it is
> reasonable to do this on first use for Samba 4.8, with adequate warning
> in the WHATSNEW.txt

What do you mean with "the new output being printed"?

metze

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170912/d45d9ecc/signature.sig>