Re-index performance - default tdb sizes.

[Gary Lockyer]

Extra re-index timings attached below, but it looks like for databases
of around 5,000k users the current settings are ok.

Going forward, I'm inclined to add a batch_transaction_start to the ldb
api which will set the index hash size to the number of records in the
database.  LMDB supports this, and for TDB we could use map_size/100 as
an approximation.

On 20/03/19 12:27, Andrew Bartlett via samba-technical wrote:
> On Wed, 2019-03-20 at 12:08 +1300, Gary Lockyer via samba-technical
> wrote:
>> With the expectation that it would make no difference I changed the size
>> the in memory tdb used by re-indexing from 1000 to 100,000 and ran it
>> against a 100,000 user domain.  Times went from 16 minutes 30 seconds to
>> 44 seconds.
>>
>> This does suggest that we should look at the default sizes we use to
>> create our tdb databases with especially on larger domains.
> 
> In particular, I'm wondering if the default size used for the on-disk
> ldb (10,000), winbind's caches (at 10,000) are still appropriate.
> 
> I'm also quite concerned that many seem to have 0 passed as the hash
> size, which is defaulting to just 131 as I read it! 
> 
> Andrew Bartlett
>

Extra timings in mm:ss.ss

100,000 users approx 230,000 records.
  Hash size        re-index time
      1,000             14:42.06
     10,000              1:59.56
    100,000                39.92
    200,000                37.48
    300,000                43.16

50,000 users approx 110,000 records.
  Hash size        re-index time
      1,000              3:46:93
     10,000                37:29
    100,000                18.95

5,000 users approx 11,000 records
  Hash size        re-index time
      1,000                 3.43
     10,000                 2.24
    100,000                 2.00


Ngā mihi
Gary

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190320/b8413a95/signature.sig>