Re-index performance - default tdb sizes.

[Jeremy Allison]

On Wed, Mar 20, 2019 at 12:08:19PM +1300, Gary Lockyer via samba-technical wrote:
> With the expectation that it would make no difference I changed the size
> the in memory tdb used by re-indexing from 1000 to 100,000 and ran it
> against a 100,000 user domain.  Times went from 16 minutes 30 seconds to
> 44 seconds.
> 
> This does suggest that we should look at the default sizes we use to
> create our tdb databases with especially on larger domains.

Oh wow - that's a very interesting result !

Any chance you can get a range of results with different sizes
so we get more data ?

Maybe we can make this a tunable ?

Either way, fantastic work !

> diff is from my working tree, so line numbers may not line up with master.
> 
> diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c
> b/lib/ldb/ldb_key_value/ldb_kv_index.c
> index c80d94b2560..ee981d718e1 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv_index.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
> @@ -736,7 +736,7 @@ static int ldb_kv_dn_list_store(struct ldb_module
> *module,
> 
>         if (ldb_kv->idxptr->itdb == NULL) {
>                 ldb_kv->idxptr->itdb =
> -                   tdb_open(NULL, 1000, TDB_INTERNAL, O_RDWR, 0);
> +                   tdb_open(NULL, 100000, TDB_INTERNAL, O_RDWR, 0);
>                 if (ldb_kv->idxptr->itdb == NULL) {
>                         return LDB_ERR_OPERATIONS_ERROR;
>                 }
> 
> 
> Ngā mihi
> Gary
>