Re-index performance - default tdb sizes.

[Garming Sam]

Hi,

Note that this tdb is used for caching during transactions for LDB
(removing it entirely has major performance impacts). There might be
some TDB in winbind which might be using much smaller default sizes than
optimal. Certainly we've been seeing some issues with winbind with
timely responses at the 100k user scale, although it's not clear it
would be at this layer (but I have seen a decent amount of CPU work done
for the winbind caching).

(The other thing that would be useful information is how much memory
this consumes as you change this value)

Cheers,

Garming

On 20/03/19 12:08 PM, Gary Lockyer via samba-technical wrote:
> With the expectation that it would make no difference I changed the size
> the in memory tdb used by re-indexing from 1000 to 100,000 and ran it
> against a 100,000 user domain.  Times went from 16 minutes 30 seconds to
> 44 seconds.
>
> This does suggest that we should look at the default sizes we use to
> create our tdb databases with especially on larger domains.
>
>
> diff is from my working tree, so line numbers may not line up with master.
>
> diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c
> b/lib/ldb/ldb_key_value/ldb_kv_index.c
> index c80d94b2560..ee981d718e1 100644
> --- a/lib/ldb/ldb_key_value/ldb_kv_index.c
> +++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
> @@ -736,7 +736,7 @@ static int ldb_kv_dn_list_store(struct ldb_module
> *module,
>
>         if (ldb_kv->idxptr->itdb == NULL) {
>                 ldb_kv->idxptr->itdb =
> -                   tdb_open(NULL, 1000, TDB_INTERNAL, O_RDWR, 0);
> +                   tdb_open(NULL, 100000, TDB_INTERNAL, O_RDWR, 0);
>                 if (ldb_kv->idxptr->itdb == NULL) {
>                         return LDB_ERR_OPERATIONS_ERROR;
>                 }
>
>
> Ngā mihi
> Gary
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190320/71456b07/signature.sig>