[WIP] [PATCH] ldb: new on-disk pack format

[Garming Sam]

Some initial testing shows that the new format has 50% more throughput
(for unpacking group-like objects with a few thousand (member) values
like what happens in the LDAP bind). Or in other words it's 33% faster
to finish a single call. It also leads to much better memory performance
under load, which is harder to predict in terms of overall improvement.

ldb_kv: Avoid memdup of database records in the case of base searches

a76d2865372988c29baef42ecc4257e861692e7b

The pack format specifically helps only after patch which is already in
master which avoids making a copy of a full database record, which in
the case of large groups might be megabytes large. In such a database,
the LDAP bind median performance does not actually appear that bad, but
under even moderate (or even small) load binds can blow out to seconds
long. With the new pack format, this should be more reliable and
predictable (at least until the next bottleneck). Still working on some
current numbers, but this was the observation during some PoC work.

To Aaron: One more thing is that looking at the patches, avoiding the
use of function pointers could be desirable given how hot this code is.
I think the overall approach is fine (and you don't necessarily need to
change it right now), but I don't know how much better or worse the
compiler is in these cases, where these functions might be better off
inlined. In theory, it's just a marginal improvement but maybe someone
else could comment (Douglas?).

Cheers,

Garming

On 30/04/19 2:25 PM, Aaron Haslett via samba-technical wrote:
> Garming discovered poor performance when recursively calculating group
> membership for a user during LDAP bind.  This WIP patch attempts to fix
> the problem by separating values from the rest of the data in our LDB
> pack format.  This should dramatically reduce the amount of data loaded
> into cache while unpacking with flag LDB_UNPACK_DATA_FLAG_NO_DATA_ALLOC.
>
> Correctness testing is included and a CI run is here:
>
> https://gitlab.com/samba-team/devel/samba/pipelines/59051539
>
> To be done:
>
>   * Performance testing
>   * Research into OpenLDAP's pack format and possible modifications to
>     ours based on theirs
>
>
> Thanks :)
>
> Aaron H.
>