[WIP] [PATCH] ldb: new on-disk pack format

[Andrew Bartlett]

On Wed, 2019-05-01 at 16:33 +1200, Garming Sam via samba-technical
wrote:
> Some initial testing shows that the new format has 50% more throughput
> (for unpacking group-like objects with a few thousand (member) values
> like what happens in the LDAP bind). Or in other words it's 33% faster
> to finish a single call. It also leads to much better memory performance
> under load, which is harder to predict in terms of overall improvement.
> 
> ldb_kv: Avoid memdup of database records in the case of base searches
> 
> a76d2865372988c29baef42ecc4257e861692e7b

Just a reminder to the list that we are still working on this.  It is
really important for the 100,000 user case, our performance tests are
unable to run (acceptably) without it, the responses just get too slow
on the large DB.

This is particularly due to the large number of user groups in that
example DB.

A set of patches to remove other inefficienies is in autobuild, but the
main change now almost ready (just working out the upgrade process for
example).

At this stage we are thinking to have GUID indexed databases use the
new pack format, with an upgrade on first use, exclusively.  This will
avoid changes to non-Samba LDB users while keeping the test matrix
compact.

Aaron will post some updated patches soon.

Thanks!

Andrew Bartlett
-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba