[PATCH] WIP: indexing for <= and >=

[Garming Sam]

On 20/03/19 5:30 PM, Aaron Haslett via samba-technical wrote:
> Here's version 2 of <= and >= indexing patch:
>
>   * bug fixes
>   * bumping index version to trigger reindex
>   * new LDB version due to adding index_format_fn field to schema syntax
>     structure
>
> Testing was done on a 300,000 record DC with and without the patch to
> find what speedup the patch achieved:
>
>   * Join time reduced from 3186s to 1446s

This improvement is using the changes to the in-memory tdb size that
Gary is talking about in "Re-index performance - default tdb sizes." --
but that is a pretty noticeable improvement. We need to do some more
testing of just this patch (or Gary's), and also what effect this has on
ongoing full synchronizations.

>   * Manual replication call with no data replicated reduced from 4s to 1s
>   * Manual replication call with 200 record replicated reduced from 6.5s
>     to 2s

To improve the performance from here, there likely needs to be more work
done on the rest of the replication call chain. (Around 0.2 seconds of
the call is in the actual GUID scan search, although there are still
some ways we can improve this time too by avoiding allocations of DNs).

Cheers,

Garming

>
> On 19/03/19 2:12 PM, Aaron Haslett via samba-technical wrote:
>> Sorry, I forgot to say why we're doing this.  We want to be able to
>> greatly speed up "uSNChanged >= x" queries which are used during
>> replication.  Testing on a DC with 300,000 records showed a speed up
>> from ~4s to ~0.2s.  The patch only supports integers on LMDB for now.
>>
>> On 19/03/19 12:42 PM, Aaron Haslett via samba-technical wrote:
>>> Work in progress implementation of <= and >= indexing.  Adds
>>> iterate_range to LDB API and uses it for indexing. Modifications to come
>>> because right now this patch modifies a core LDB type which will
>>> probably break things.  I'll change it soon to add a new ordered integer
>>> type with a new OID.
>>>
>>>
>>> CI: https://gitlab.com/catalyst-samba/samba/pipelines/52460313
>>>