[PATCH] WIP: indexing for <= and >=

Aaron Haslett aaronhaslett at catalyst.net.nz
Thu Mar 28 23:59:26 UTC 2019

Garming is right, my mistake.  Please ignore the join timing changes. 
Only the manual replication timing is relevant.

Here's version 3 of <= and >= indexing patch.  This version cleans up
the patches to submission standard, and increases the test coverage.

CI: https://gitlab.com/catalyst-samba/samba/pipelines/53976743

MR: https://gitlab.com/samba-team/samba/merge_requests/348

On 21/03/19 11:03 AM, Garming Sam wrote:
> On 20/03/19 5:30 PM, Aaron Haslett via samba-technical wrote:
>> Here's version 2 of <= and >= indexing patch:
>>   * bug fixes
>>   * bumping index version to trigger reindex
>>   * new LDB version due to adding index_format_fn field to schema syntax
>>     structure
>> Testing was done on a 300,000 record DC with and without the patch to
>> find what speedup the patch achieved:
>>   * Join time reduced from 3186s to 1446s
> This improvement is using the changes to the in-memory tdb size that
> Gary is talking about in "Re-index performance - default tdb sizes." --
> but that is a pretty noticeable improvement. We need to do some more
> testing of just this patch (or Gary's), and also what effect this has on
> ongoing full synchronizations.
>>   * Manual replication call with no data replicated reduced from 4s to 1s
>>   * Manual replication call with 200 record replicated reduced from 6.5s
>>     to 2s
> To improve the performance from here, there likely needs to be more work
> done on the rest of the replication call chain. (Around 0.2 seconds of
> the call is in the actual GUID scan search, although there are still
> some ways we can improve this time too by avoiding allocations of DNs).
> Cheers,
> Garming
>> On 19/03/19 2:12 PM, Aaron Haslett via samba-technical wrote:
>>> Sorry, I forgot to say why we're doing this.  We want to be able to
>>> greatly speed up "uSNChanged >= x" queries which are used during
>>> replication.  Testing on a DC with 300,000 records showed a speed up
>>> from ~4s to ~0.2s.  The patch only supports integers on LMDB for now.
>>> On 19/03/19 12:42 PM, Aaron Haslett via samba-technical wrote:
>>>> Work in progress implementation of <= and >= indexing.  Adds
>>>> iterate_range to LDB API and uses it for indexing. Modifications to come
>>>> because right now this patch modifies a core LDB type which will
>>>> probably break things.  I'll change it soon to add a new ordered integer
>>>> type with a new OID.
>>>> CI: https://gitlab.com/catalyst-samba/samba/pipelines/52460313
-------------- next part --------------
A non-text attachment was scrubbed...
Name: index_comparison_v3.patch
Type: text/x-patch
Size: 71366 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190329/9f8aca9e/index_comparison_v3.bin>

More information about the samba-technical mailing list