Fwd: Regression: ldb performance with indexes

[Andréas LEROUX]

Le 01/05/2024 à 02:42, Andrew Bartlett a écrit :
> On Wed, 2024-04-17 at 15:53 +0200, Andreas Schneider wrote:
>> On Friday, 22 March 2024 21:35:27 GMT+2 Andrew Bartlett via samba-technical
>> wrote:
>>> Firstly, these are very impressive improvements.
>> Hi Andréas,
>> I agree, this is great work.
>>   
>>> Thanks so much for your work debugging this and getting to the root of
>>> the problem, this is very much appreciated.
>>> Do you have any data on how much of the improvement is due to ldb
>>> patch, and how much is due to each of the other patches?
>>> Did you happen to use Brendan Greg's FlameGraph tool for the debugging
>>> (this is what we often use), and if so, can you share the graphs?  I
>>> want to understand if perhaps we need to consider restructuring the
>>> caller.
>>> https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions
>>> <https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions> 
>>>
>>> For this series if you could please:
>>>   - send in a send in a Samba Developer Declaration per
>>> https://www.samba.org/samba/devel/copyright-policy.html
>>> <https://www.samba.org/samba/devel/copyright-policy.html>
>>>   - create a gitlab account
>>>   - let me know the username
>>> Once I give you access to the devel repo, so you can run the full
>>> testsuite under our quota, please follow our contribution steps here:
>>> https://wiki.samba.org/index.php/Contribute#Subsequent_Merge_Requests_(and_c
>>> <https://wiki.samba.org/index.php/Contribute#Subsequent_Merge_Requests_(and_c> 
>>>
>>> omplex_first_requests)
>> I didn't see a merge request so far? Any update on that?
> Andréas,
> Thanks for creating a GitLab account and following up on our 
> paperwork. I can see your MR for the tdb_find patches, but it would be 
> awesome if you could raise one for the LDB in-memory-TDB change also.
>
> In the meantime Douglas is running a performance test on our test 
> server to confirm things, but it would be great to move this along so 
> our users can benefit.
>
> Make sure to include your Signed-off-by and
>
> BUG:https://bugzilla.samba.org/show_bug.cgi?id=15590
> Thanks,
> Andrew Bartlett

Hi,

Sorry for the time we are taking to make the MR for these patches, but 
we encountered something weird.

The LDB patch does indeed reduce the schema upgrade time on redhat 7/8/9 
(from about 3 days to one hour) and the LMDB one avoid getting 
MDB_TXN_FULL on large nested transaction. However, with the patches we 
attached the following repro.sh output showing great performances:

|[root at srvads1-bl1cw ~]# bash repro.sh 20000 indexes Added 2 records 
successfully Added 20000 records successfully
| real 0m0.536s user 0m0.798s sys 0m0.105s

But this speedup is only on centos 7. We tried the same patchset on 
alma8/9 but no performance change there and we can't find the reason. 
This improvement was a total side effect because our target was the 
schemaupgrade 2019 (LDB patch) and functionalprep 2016 on large domains 
(LMDB patch).
 From what I saw when debugging, the LDB patch does not affect the flow 
of the repro.sh script, neither does the LMDB patch. So I assume it 
comes from my tdb_find optimization but we can't tell why.

Understanding this may help increase the performances on other 
distributions and, more important, ensure it doesn't have bad side effects.

Thanks,
Andréas Leroux