s4 ldb tdb limits

[Nadezhda Ivanova]

Hi guys,
I've somehow missed this email, sorry! As you know, I've been engaged on 
another project for quite some time, longer that I expected, so no real 
progress has been made this year. The good news is, I'm getting back to 
it very soon. When I switch back the context I'll be able to give you 
more details.

Best Regards,
Nadya

On 08/25/2015 07:34 PM, Jeremy Allison wrote:
> On Tue, Aug 25, 2015 at 01:52:21PM +0200, Oliver Liebel wrote:
>> Hi,
>>
>> i have a few questions regarding S4 Performance, Limits,
>> and maybe upcomping patches / workarounds / new strategies in the
>> near future.
>>
>> I have three large size companys, who would like to switch to S4,
>> but they all have more than 400-500.000 Objects in their DIT.
>>
>> After intensive Testing and research (latest 4.2 stable and 4.3 rc
>> fresh compiled,
>> Centos 7.1, real Hardware, 6 Cores, 32 GB RAM, only SSDs,
>> S4-ldb-Files on 8GB RAMDisk),
>> there is actually -from my actual point of view- no chance do get
>> this job done with S4,
>> (but maybe i've missed something):
>>
>> 1)
>> Extremly poor Performance of the LDB/TDB Database Stack:
>> Initial Bulk load of 350.000 small User-Objects (LDIF, with unicodePwd)
>> took almost roundabout 6 hours (with sam.ldb*s on RAMDisk),
>> (nested) groups complete left aside.
>> Nearly same time amount with direct Bulk load into sam.ldb, without
>> Protocol Overhead.
>>
>>   2)
>> Absolute Showstopper:
>> 4GB Limit of the LDB/TDB Stack, caused by the 32bit TDB internals.
>> (Last 64 Bit TDB Patch is from S3 and the year 2005).
>> This Limit will be reached with roundabout 350-400.000 Objects.
>>
>> 3)
>> Threading Model not working. 'samba -M thread'
>> doesnt start at all (default compile Settings).
>>
>> 4)
>> Only 1 CPU Working with high load on import, no matter if there are
>> several ldbadd-Processes
>> (every ldbadd Process explicit dedicated to a different CPU per taskset).
>>
>>
>>
>> For some of the above mentioned points,
>>   i've tried the samba-ldb-mdb branch from Jakub Hrozek in a short test,
>> but Performance seems to be the same, limits not fully tested yet.
>>
>>
>> Are there
>> -any Ideas / future plans to speed up the Performance?
>> -any Plans for a 64Bit TDB Patch to get rid of the 4G Limit?
>> -any chances, that Howard's LMDB will be available in the next future
>> as an alternative Key/Value Backend for the sam.ldb* Databases?
> Yeah, lmdb and backending with OpenLDAP is our
> long term solution for these limits I think.
>
> Nadia, any progress report on migrating us over ?
>
> Cheers,
>
> 	Jeremy.
>