s4 ldb tdb limits

[Oliver Liebel]

Am 26.08.2015 um 04:34 schrieb Andrew Bartlett:
> On Tue, 2015-08-25 at 09:34 -0700, Jeremy Allison wrote:
>> On Tue, Aug 25, 2015 at 01:52:21PM +0200, Oliver Liebel wrote:
>>> Hi,
>>>
>>> i have a few questions regarding S4 Performance, Limits,
>>> and maybe upcomping patches / workarounds / new strategies in the
>>> near future.
>>>
>>> I have three large size companys, who would like to switch to S4,
>>> but they all have more than 400-500.000 Objects in their DIT.
>>>
>>> After intensive Testing and research (latest 4.2 stable and 4.3 rc
>>> fresh compiled,
>>> Centos 7.1, real Hardware, 6 Cores, 32 GB RAM, only SSDs,
>>> S4-ldb-Files on 8GB RAMDisk),
>>> there is actually -from my actual point of view- no chance do get
>>> this job done with S4,
>>> (but maybe i've missed something):
>>>
>>> 1)
>>> Extremly poor Performance of the LDB/TDB Database Stack:
>>> Initial Bulk load of 350.000 small User-Objects (LDIF, with
>>> unicodePwd)
>>> took almost roundabout 6 hours (with sam.ldb*s on RAMDisk),
>>> (nested) groups complete left aside.
>>> Nearly same time amount with direct Bulk load into sam.ldb, without
>>> Protocol Overhead.
>>>
>>>   2)
>>> Absolute Showstopper:
>>> 4GB Limit of the LDB/TDB Stack, caused by the 32bit TDB internals.
>>> (Last 64 Bit TDB Patch is from S3 and the year 2005).
>>> This Limit will be reached with roundabout 350-400.000 Objects.
>>>
>>> 3)
>>> Threading Model not working. 'samba -M thread'
>>> doesnt start at all (default compile Settings).
>>>
>>> 4)
>>> Only 1 CPU Working with high load on import, no matter if there are
>>> several ldbadd-Processes
>>> (every ldbadd Process explicit dedicated to a different CPU per
>>> taskset).
>>>
>>>
>>>
>>> For some of the above mentioned points,
>>>   i've tried the samba-ldb-mdb branch from Jakub Hrozek in a short
>>> test,
>>> but Performance seems to be the same, limits not fully tested yet.
>>>
>>>
>>> Are there
>>> -any Ideas / future plans to speed up the Performance?
>>> -any Plans for a 64Bit TDB Patch to get rid of the 4G Limit?
>>> -any chances, that Howard's LMDB will be available in the next
>>> future
>>> as an alternative Key/Value Backend for the sam.ldb* Databases?
> Oliver,
>
> I'm glad to hear of interest in taking Samba to this scale.  I have
> discussed similarly with other interested parties who which to scale
> Samba up.  Sadly other than the initial efforts on the OpenLDAP backend
> backed by Symas, and an lmdb prototype from Red Hat, nobody else has
> stepped up with the engineering resources or customer funding at the
> level required.
>
> Even if we got the best database under the hood, many other aspects of
> the system would need work.
>
> The good news is that at this scale, the engineering effort is
> reasonable compared to the cost of a Windows CAL per user, so
> interested organisations still stand to both make savings and improve
> their freedom.
>
>> Yeah, lmdb and backending with OpenLDAP is our
>> long term solution for these limits I think.
>>
>> Nadia, any progress report on migrating us over ?
>>
> 	
>
> Jeremy,
>
> I should caution you that while moving to lmdb rather than tdb may be
> reasonably practical (a prototype has been developed so far), the task
> of moving to using OpenLDAP is I feel an order of magnitude larger than
> the task to move us to using MIT Kerberos.  Equally, when finished it
> promises great improvements, but we should be very clear that it needs
> commensurate resources.
>
> The issue is this:
>
> At it's heart, Samba4 turned out to be a series of RPC services
> clustered around the LDB module stack.  The vast majority of the
> complexity is in that stack.   As I understand it, the OpenLDAP backend
> project seeks to 'simply' replace that stack, using a number of Samba
> libraries in the process.
>
> I hope this helps give an idea of the scale involved here.
>
> Thanks,
>
> Andrew Bartlett

I fully agree with Andrew.

 From my point of view (and that of many customers) there are at this point
2 major tasks to get S4 into larger/enterprise scale:

- Fast (LM)DB Backend
- W2K12 DC compatibility

I know for sure how big the task is, to get OpenLDAP as an
S4 Backend with full Schema Semantics and DRSUAPI or any kind of DRS
working, as Andrew and I already worked a few years ago on the 
S4/OpenLDAP Backend.

And i would love to see OpenLDAP as an S4 Backend, but from a realistic 
point of
view, this task can't be finished very shortly, because its really complex.


I think in the meantime it is the best approache,
to get the LDB/TDB Stack enhanced / replaced with LMDB,
so that large(r) scale Installations could be satisfied.
We should get ' the best Database under the hood ' working. The quicker, 
the better.

One side effect: When the S4/OpenLDAP Job is done, we got already a mostly
compatible LMDB-Backend inside S4 working.


@Andrew, Nadezhda and Jakub:
could you please submitt  a full description / explanation  of the minor 
tasks
to get TDB replaced with LMDB.
I will try to get some hr, to get this done (but no promises,
i'll keep you informed)



W2K12 DC compatibility is another very important point ro talk about.
i know, it's just not adding schema extensions,  but it's an important
task to finish, to get S4 more widespread adapted by customers.
We should talk about that in another thread.



Thanks,
Oliver