s4 ldb tdb limits

[Andrew Bartlett]

On Tue, 2015-08-25 at 09:34 -0700, Jeremy Allison wrote:
> On Tue, Aug 25, 2015 at 01:52:21PM +0200, Oliver Liebel wrote:
> > Hi,
> > 
> > i have a few questions regarding S4 Performance, Limits,
> > and maybe upcomping patches / workarounds / new strategies in the
> > near future.
> > 
> > I have three large size companys, who would like to switch to S4,
> > but they all have more than 400-500.000 Objects in their DIT.
> > 
> > After intensive Testing and research (latest 4.2 stable and 4.3 rc
> > fresh compiled,
> > Centos 7.1, real Hardware, 6 Cores, 32 GB RAM, only SSDs,
> > S4-ldb-Files on 8GB RAMDisk),
> > there is actually -from my actual point of view- no chance do get
> > this job done with S4,
> > (but maybe i've missed something):
> > 
> > 1)
> > Extremly poor Performance of the LDB/TDB Database Stack:
> > Initial Bulk load of 350.000 small User-Objects (LDIF, with 
> > unicodePwd)
> > took almost roundabout 6 hours (with sam.ldb*s on RAMDisk),
> > (nested) groups complete left aside.
> > Nearly same time amount with direct Bulk load into sam.ldb, without
> > Protocol Overhead.
> > 
> >  2)
> > Absolute Showstopper:
> > 4GB Limit of the LDB/TDB Stack, caused by the 32bit TDB internals.
> > (Last 64 Bit TDB Patch is from S3 and the year 2005).
> > This Limit will be reached with roundabout 350-400.000 Objects.
> > 
> > 3)
> > Threading Model not working. 'samba -M thread'
> > doesnt start at all (default compile Settings).
> > 
> > 4)
> > Only 1 CPU Working with high load on import, no matter if there are
> > several ldbadd-Processes
> > (every ldbadd Process explicit dedicated to a different CPU per 
> > taskset).
> > 
> > 
> > 
> > For some of the above mentioned points,
> >  i've tried the samba-ldb-mdb branch from Jakub Hrozek in a short 
> > test,
> > but Performance seems to be the same, limits not fully tested yet.
> > 
> > 
> > Are there
> > -any Ideas / future plans to speed up the Performance?
> > -any Plans for a 64Bit TDB Patch to get rid of the 4G Limit?
> > -any chances, that Howard's LMDB will be available in the next 
> > future
> > as an alternative Key/Value Backend for the sam.ldb* Databases?

Oliver,

I'm glad to hear of interest in taking Samba to this scale.  I have
discussed similarly with other interested parties who which to scale
Samba up.  Sadly other than the initial efforts on the OpenLDAP backend
backed by Symas, and an lmdb prototype from Red Hat, nobody else has
stepped up with the engineering resources or customer funding at the
level required.  

Even if we got the best database under the hood, many other aspects of
the system would need work. 

The good news is that at this scale, the engineering effort is
reasonable compared to the cost of a Windows CAL per user, so
interested organisations still stand to both make savings and improve
their freedom. 

> Yeah, lmdb and backending with OpenLDAP is our
> long term solution for these limits I think.
> 
> Nadia, any progress report on migrating us over ?
> 
	

Jeremy,

I should caution you that while moving to lmdb rather than tdb may be
reasonably practical (a prototype has been developed so far), the task
of moving to using OpenLDAP is I feel an order of magnitude larger than
the task to move us to using MIT Kerberos.  Equally, when finished it
promises great improvements, but we should be very clear that it needs
commensurate resources. 

The issue is this:

At it's heart, Samba4 turned out to be a series of RPC services
clustered around the LDB module stack.  The vast majority of the
complexity is in that stack.   As I understand it, the OpenLDAP backend
project seeks to 'simply' replace that stack, using a number of Samba
libraries in the process.

I hope this helps give an idea of the scale involved here.

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba