s4 ldb tdb limits

[Jeremy Allison]

On Tue, Aug 25, 2015 at 01:52:21PM +0200, Oliver Liebel wrote:
> Hi,
> 
> i have a few questions regarding S4 Performance, Limits,
> and maybe upcomping patches / workarounds / new strategies in the
> near future.
> 
> I have three large size companys, who would like to switch to S4,
> but they all have more than 400-500.000 Objects in their DIT.
> 
> After intensive Testing and research (latest 4.2 stable and 4.3 rc
> fresh compiled,
> Centos 7.1, real Hardware, 6 Cores, 32 GB RAM, only SSDs,
> S4-ldb-Files on 8GB RAMDisk),
> there is actually -from my actual point of view- no chance do get
> this job done with S4,
> (but maybe i've missed something):
> 
> 1)
> Extremly poor Performance of the LDB/TDB Database Stack:
> Initial Bulk load of 350.000 small User-Objects (LDIF, with unicodePwd)
> took almost roundabout 6 hours (with sam.ldb*s on RAMDisk),
> (nested) groups complete left aside.
> Nearly same time amount with direct Bulk load into sam.ldb, without
> Protocol Overhead.
> 
>  2)
> Absolute Showstopper:
> 4GB Limit of the LDB/TDB Stack, caused by the 32bit TDB internals.
> (Last 64 Bit TDB Patch is from S3 and the year 2005).
> This Limit will be reached with roundabout 350-400.000 Objects.
> 
> 3)
> Threading Model not working. 'samba -M thread'
> doesnt start at all (default compile Settings).
> 
> 4)
> Only 1 CPU Working with high load on import, no matter if there are
> several ldbadd-Processes
> (every ldbadd Process explicit dedicated to a different CPU per taskset).
> 
> 
> 
> For some of the above mentioned points,
>  i've tried the samba-ldb-mdb branch from Jakub Hrozek in a short test,
> but Performance seems to be the same, limits not fully tested yet.
> 
> 
> Are there
> -any Ideas / future plans to speed up the Performance?
> -any Plans for a 64Bit TDB Patch to get rid of the 4G Limit?
> -any chances, that Howard's LMDB will be available in the next future
> as an alternative Key/Value Backend for the sam.ldb* Databases?

Yeah, lmdb and backending with OpenLDAP is our
long term solution for these limits I think.

Nadia, any progress report on migrating us over ?

Cheers,

	Jeremy.