s4 ldb tdb limits

[Oliver Liebel]

Hi,

i have a few questions regarding S4 Performance, Limits,
and maybe upcomping patches / workarounds / new strategies in the near 
future.

I have three large size companys, who would like to switch to S4,
but they all have more than 400-500.000 Objects in their DIT.

After intensive Testing and research (latest 4.2 stable and 4.3 rc fresh 
compiled,
Centos 7.1, real Hardware, 6 Cores, 32 GB RAM, only SSDs, S4-ldb-Files 
on 8GB RAMDisk),
there is actually -from my actual point of view- no chance do get this 
job done with S4,
(but maybe i've missed something):

1)
Extremly poor Performance of the LDB/TDB Database Stack:
Initial Bulk load of 350.000 small User-Objects (LDIF, with unicodePwd)
took almost roundabout 6 hours (with sam.ldb*s on RAMDisk),
(nested) groups complete left aside.
Nearly same time amount with direct Bulk load into sam.ldb, without 
Protocol Overhead.

  2)
Absolute Showstopper:
4GB Limit of the LDB/TDB Stack, caused by the 32bit TDB internals.
(Last 64 Bit TDB Patch is from S3 and the year 2005).
This Limit will be reached with roundabout 350-400.000 Objects.

3)
Threading Model not working. 'samba -M thread'
doesnt start at all (default compile Settings).

4)
Only 1 CPU Working with high load on import, no matter if there are 
several ldbadd-Processes
(every ldbadd Process explicit dedicated to a different CPU per taskset).



For some of the above mentioned points,
  i've tried the samba-ldb-mdb branch from Jakub Hrozek in a short test,
but Performance seems to be the same, limits not fully tested yet.


Are there
-any Ideas / future plans to speed up the Performance?
-any Plans for a 64Bit TDB Patch to get rid of the 4G Limit?
-any chances, that Howard's LMDB will be available in the next future
as an alternative Key/Value Backend for the sam.ldb* Databases?



Thanks in Advance


Oliver