[Samba] LDB object number limitations

[Andrew Bartlett]

On Wed, 2017-10-25 at 13:40 +0200, Ervin Hegedüs via samba wrote:
> Hi Andrew,
> 
> thanks for the reply,
> 
> On Wed, Oct 25, 2017 at 11:56:22PM +1300, Andrew Bartlett wrote:
> > On Wed, 2017-10-25 at 09:58 +0200, Ervin Hegedüs via samba wrote:
> > > 
> > > here is a Samba4 AD with LDB.
> > > 
> > > Is there any limit for number of possible objects? I mean users, computers,
> > > and other kind of nodes.
> > > 
> 
> [...]
> 
> > > May be an ugly question, but is that means the Samba4 supports "only" about
> > > 300k objects per db?
> > 
> > There has been and will continue to be work to make Samba scale much
> > further, both though incremental efforts (such as the recently
> > completed GUID index work), bug fixes (the locking fixes in 4.7 also
> > fixed a major bottleneck) and more radical steps.
> > 
> > As an example, with the git master current code I recently showed
> > adding 100,000 user objects to Samba, and adding those to either 1, 2,
> > 3 or 4 groups in 2 hours.  The bulk of that time was in the group
> > handling, interestingly the user insertion is no longer the dominant
> > factor.
> > 
> > Can you please give me some more details about what you propose to use
> > Samba for, so I can understand your concerns further?
> 
> We have to build a Samba AD, with several DC's.
> 
> The designed customer number is about 500 000, or more (user).
> 
> The main function of the AD is only the authentication, so there
> is a captive portal and some firewall system, user connects to
> the network (after auth, with 802.1x), and based on its group
> memberships it can use the internet connection.
> 
> The spped of each methods (add new user, modify its data, eg.
> password, group membership) is no matter. Only the read
> capability is important.
> 
> But I don't know after the two articles (descibed in my first
> e-mail) that should we store in all users in one domain, or we
> have to plan that split them to more domains.
> 
> And if it needs (to split them), how many objects is the maximum
> in an LDB?

So, if I were you and the business application can be easily changed to
shard the users into multiple domains I would do that.  Samba is
improving but 500,000 users is much above what anybody is know to have
deployed Samba with. 

Naturally, you should do your own testing with Samba master and see how
far you can push it, as each use case is a little different. 

My primary concern is that:
 - loading an index of objectclass=user would get increasingly slow
 - full scan of the database (still done occasionally, particularly for
replication) would be an issue.
 - it is likely that joining a new DC and replication would be quite
painful

Finally, depending on what is in the record it simply might not fit, as
there is a 4GB limit on tdb right now. 

For the size 64-bit aspect, I hope to see this situation change soon,
perhaps for Samba 4.9.  This isn't a promise yet, as it all depends on
funding coming though, but I want to mention this so you know there is
the potential that we will have a 64bit ldb re-based onto LDMB,
removing the 32 bit limitation. 

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba