Samba at 100k users inc groups! (was: Re: Filtering links by the uptodateness vector, sorting them client-side (was: Re: [PATCH] drs exop test & refactoring))

[Andrew Bartlett]

On Wed, 2017-01-04 at 21:39 +1300, Andrew Bartlett wrote:
> On Thu, 2016-12-22 at 22:36 +1300, Andrew Bartlett wrote:
> > On Mon, 2016-12-19 at 23:48 +1300, Andrew Bartlett wrote:
> > 
> > I'm glad to say Bob's changes are in.  A patch set for the link
> > sorting
> > is at:
> > 

http://git.catalyst.net.nz/gw?p=samba.git;a=shortlog;h=refs/heads/ldb-sorted-links

> This work as continued, and it is almost finished, for the case of
> adding links.  Delete and replace are still inefficient however.

I've been running the numbers on the new patches by Douglas, in
particular seeing how many users we can add, including adding those
users to 4 groups, in a 2 hour time limit.

On master, we peak at 55,000 users in that two hour window.  However,
with Douglas' patch, we get to 55,000 users in just 50mins!

I'll know when my full test run completes - and probably find something
else blows up first, but on current projections it will be quite
practical to get 100,000 users into a samba DB, without excessive
waiting.

As a comparison, with Samba 4.4, we had trouble running this test up to
even 20,000 users, as too many steps in our handlers were O(n^2). 

This is why I'm still so keen to try and land these patches for 4.6 -
as finally, Samba as an AD DC really can scale!

(I actually think the patches are ready, after some squashing, and with
just the tests for my new compatibleFeatures/incompatibleFeatures
needed!)

Andrew Bartlett