[PATCH] Measure parallel performance and use standard process model in ldap

[Andrew Bartlett]

On Wed, 2017-04-05 at 14:22 +0200, Stefan Metzmacher via samba-
technical wrote:
> Am 05.04.2017 um 12:42 schrieb Andrew Bartlett:
> > On Wed, 2017-04-05 at 12:00 +0200, Stefan Metzmacher wrote:
> > > Am 05.04.2017 um 06:36 schrieb Andrew Bartlett via samba-
> > > technical:
> > > > The attached patches make our LDAP server multi-process.
> > > > 
> > > > This brings performance gains and losses.  Essentially if we
> > > > have
> > > > one
> > > > client binding and going away, strictly one at a time, then
> > > > there
> > > > is a
> > > > significant overhead.  However, with multiple clients we get to
> > > > use
> > > > multiple CPUs and it is worth it if there are multiple incoming
> > > > connections or the connections are long-lasting. 
> > > > 
> > > > We hope to fix up and restore the prefork process model in the
> > > > future,
> > > > but this isn't on the cards right now, and our users need the
> > > > extra
> > > > speed.
> > > > 
> > > > Please review/push the attached patch and the performance
> > > > tests. 
> > > 
> > > I guess this should be on top of the ldb/tdb patches?
> > 
> > I've tested it with and without, and it is viable both
> > ways.  Naturally
> > I want both in, the perf tests first ideally. 
> > 
> > > Is a server with these and the ldb/tdb patches able to handle
> > > a zarafa server which serves about 500-1000 users which make use
> > > of
> > > the webmail interface? We have a customer were we used a OpenLDAP
> > > proxy in front of the samba servers to handle the load.
> > > The proxy was able to deliver responses under 10 msecs,
> > > while samba needed about 350 msecs. The problem is that the proxy
> > > db gets corrupted every few days.
> > 
> > I've not tested it with that, but it was the thread from a similar
> > user
> > that made me pick this back up (it has been in my backlog for a
> > while).
> > 
> > What I will say is this.  On my workstation, the tests did:
> > 
> > test_500_concurrent_unindexed_search_10_workers(ad_dc_ntvfs)":
> > 94.185378
> > 
> > that is 180ms per search
> > 
> > With just this patch I get:
> > test_500_concurrent_unindexed_search_10_workers(ad_dc_ntvfs)":
> > 52.460954,
> > 
> > That is 100ms per search.  Still not 10ms, but then there is also a
> > bind in there.  Also still not scaling linearly with CPUs. 
> > 
> > With the ldb patch, we get:
> > test_500_concurrent_unindexed_search_10_workers(ad_dc_ntvfs)":
> > 13.458421,
> > 
> > That is 26ms per search, much closer to what we need. 
> > 
> > If I had a client suffering the consequences of the corrupted DB on
> > the
> > Zarafa proxy, I would want to try these patches :-)
> 
> Not in this form, I'll write a mail in the other thread...
> 
> Do you also have numbers for indexed searches?
> With a complex expressions like:
> 
> (&(objectClass=user)(|(cn=user1)(cn=user2)(cn=user3)(cn=user4)(cn=use
> r5)))
> or more complex things.

Later today I'll forward you the full set of results and links to the
scripts used, plus you can run 'make perftest' yourself any time you
like :-). 

We can also add new tests, as long as they are under a new name they
will just show up as additional lines on the graphs.  (We are trying
not to change the old ones, to allow consistent comparisons with
history). 

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba