[PATCH] Measure parallel performance and use standard process model in ldap

[Andrew Bartlett]

On Wed, 2017-04-05 at 14:22 +0200, Stefan Metzmacher wrote:
> Am 05.04.2017 um 12:42 schrieb Andrew Bartlett:
> > On Wed, 2017-04-05 at 12:00 +0200, Stefan Metzmacher wrote:
> > > Am 05.04.2017 um 06:36 schrieb Andrew Bartlett via samba-
> > > technical:
> > > > The attached patches make our LDAP server multi-process.
> > > > 
> > > > This brings performance gains and losses.  Essentially if we
> > > > have
> > > > one
> > > > client binding and going away, strictly one at a time, then
> > > > there
> > > > is a
> > > > significant overhead.  However, with multiple clients we get to
> > > > use
> > > > multiple CPUs and it is worth it if there are multiple incoming
> > > > connections or the connections are long-lasting. 
> > > > 
> > > > We hope to fix up and restore the prefork process model in the
> > > > future,
> > > > but this isn't on the cards right now, and our users need the
> > > > extra
> > > > speed.
> > > > 
> > > > Please review/push the attached patch and the performance
> > > > tests. 
> > > 
> > > I guess this should be on top of the ldb/tdb patches?
> > 
> > I've tested it with and without, and it is viable both
> > ways.  Naturally
> > I want both in, the perf tests first ideally. 
> > 
> > > Is a server with these and the ldb/tdb patches able to handle
> > > a zarafa server which serves about 500-1000 users which make use
> > > of
> > > the webmail interface? We have a customer were we used a OpenLDAP
> > > proxy in front of the samba servers to handle the load.
> > > The proxy was able to deliver responses under 10 msecs,
> > > while samba needed about 350 msecs. The problem is that the proxy
> > > db gets corrupted every few days.
> > 
> > I've not tested it with that, but it was the thread from a similar
> > user
> > that made me pick this back up (it has been in my backlog for a
> > while).
> > 
> > What I will say is this.  On my workstation, the tests did:
> > 
> > test_500_concurrent_unindexed_search_10_workers(ad_dc_ntvfs)":
> > 94.185378
> > 
> > that is 180ms per search
> > 
> > With just this patch I get:
> > test_500_concurrent_unindexed_search_10_workers(ad_dc_ntvfs)":
> > 52.460954,
> > 
> > That is 100ms per search.  Still not 10ms, but then there is also a
> > bind in there.  Also still not scaling linearly with CPUs. 
> > 
> > With the ldb patch, we get:
> > test_500_concurrent_unindexed_search_10_workers(ad_dc_ntvfs)":
> > 13.458421,
> > 
> > That is 26ms per search, much closer to what we need. 
> > 
> > If I had a client suffering the consequences of the corrupted DB on
> > the
> > Zarafa proxy, I would want to try these patches :-)
> 
> Not in this form, I'll write a mail in the other thread...

Just to be clear, is this a comment on the patches in *this* thread
(the perf tests and process model change) or the broader patches?

> Do you also have numbers for indexed searches?
> With a complex expressions like:
> 
> (&(objectClass=user)(|(cn=user1)(cn=user2)(cn=user3)(cn=user4)(cn=use
> r5)))
> or more complex things.

Do you need this before you review, or are you just interested?

As you know, the way LDB works, that isn't really much different to an
indexed search for one item, but how much is probably worth measuring
and Douglas plans to extend all the things in 'make perftest'

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 862 bytes
Desc: This is a digitally signed message part
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170406/4cb4f6bf/signature.sig>