[Samba] samba4 ldap high load and port queue overflow

[Andrew Bartlett]

On Mon, 2015-11-23 at 16:50 +0300, Yuriy Tabolin wrote:
> Hi all.
> I have samba 4.2.3 on freebsd 10.1 server. There are three DC and
> about 
> 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some servers with
> services 
> like apache, exim, dovecot, etc use samba4 ldap (port 389) for user 
> authentication. Some times ago after adding some services to use ldap
> I 
> found, that samba4 cannot serve all ldap requests. Every 10-30
> minutes I 
> see in DCs logs:
> dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue overflow:
> 16 
> already in queue awaiting acceptance (28 occurrences)
> 
> After that I have used tcpdump for recording ldap-traffic and have
> seen 
> that after TCP handshaking, server some times suddenly send TCP-RST
> to 
> close connection. I have enlarged DCs resources (CPU and RAM),  
> kern.ipc.somaxconn, did some other system tuning but all that didn't 
> help. Load average on DCs permanently near 0.9-1.0 and samba cannot 
> serve all ldap conncetions. ldap clients works well because they use
> as 
> minimum as two domain controllers as ldap servers. Is there a some 
> performance problem in samba4, slow processing ldap requests or 
> something else?
> Thanks for any help.

G'Day,

Samba typically handles installations such as yours without much
difficulty, so I'm surprised.  

However, if your web applications hit the LDAP server hard, this could
certainly be an issue.  Samba's LDAP server isn't designed for high
performance - frankly we were glad when it worked at all - but works
very well for most of our deployments at your scale.  

We certainly could re-architect the LDAP server for better performance,
but at your scale it really shouldn't bit hitting limits.  Perhaps your
web applications are doing a lot of unindexed operations, or are in
-efficient in their authentication processing?  Try using PAM backed by
pam_winbind intead of LDAP for example, this would put authentication
though a different protocol, and not open a TCP socket per
authentication. 

I don't normally suggest changing platform, and if you use FreeBSD,
then I understand your passion for that platform, but you might also
wish to compare with a trial server running on Ubuntu, to see if there
is a platform-specific issue.

Finally, there certainly is hope for improved performance, but it needs
some backing from the users who need it most, either in engineering
effort or working with a commercial support vendor.  

Those of us involved in Samba day to day know there are many, many
things we can work on, we just haven't had the bandwidth or client
requirement to focus on it yet.  For example, until recently we had a
'prefork' process mode, that would allow for example once LDAP process
per port.  It hasn't ever been used in that combination, but interested
parties could put it back, add a test and see if it helps. 

More likely to help (but more effort) are merging the ldb_perfs branch,
profiling the rest of the code in search of performance issues,
replacing the ldb_tdb backend with ldb_lmdb.  

However, I would go back to looking at the sources of the queries - if
your DB is constantly under full-DB scan, it will very likely show
symptoms like you see. 

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba