[Samba] Samba 4 and squid ntlm auth

[Andrew Bartlett]

On Thu, 2013-10-10 at 16:36 +0100, Julian Pilfold-Bagwell wrote:
> Hi List,
> 
> Looking for assistance with a squid authentication problem against Samba 4.
> 
> The squid proxy we're using worked fine on our old Samba 3 domain with 
> 500+ users but keeps freezing on our new Samba 4 domain.  I've joined 
> the proxy using net ads join and the samba 4 network is a clean build as 
> we wanted to leave any baggage from the old one behind.
> 
> What we now have is a situation where Samba 4 authenticates squid using 
> NTLM perfectly up until around 120 users are using it. Once we get above 
> 120, it starts to down and as we approach 140 it dies altogether.  At 
> this point, we restart samba and it works perfectly well for a period of 
> about 5 minutes with the 140+ users connected at which point it will 
> either slow to a crawl then fall over or sometimes will just fall over.
> 
> The network has three Samba 4 Domain controllers.  replication works 
> across the three and at any given time, they are running at around 25% 
> CPU load and consuming around 500MB of RAM.  All three are 3GHz, quad 
> core Xeons with between 4 and 12GB of RAM.
> 
> The odd thing is that at no point when Samba seems to be hanging, do we 
> lose access to shares on our fileserver and I also have Owncloud 
> authenticating via a read only LDAP proxy which is caching.  The really 
> odd thing is that I'm not seeing any obvious messages on either squid, 
> the samba 3 install or the DCs that points towards any major problem.  
> Given the numbers issue, I thought maybe I was hitting a ulimit wall but 
> the hard and soft limits are both unlimited.
> 
> Does anyone have a similar setup and any info on where to go from here, 
> i.e. which logs to check, etc.?
> 
> The OS details are as follows:
> 
> DC1 and DC1 - centos 6.4 Samba 4.0.10 (compiled from source) with 
> internal DNS
> DC3 - Debian Squeeze with Samba 4.0.10 (compiled from source) with Bind 
> 9.8 with dlz
> Squid proxy - Debian squeeze with Squid 2.7 Stable 9.2 from .deb package

My guess is that the single thread that is doing the lookups in the
sam.ldb and the subsequent authentication is choking on the constant
barrage of NTLM authentication traffic.

You might want to look into using kerberos, rather than NTLM
authentication, now you have an AD domain.  This will not need to place
load on the DC for each page load.

However, we should cope with lots of authentication, so if you have the
skill, running 'perf record -g PID' on the busy PID could be quite
illuminating, once analyzed with 'perf report -g'.  Please don't try and
mail me the perf.data output (it needs the build tree and symbols), but
examine it and tell me where the CPU is being used and what callers
responsible for it (screen-shots are OK in this specific instance). 

Also, just have a look at a wireshark trace of the success and failure
modes, and see if you can show a difference.  If the traces are not
massive, these you can mail to me.  Either way, the wireshark 'service
response time' over DCE/RPC would be particularly interesting to see. 

I hope this helps,

Andrew Bartlett

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Catalyst IT                   http://catalyst.net.nz