[Samba] Unable to use more than 1000 concurrent ntlm_auth processes

[Michael Hendrie]

On 03/09/2012, at 4:04 PM, Andrew Bartlett <abartlet at samba.org> wrote:

> On Sat, 2012-08-18 at 23:03 +0930, Michael Hendrie wrote:
>> Hi List,
>> 
>> I'm running a heavily loaded squid server that uses ntlm_auth to provide NTLM authentication.
>> 
>> As load has increased over time, I've found the need to increase the
>> number of ntlm_auth processes available to squid as well as the
>> "winbind max clients" value in the smb.conf file.  This has worked
>> well up until now but seems I've hit some sort of limit.
>> 
>> If I keep the number of ntlm_auth processes under 1000, all is good.  Going above continually produces the messages below in /var/log/messages and the additional helpers unusable:
>> 
>> Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.342283,  0] utils/ntlm_auth.c:186(get_winbind_domain) 
>> Aug 16 22:34:17 prox (ntlm_auth):   could not obtain winbind domain name! 
>> Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.345335,  0] utils/ntlm_auth.c:186(get_winbind_domain) 
>> Aug 16 22:34:17 prox (ntlm_auth):   could not obtain winbind domain name! 
>> Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.353230,  0] utils/ntlm_auth.c:186(get_winbind_domain) 
>> Aug 16 22:34:17 prox (ntlm_auth):   could not obtain winbind domain name! 
>> Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.358237,  0] utils/ntlm_auth.c:186(get_winbind_domain) 
>> Aug 16 22:34:17 prox (ntlm_auth):   could not obtain winbind domain name!
>> 
>> And with winbindd log level on 9, /var/log/samba/winbindd.log shows:
>> 
>> [2012/08/16 22:33:42.352991,  6] winbindd/winbindd.c:768(new_connection)
>>  accepted socket 1032
>> [2012/08/16 22:33:42.359183,  6] winbindd/winbindd.c:768(new_connection)
>>  accepted socket 1036
>> [2012/08/16 22:37:59.337941,  2] winbindd/winbindd.c:710(winbind_client_response_written)
>>  Could not write response[14772:INTERFACE_VERSION] to client: Broken pipe
> 
>> Running distro supplied samba versions:
>> 
>> samba3x.x86_64:  3.5.10-0.110.el5_8
>> samba3x-common.x86_64:  3.5.10-0.110.el5_8
>> samba3x-winbind.x86_64:  3.5.10-0.110.el5_8
>> 
>> Does anyone have any suggestions on how to overcome this issue, I am happy to compile from source if there are any options that could help?
> 
> In relation to a similar query, it was suggested that with master (or a
> Samba 4.0 beta) you could set:
> 
> winbind max domain connections = <larger number than 1>
> 
> This might increase the throughput, and avoid the backlog getting to
> 1000.
> 
> I still think that you are hitting an OS limit somewhere

Yes seems to be…. Further testing has shown that it's not the number of ntlm_auth process, these can be well in excess of 1500 providing "winbind max clients" value is less than 1002 in my environment.

Using "lsof |grep winbindd" I see there is direct relation between "winbind max clients" and the number of connections to /path/to/winbindd_privileged/pipe and this would cause the error once FD 1023 was reached.  Would never see FD greater than 1023 even though /proc/winbindd_pid/limits reports 65535 for open files limit.

Compiled 3.6.7 from source (as standard 3.5.10 doesn't seem to recognise "winbind max clients" smb.conf option) and this has overcome my issue so seems that distro supplied winbindd package is restricted to 1024 FD regardless of ulimit setting.


> (perhaps on the
> total of the ntlm_auth children, rather than winbindd?), but having 1000
> outstanding authentications would be painful in any case.
> 
> Andrew Bartlett
> 
> -- 
> Andrew Bartlett                                http://samba.org/~abartlet/
> Authentication Developer, Samba Team           http://samba.org
> 
>