[Samba] authn timeouts enumerating (and connecting to) shares

[Aaron Johnson]

Hello Samba users!

I’m experiencing an odd (hopefully, it’s odd to everyone and not just me) issue with Alma Linux 8.6’s samba-4.15.5-8.el8_6.x86_64 (and related) release.

In short, I have a domain member Samba server with just the magic [homes] share defined in smb.conf.  Mildly sanitized “testparm -s” output:

Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_DOMAIN_MEMBER

# Global parameters
[global]
                ldap connection timeout = 3
                ldap timeout = 3
                load printers = No
                log file = /var/log/samba/%m.log
log level = kerberos:10 auth:10 auth_audit:10 winbind:10
                ntlm auth = ntlmv1-permitted
                printcap name = /dev/null
                realm = MYDOMAIN.MYORG.COM
                security = ADS
                server role = member server
                winbind max domain connections = 10
                workgroup = MYDOMAIN
                idmap config MYDOMAIN : range = 100000-9999999
                idmap config MYDOMAIN : schema_mode = rfc2307
                idmap config MYDOMAIN : backend = ad
                idmap config * : range = 0-99999
                idmap config * : backend = tdb

[homes]
                browseable = No
                comment = Home Directories
                inherit acls = Yes
               read only = No
                valid users = %S %D%w%S

(I’ve added the “log level” setting in there as testparm didn’t print it.)

Trying to list out any shares on this server results in an NT_STATUS_IO_TIMEOUT like so:

[myuser at myserver ~]$ time smbclient -d 2 -U MYDOMAIN\\myuser -L myserver.myorg.com
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
added interface eth0 ip=10.10.10.10 bcast=10.10.11.255 netmask=255.255.252.0
tdb(/var/lib/samba/lock/gencache.tdb): tdb_open_ex: could not open file /var/lib/samba/lock/gencache.tdb: Permission denied
Password for [MYDOMAIN\ajohnson1]:
session setup failed: NT_STATUS_IO_TIMEOUT

real        0m27.191s
user       0m0.040s
sys          0m0.034s
[myuser at myserver ~]$

Watching the logs, I can see that smbd sends a query to winbind which is promptly responded to with an NT_STATUS_OK:
[2022/08/08 14:52:25.779975, 10, pid=2686623, effective(0, 0), real(0, 0), class=winbind] ../../source3/winbindd/winbindd.c:805(process_request_done)
  process_request_done: [smbd(2742274):PAM_AUTH_CRAP]: NT_STATUS_OK
[2022/08/08 14:52:25.780085, 10, pid=2686623, effective(0, 0), real(0, 0), class=winbind] ../../source3/winbindd/winbindd.c:849(process_request_written)
  process_request_written: [smbd(2742274):PAM_AUTH_CRAP]: delivered response to client
[2022/08/08 14:52:30.888462,  5, pid=2686623, effective(0, 0), real(0, 0), class=winbind] ../../source3/winbindd/winbindd_dual.c:856(winbind_child_died)
  Already reaped child 2742291 died

Smbd then seems to do nothing with that for 2 minutes:
[2022/08/08 14:54:32.008739, 10, pid=2741857, effective(0, 0), real(0, 0), class=auth] ../../source3/auth/auth_util.c:1924(check_account)
  check_account: Failed to find authenticated user MYDOMAIN\myuser via getpwnam(), fallback to sid_to_uid(S-1-5-21-1632765165-691681574-1546849883-1185380).
[2022/08/08 14:54:32.009822,  3, pid=2741857, effective(0, 0), real(0, 0), class=auth] ../../source3/auth/auth.c:269(auth_check_ntlm_password)
  auth_check_ntlm_password: winbind authentication for user [myuser] succeeded
[2022/08/08 14:54:32.010332,  5, pid=2741857, effective(0, 0), real(0, 0), class=auth] ../../source3/auth/auth.c:296(auth_check_ntlm_password)
  check_ntlm_password:  PAM Account for user [myuser] succeeded
[2022/08/08 14:54:32.010480,  3, pid=2741857, effective(0, 0), real(0, 0), class=auth_audit] ../../auth/auth_log.c:653(log_authentication_event_human_readable)
  Auth: [SMB2,(null)] user [MYDOMAIN]\[myuser] at [Mon, 08 Aug 2022 14:54:32.010447 MST] with [NTLMv2] status [NT_STATUS_OK] workstation [MYSAMBASERVER] remote host [ipv4:10.10.10.10:48880] became [MYDOMAIN]\[myuser] [S-1-5-21-1632765165-69168157
4-1546849883-1185380]. local host [ipv4:10.10.10.10:445]
[2022/08/08 14:54:32.010573,  2, pid=2741857, effective(0, 0), real(0, 0), class=auth] ../../source3/auth/auth.c:330(auth_check_ntlm_password)
  check_ntlm_password:  authentication for user [myuser] -> [myuser] -> [myuser] succeeded
[2022/08/08 14:54:32.011362, 10, pid=2741857, effective(0, 0), real(0, 0), class=auth] ../../auth/gensec/gensec.c:456(gensec_update_send)
  gensec_update_send: ntlmssp[0x563bc7f52c70]: subreq: 0x563bc7f43740

And smbclient has long since given up on getting a response.

Does anyone out there have any ideas why the 2 minute delay is happening?  I’d really love to get this working correctly – we’d like to retire of all our proprietary appliance based filers and move to clustered Samba with a CephFS backend.  As you might imagine, having clients unable to authenticate is a pretty big road block in that right now.

Thanks in advance.