[Samba] AD domain member cannot authenticate user in remote forest unless smbclient uses "localhost"

[Nathaniel W. Turner]

Hi folks,

I'm trying to support a customer with multiple AD forests, and during my
research, I've observed some odd behavior. In my lab tests, it seems like
authentication works for users in all trusted forests, but only if NTLMSSP
is used. When Kerberos ends up being used, authentication only seems to
work for users in the local domain.

Here's the test setup:
- Two Active Directory forests, tc83.local and tc84.local, with a forest
trust between them.
- The Linux server is a member of domain tc83.local.
- Samba built from git master this afternoon (commit 2669cecc51f) on Ubuntu
19.10. (I first reproduced this on CentOS 7, but wanted to test against
latest code before asking this list.)

ubuntu at kvm7246-vm022:~/samba$ sudo realm join --client-software=winbind
tc83.local
Password for Administrator:

ubuntu at kvm7246-vm022:~/samba$ realm list
tc83.local
  type: kerberos
  realm-name: TC83.LOCAL
  domain-name: tc83.local
  configured: kerberos-member
  server-software: active-directory
  client-software: winbind
  required-package: winbind
  required-package: libpam-winbind
  required-package: samba-common-bin
  login-formats: TC83\%U
  login-policy: allow-any-login

ubuntu at kvm7246-vm022:~/samba$ testparm
Load smb config files from //etc/samba/smb.conf
Loaded services file OK.
Server role: ROLE_DOMAIN_MEMBER

Press enter to see a dump of your service definitions

# Global parameters
[global]
kerberos method = system keytab
logging = systemd
realm = TC83.LOCAL
security = ADS
template homedir = /home/%U@%D
template shell = /bin/bash
winbind offline logon = Yes
winbind refresh tickets = Yes
workgroup = TC83
idmap config * : range = 10000-999999
idmap config * : backend = tdb


[test]
path = /srv/test
valid users = "@tc83.local\domain users" "@tc84.local\domain users"

Authentication works for a user in either forest when accessing the server
as "localhost", but fails for user in the remote forest when the real
hostname is used:

ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
administrator at tc83.local
Enter administrator at tc83.local's password:
Try "help" to get a list of possible commands.
smb: \> exit
ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
administrator at tc84.local
Enter administrator at tc84.local's password:
Try "help" to get a list of possible commands.
smb: \> exit
ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
administrator at tc83.local
Enter administrator at tc83.local's password:
Try "help" to get a list of possible commands.
smb: \> exit
ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
administrator at tc84.local
Enter administrator at tc84.local's password:
session setup failed: NT_STATUS_LOGON_FAILURE
ubuntu at kvm7246-vm022:~/samba$

(Logs from each smbclient attempt are at
https://drive.google.com/open?id=1_355NuN1L9BW5JvtP9WG-dEGkaQqNT3Y)

The logs seem to show that in the "localhost" cases, the final
authentication step uses "GENSEC submechanism gse_krb5", while in the cases
where the actual hostname is specified, the final authentication step uses
"GENSEC submechanism ntlmssp". The Kerberos auth seems only to work if the
authenticating user is in the local domain; if the user is in the other
domain, it fails looking for a keytab entry that does not exist:

Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.429043,  5]
../../auth/gensec/gensec_start.c:737(gensec_start_mech)
Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   Starting GENSEC submechanism
gse_krb5
Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.430349,  1]
../../source3/librpc/crypto/gse.c:659(gse_get_server_auth_token)
Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   gss_accept_sec_context failed
with [ Miscellaneous failure (see text): Failed to find
cifs/kvm7246-vm022 at TC84.LOCAL(kvno 10) in keytab MEMORY:cifs_srv_keytab
(aes256-cts-hmac-sha1-96)]

Is this expected behavior? A known issue? Am I doing something silly?