Missing domain user tickets with winbind

[Shyam Prasad N]

Hi,

My name is Shyam Prasad. I work at Microsoft in the Azure Files team.
For the past few days, I've been working on getting the Azure Linux VMs to
join the AD domain in Azure, login as domain users, and mount Azure file
shares over SMB3.

Most things work fine. Except that I need perform a few Kerberos related
tasks manually, for the SMB3 mount to work with domain user credentials.
I did some debugging of the issue, and looks like cifs.upcall (the
userspace helper program for cifs.ko) is unable to find the krb5 TGT for
the domain user in the cred-cache. If the cred-cache is missing, it looks
for it in the system krb5.keytab.

Since winbind is configured with kerberos method "secrets and keytab", I
would expect either the secrets.tdb or the krb5.keytab to have an entry for
the domain user lxsmbadmin. Even with the domain user already logged in
through ssh, I'm unable to get those in both those places. cred-cache file
is not created in the first place.

With the domain user already logged in through ssh, I expected that the
kerberos TGT would already have been retrieved and stored locally.
Where does winbind store its Kerberos tickets, so that I can point
cifs.upcall to look there for tickets instead?

The mount only works when I use kinit to populate the cred-cache with the
domain user.

Any help in troubleshooting this issue is appreciated.

Also, I'm interested to know, how can I enable the debug logs in the
libkrb5 shared libraries that are built from the samba source code? I don't
see the debug logs in that code being logged, even if log level is set to
maximum in smb.conf.

Regards,
Shyam

=======================================================
Details of my setup:
I'm using an Ubuntu 19.10 server VM.
I'm mounting as the local root user, however, I'm using a domain user
credentials for mounting the using sec=krb5.
Below are my mount options:
vers=3.0,sec=krb5,credentials=/home/localadmin/.smb3credentials,serverino,noperm,nosharesock,mfsymlinks,uid=lxsmbadmin,gid='domain
users'

The VM is already joined to the AD domain aaddomain.example.com using
winbind.
This is what my smb.conf looks like for winbind:
localadmin at lxsmb-canvm13:~$ cat /etc/samba/smb.conf
[global]
   workgroup = AADDOMAIN
   security = ADS
   realm = AADDOMAIN.EXAMPLE.COM

   winbind refresh tickets = Yes
   vfs objects = acl_xattr
   map acl inherit = Yes
   store dos attributes = Yes

   dedicated keytab file = /etc/krb5.keytab
   kerberos method = secrets and keytab

   winbind use default domain = Yes

   load printers = No
   printing = bsd
   printcap name = /dev/null
   disable spoolss = Yes

   log file = /var/log/samba/log.%m
   log level = 10

   idmap config * : backend = tdb
   idmap config * : range = 3000-7999

   idmap config AADDOMAIN : backend = rid
   idmap config AADDOMAIN : range = 10000-999999

   template shell = /bin/bash
   template homedir = /home/%U

localadmin at lxsmb-canvm13:~$ cat /etc/krb5.conf
[libdefaults]
        default_realm = AADDOMAIN.EXAMPLE.COM
        dns_lookup_realm = false
        dns_lookup_kdc = true

Initially, I tried to use the ubuntu apt packages to install winbind and
related packages.
After going through a bit of code, I wanted to be able to print the debug
logs.
So I decided to install winbind from the latest source:
master branch on git://git.samba.org/samba.git

Here is the configure I used to build it:
./configure --with-systemd --bindir=/usr/bin --sbindir=/usr/sbin
--libdir=/usr/lib/x86_64-linux-gnu/samba --sysconfdir=/etc/samba
--localstatedir=/run/samba --includedir=/usr/include/
--datadir=/usr/share/samba --mandir /usr/share/man --enable-debug
--enable-developer --systemd-install-services
--with-systemddir=/usr/lib/systemd/system
--with-privatedir=/var/lib/samba/private --with-systemd --with-pam

After tweaking a few config files here and there, I've now reached the same
state as when I was running winbind from Ubuntu packages.
I'm now able to ssh/su as the domain user to this system.

However, I do not see the cred-cache populated.
localadmin at lxsmb-canvm13:~/samba$ sudo klist
klist: No ticket file: /tmp/krb5cc_0
localadmin at lxsmb-canvm13:~/samba$ ls /tmp/krb*
ls: cannot access '/tmp/krb*': No such file or directory

After a bit of code reading of cifs.upcall, it looks to me like the
expectation is that cred-cache would be populated for the domain user.
If in case the cred-cache is missing, then it creates a new cred-cache from
the keytab at /etc/krb5.keytab

So clearly, the expectation is that atleast the keytab is already
populated.

The kerberos method that I've chosen in smb.conf is "secrets and keytab".
So I expect either the secrets.tdb or the krb5.keytab to have an entry for
the domain user lxsmbadmin.
However, I do not see those entries in either of them:

localadmin at lxsmb-canvm13:~$ sudo tdbdump
/var/lib/samba/private/secrets.tdb|grep -i lxsmbadmin
localadmin at lxsmb-canvm13:~$

localadmin at lxsmb-canvm13:~$ sudo ktutil list|grep -i lxsmbadmin
localadmin at lxsmb-canvm13:~$

With the domain user already logged in through ssh, I expected that the
kerberos TGT would already have been retrieved and stored locally.
Where would I find that?

Do note that if I populate the cred-cache manually with the kinit utility
like so:
localadmin at lxsmb-canvm13:~$ sudo kinit lxsmbadmin at aaddomain.example.com
lxsmbadmin at aaddomain.example.com's Password:
localadmin at lxsmb-canvm13:~$

The cred-cache does get populated and I'm then able to mount the file share
successfully.

With the log level set to 10 in smb.conf, the logging in /var/log/samba/ is
pretty verbose. I can share those if needed for further debugging.

=======================================================