[Samba] samba AD problem after re-join domain

[Jason Keltz]

I've been working on a Samba AD setup with a bunch of test machines - 
the one DC, and a bunch of clients.  Last night, I ended up switching 
the name of the test machines temporarily (except the DC), and 
re-joining the domain (that's for another e-mail later).  When things 
didn't work the way I had planned,  I switched the hostnames back, and 
re-joined the domain today on all the test machines.  I was shocked to 
find that I am only able to login to the domain on one of my hosts.  It 
fails on all the other ones.  I ensured that I deleted the machine 
entries from AD.  I haven't changed my Samba config in months which 
Rowland had last verified was fine.  I haven't changed my /etc/krb5.conf 
Kerberos config in months.  I even did a complete rebuild of one of the 
machines since I automated the installation process, and that rebuild 
was working perfectly many many times, but now it is failed.  In winbind 
log every time I try to login I'm mostly seeing:

[2020/10/11 21:33:45.498701,  1, pid=3637, effective(1004, 0), 
real(1004, 0)] ../../source3/libads/authdata.c:177(kerberos_return_pac)
   kinit failed for 'jas at AD.EECS.YORKU.CA' with: Preauthentication 
failed (-1765328360)

.. which clearly doesn't make sense given that the net ads join 
completed successfully, the computer entry is there, just like before.  
In fact, I can login to the system console as root, then do a "kinit 
jas", and it gets a ticket just fine so the system is able to talk to 
the DC.   Winbind is unhappy about something, but I just can't figure 
out what that is.  On the DC, I can still query all the users, groups, etc.

    I enabled log level 3 and get:

[2020/10/11 21:33:45.426469,  3, pid=3637, effective(0, 0), real(0, 0)] 
../../source3/winbindd/winbindd_pam.c:2089(winbindd_dual_pam_auth)
   [ 3635]: dual pam auth EECSYORKUCA\jas
[2020/10/11 21:33:45.498701,  1, pid=3637, effective(1004, 0), 
real(1004, 0)] ../../source3/libads/authdata.c:177(kerberos_return_pac)
   kinit failed for 'jas at AD.EECS.YORKU.CA' with: Preauthentication 
failed (-1765328360)
[2020/10/11 21:33:45.498763,  2, pid=3637, effective(0, 0), real(0, 0)] 
../../source3/winbindd/winbindd_pam.c:2410(winbindd_dual_pam_auth)
   Plain-text authentication for user EECSYORKUCA\jas returned 
NT_STATUS_LOGON_FAILURE (PAM: 7)
[2020/10/11 21:33:45.498779,  3, pid=3637, effective(0, 0), real(0, 0)] 
../../libcli/security/dom_sid.c:215(dom_sid_parse_endp)
   string_to_sid: SID  is not in a valid format
[2020/10/11 21:33:45.498807,  2, pid=3637, effective(0, 0), real(0, 0)] 
../../auth/auth_log.c:653(log_authentication_event_human_readable)
   Auth: [winbind,PAM_AUTH, nss_winbind, 3635] user [EECSYORKUCA]\[jas] 
at [Sun, 11 Oct 2020 21:33:45.498795 EDT] with [Plaintext] status [NT_ST
ATUS_LOGON_FAILURE] workstation [(null)] remote host [unix:] mapped to 
[(null)]\[(null)]. local host [unix:]
   {"timestamp": "2020-10-11T21:33:45.498912-0400", "type": 
"Authentication", "Authentication": {"version": {"major": 1, "minor": 
2}, "eventId":
  4625, "logonId": "c6dad50c7ecbb3a4", "logonType": 8, "status": 
"NT_STATUS_LOGON_FAILURE", "localAddress": "unix:", "remoteAddress": 
"unix:", "
serviceDescription": "winbind", "authDescription": "PAM_AUTH, 
nss_winbind, 3635", "clientDomain": "EECSYORKUCA", "clientAccount": 
"jas", "works
tation": null, "becameAccount": "", "becameDomain": "", "becameSid": 
null, "mappedAccount": null, "mappedDomain": null, "netlogonComputer": nul
l, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", 
"netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "pass
wordType": "Plaintext", "duration": 72496}}
[2020/10/11 21:33:48.636206,  3, pid=3637, effective(0, 0), real(0, 0)] 
../../source3/winbindd/winbindd_pam.c:2089(winbindd_dual_pam_auth)
   [ 3635]: dual pam auth EECSYORKUCA\jas
[2020/10/11 21:33:48.726636,  1, pid=3637, effective(1004, 0), 
real(1004, 0)] ../../source3/libads/authdata.c:177(kerberos_return_pac)
   kinit failed for 'jas at AD.EECS.YORKU.CA' with: Preauthentication 
failed (-1765328360)
[2020/10/11 21:33:48.726690,  2, pid=3637, effective(0, 0), real(0, 0)] 
../../source3/winbindd/winbindd_pam.c:2410(winbindd_dual_pam_auth)
   Plain-text authentication for user EECSYORKUCA\jas returned 
NT_STATUS_LOGON_FAILURE (PAM: 7)
[2020/10/11 21:33:48.726705,  3, pid=3637, effective(0, 0), real(0, 0)] 
../../libcli/security/dom_sid.c:215(dom_sid_parse_endp)
   string_to_sid: SID  is not in a valid format

I don't know if that SID error is the problem, but I've seen that in 
other debug logs before, so I think it's probably not.

One the one system that works, I'm seeing the following error in the log:

../../source3/librpc/crypto/gse_krb5.c:417: krb5_kt_start_seq_get failed 
(Permission denied)
[2020/10/11 20:54:46.663685,  3, pid=26219, effective(4481, 0), 
real(4481, 0)] 
../../source3/librpc/crypto/gse_krb5.c:577(gse_krb5_get_server_keytab)
   ../../source3/librpc/crypto/gse_krb5.c:577: Warning! Unable to set 
mem keytab from system keytab!

Any thoughts?  I've just spent the last 9 hours looking at this on a 
Sunday of a holiday weekend and have unfortunately not got anywhere.

Jason.