[Samba] winbind trust account password management

[Andrew Morgan]

I'm running Samba v4.4.4 as a domain member server in security=domain 
mode.  Our 3 domain controllers are Server 2012r2.

Every 3-4 days, I see log messages from winbind saying 
"winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED". 
Sometimes this corresponds to a trust password change, but not always. 
Today, new connections to Samba were failing with the error 
"SPNEGO(gse_krb5) creating NEG_TOKEN_INIT failed: 
NT_STATUS_INTERNAL_ERROR" for an hour.  I restored service by re-running 
"net rpc join" and restarting winbindd.

I search bugzilla for any issues like this, and I looked at the release 
notes for versions newer than v4.4.4.  I don't see anything specifically 
related to this.

Here are the recent winbind log entries that show this problem:

Oct 15 08:10:40 onid-fs1 winbindd[11194]: [2016/10/15 08:10:40.373760,  1] ../source3/libsmb/trusts_util.c:264(trust_pw_change)
Oct 15 08:10:40 onid-fs1 winbindd[11194]:   2016/10/15 08:10:40 : trust_pw_change(ONID): Changed password locally
Oct 15 08:10:40 onid-fs1 winbindd[11194]: [2016/10/15 08:10:40.426325,  1] ../source3/libsmb/trusts_util.c:278(trust_pw_change)
Oct 15 08:10:40 onid-fs1 winbindd[11194]:   2016/10/15 08:10:40 : trust_pw_change(ONID): Changed password remotely.
Oct 19 08:13:53 onid-fs1 winbindd[11194]: [2016/10/19 08:13:53.347255,  1] ../source3/winbindd/winbindd_pam.c:1439(winbind_samlogon_retry_loop)
Oct 19 08:13:53 onid-fs1 winbindd[11194]:   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.  Maybe the DC has Restrict NTLM set or the trust account password was changed and we didn't know it. Killing connections to domain ONID
Oct 19 08:13:53 onid-fs1 winbindd[11194]: [2016/10/19 08:13:53.931669,  1] ../auth/gensec/spnego.c:619(gensec_spnego_create_negTokenInit)
Oct 19 08:13:53 onid-fs1 winbindd[11194]:   SPNEGO(gse_krb5) creating NEG_TOKEN_INIT failed: NT_STATUS_INTERNAL_ERROR
Oct 22 08:10:40 onid-fs1 winbindd[11194]: [2016/10/22 08:10:40.328862,  1] ../source3/libsmb/trusts_util.c:264(trust_pw_change)
Oct 22 08:10:40 onid-fs1 winbindd[11194]:   2016/10/22 08:10:40 : trust_pw_change(ONID): Changed password locally
Oct 22 08:10:40 onid-fs1 winbindd[11194]: [2016/10/22 08:10:40.412899,  1] ../source3/libsmb/trusts_util.c:278(trust_pw_change)
Oct 22 08:10:40 onid-fs1 winbindd[11194]:   2016/10/22 08:10:40 : trust_pw_change(ONID): Changed password remotely.
Oct 26 08:24:04 onid-fs1 winbindd[11194]: [2016/10/26 08:24:04.475864,  1] ../source3/winbindd/winbindd_pam.c:1439(winbind_samlogon_retry_loop)
Oct 26 08:24:04 onid-fs1 winbindd[11194]:   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.  Maybe the DC has Restrict NTLM set or the trust account password was changed and we didn't know it. Killing connections to domain ONID
Oct 26 08:24:04 onid-fs1 winbindd[11194]: [2016/10/26 08:24:04.857873,  1] ../auth/gensec/spnego.c:619(gensec_spnego_create_negTokenInit)
Oct 26 08:24:04 onid-fs1 winbindd[11194]:   SPNEGO(gse_krb5) creating NEG_TOKEN_INIT failed: NT_STATUS_INTERNAL_ERROR
Oct 26 08:24:05 onid-fs1 winbindd[11194]: [2016/10/26 08:24:05.061340,  1] ../auth/gensec/spnego.c:619(gensec_spnego_create_negTokenInit)
Oct 26 08:24:05 onid-fs1 winbindd[11194]:   SPNEGO(gse_krb5) creating NEG_TOKEN_INIT failed: NT_STATUS_INTERNAL_ERROR
Oct 26 08:24:25 onid-fs1 winbindd[11194]: [2016/10/26 08:24:25.402327,  1] ../source3/rpc_client/cli_pipe.c:421(cli_pipe_validate_current_pdu)
Oct 26 08:24:25 onid-fs1 winbindd[11194]:   ../source3/rpc_client/cli_pipe.c:421: Bind NACK received from host DC1-ONID!
Oct 26 08:24:25 onid-fs1 winbindd[11194]: [2016/10/26 08:24:25.403217,  1] ../source3/rpc_client/cli_pipe.c:3316(cli_rpc_pipe_open_schannel_with_creds)
Oct 26 08:24:25 onid-fs1 winbindd[11194]:   cli_rpc_pipe_open_schannel_with_creds: rpc_pipe_bind failed with error NT_STATUS_NETWORK_ACCESS_DENIED
Oct 29 08:10:40 onid-fs1 winbindd[11194]: [2016/10/29 08:10:40.585520,  1] ../source3/libsmb/trusts_util.c:264(trust_pw_change)
Oct 29 08:10:40 onid-fs1 winbindd[11194]:   2016/10/29 08:10:40 : trust_pw_change(ONID): Changed password locally
Oct 29 08:10:40 onid-fs1 winbindd[11194]: [2016/10/29 08:10:40.639099,  1] ../source3/libsmb/trusts_util.c:278(trust_pw_change)
Oct 29 08:10:40 onid-fs1 winbindd[11194]:   2016/10/29 08:10:40 : trust_pw_change(ONID): Changed password remotely.
Nov  2 08:14:01 onid-fs1 winbindd[11194]: [2016/11/02 08:14:01.521168,  1] ../source3/winbindd/winbindd_pam.c:1439(winbind_samlogon_retry_loop)
Nov  2 08:14:01 onid-fs1 winbindd[11194]:   winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.  Maybe the DC has Restrict NTLM set or the trust account password was changed and we didn't know it. Killing connections to domain ONID
Nov  2 08:14:02 onid-fs1 winbindd[11194]: [2016/11/02 08:14:02.039227,  1] ../auth/gensec/spnego.c:619(gensec_spnego_create_negTokenInit)
Nov  2 08:14:02 onid-fs1 winbindd[11194]:   SPNEGO(gse_krb5) creating NEG_TOKEN_INIT failed: NT_STATUS_INTERNAL_ERROR
Nov  2 08:14:02 onid-fs1 winbindd[11194]: [2016/11/02 08:14:02.366355,  1] ../auth/gensec/spnego.c:619(gensec_spnego_create_negTokenInit)
Nov  2 08:14:02 onid-fs1 winbindd[11194]:   SPNEGO(gse_krb5) creating NEG_TOKEN_INIT failed: NT_STATUS_INTERNAL_ERROR
(many of this last error message - once for each connection attempt until 
I fixed it)


Is there a known issue here?

How does winbindd manage the trust password?

Should I be using security=ads mode instead?

Thanks,
 	Andy