ad-dc-2 python3 problem

[Noel Power]

I have a problem with samba4.ldap.login_basics and some others that fail 
in the ad_dc_2 job in my attempt to run ad_dc_2 CI job under pure python3

please see

branch: 
https://gitlab.com/samba-team/devel/samba/commits/npower-py3build_ad_dc2
CI: https://gitlab.com/samba-team/devel/samba/pipelines/34355915

I'm betting it is the same underlying cause all the failing tests in 
this job, it is seems also to prevent some environments from starting 
(so looks like it will affect other CI jobs too). Interesting error 
messages are

python3: Traceback (most recent call last):
python3:   File 
"/data/samba-back2/source4/scripting/bin/samba_spnupdate", line 254, in 
<module>
python3:     local_update(add_list)
python3:   File 
"/data/samba-back2/source4/scripting/bin/samba_spnupdate", line 200, in 
local_update
python3:     res = samdb.modify(msg)
python3: _ldb.LdbError: (20, "attribute 'servicePrincipalName': value 
#26 on 'CN=OFFLINEBACKUPDC,OU=Domain 
Controllers,DC=backupdom,DC=samba,DC=example,DC=com' already exists")

which happens periodically (seems samba/smbd runs this on a timer)

Also the test seems to fail specifically with

gensec_spnego_create_negTokenInit_step: gssapi_krb5: creating 
NEG_TOKEN_INIT for ldap/offlinebackupdc failed (next[(null)]): 
NT_STATUS_LOGON_FAILURE
Failed to bind - LDAP client internal error: NT_STATUS_LOGON_FAILURE
Failed to connect to 'ldap://offlinebackupdc' with backend 'ldap': LDAP 
client internal error: NT_STATUS_LOGON_FAILURE
gensec_spnego_create_negTokenInit_step: gssapi_krb5: creating 
NEG_TOKEN_INIT for ldap/offlinebackupdc failed (next[(null)]): 
NT_STATUS_INVALID_PARAMETER
Failed to bind - LDAP client internal error: NT_STATUS_INVALID_PARAMETER
Failed to connect to 'ldap://offlinebackupdc' with backend 'ldap': LDAP 
client internal error: NT_STATUS_INVALID_PARAMETER
ime: 2018-10-26 09:45:53.673424Z
error: 
samba4.ldap.login_basics.python(offlinebackupdc).__main__.BasicUserAuthTests.test_login_basics_krb5(offlinebackupdc) 
[
Exception: Exception: Traceback (most recent call last):
   File "/data/samba-back2/source4/dsdb/tests/python/login_basics.py", 
line 56, in setUp
     super(BasicUserAuthTests, self).setUp()
   File 
"/data/samba-back2/source4/dsdb/tests/python/password_lockout_base.py", 
line 352, in setUp
     self.lockout1krb5_ldb = self._readd_user(self.lockout1krb5_creds)
   File 
"/data/samba-back2/source4/dsdb/tests/python/password_lockout_base.py", 
line 258, in _readd_user
     ldb = SamDB(url=self.host_url, credentials=creds, lp=self.lp)
   File "/data/samba-back2/bin/python/samba/samdb.py", line 65, in __init__
     options=options)
   File "/data/samba-back2/bin/python/samba/__init__.py", line 115, in 
__init__
     self.connect(url, flags, options)
   File "/data/samba-back2/bin/python/samba/samdb.py", line 80, in connect
     options=options)
_ldb.LdbError: (1, 'LDAP client internal error: 
NT_STATUS_INVALID_PARAMETER')


and some poking around with some increased debug and it looks like the 
underlying error is

Starting GENSEC mechanism spnego
Starting GENSEC submechanism gssapi_krb5
Received smb_krb5 packet of length 343
Received smb_krb5 packet of length 212
kinit for lockout1krb5 at BACKUPDOM.SAMBA.EXAMPLE.COM succeeded
Server ldap/offlinebackupdc at BACKUPDOM.SAMBA.EXAMPLE.COM is not 
registered with our KDC:  Miscellaneous failure (see text): Server 
(ldap/offlinebackupdc at BACKUPDOM.SAMBA.EXAMPLE.COM) unknown
gensec_update_send: gssapi_krb5[0x559615e44ba0]: subreq: 0x559615f44110
gensec_update_send: spnego[0x559615f42610]: subreq: 0x559615f42b30
gensec_update_done: gssapi_krb5[0x559615e44ba0]: 
NT_STATUS_INVALID_PARAMETER 
tevent_req[0x559615f44110/../source4/auth/gensec/gensec_gssapi.c:1054]: 
state[3] error[-7963671676338569203 (0x917B5ACDC000000D)] state[struct 
gensec_gssapi_update_state (0x559615f442c0)] timer[(nil)] 
finish[../source4/auth/gensec/gensec_gssapi.c:1067]
gensec_spnego_create_negTokenInit_step: gssapi_krb5: creating 
NEG_TOKEN_INIT for ldap/offlinebackupdc failed (next[ntlmssp]): 
NT_STATUS_INVALID_PARAMETER
Starting GENSEC submechanism ntlmssp

I'm feeling this must be to do with the spnupdate problem above. 
Something funky I think is happening as part of the environment 
setup/provisioning stage. I don't know enough about the ad/dc and the 
associated test environments in general :-), any ideas, hints where to 
look or even a solution would be really really awesome.


To reproduce (note: fails outside of CI too)

clone the branch mentioned above

# configure
PYTHON='python3' ./configure.developer

# make & test
PYTHON='python3' make test FAIL_IMMEDIATELY=1 
TESTS='samba4.ldap.login_basics --include-env=offlinebackupdc '

Thanks

Noel