[Samba] Replication fails after DC re-joined to domain

[Roy Eastwood]

Hi,
First some background:
==================
I had a test environment which had two samba DCs (running v 4.8.0rc2) and 1
Windows Server 2008R2 DC.    The samba DCs had been upgraded from v 4.6x and the
secrets database was not encrypted (as far as I know).    I decided to downgrade
one of the samba DCs to v 4.7.4. 

On re-starting samba after the downgrade the log shows:

ldb: unable to dlopen /usr/local/samba/lib/ldb/encrypted_secrets.so :
/usr/local/samba/lib/private/libdsdb-module-samba4.so: version `SAMBA_4.8.0RC2'
not found (required by /usr/local/samba/lib/ldb/encrypted_secrets.so)

and the samba daemons are left in a failed state.

So I removed this failed DC (dc1) from the domain, using the (working) 4.8.0rc2
samba machine after seizing the FSMO roles and using the
remove-other-dead-server option.   I removed all traces of the existing samba
installation and then installed a freshly compiled copy of v. 4.7.5 and
re-joined the domain.   All was working, including replication (as far as I
know).

I then demoted the other 4.8.0rc2 samba machine (dc2), with the same result, so
had to repeat the seizing of roles and removal then install 4.7.5 and re-join.

Current problem:
==============
Replication reports successful when running samba-tool drs showrepl on both
machines.  However, running:

samba-tool drs replicate dc1 dc2 dc=samdom,dc=example,dc=com --full-sync
or
samba-tool drs replicate dc2 dc1 dc=samdom,dc=example,dc=com --full-sync

on dc1 it runs fine.   Indeed dc1 can do a full replication to the Windows DC as
well.

But running the command
samba-tool drs replicate dc1 dc2 dc=samdom,dc=example,dc=com --full-sync
on dc2 produces an error:

ERROR(<class 'samba.drs_utils.drsException'>): DRS connection to dc1 failed -
drsException: DRS connection to dc1 failed: (-1073741643, '{Device Timeout} The
specified I/O operation on %hs was not completed before the time-out period
expired.')
  File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/drs.py", line
44, in drsuapi_connect
    (ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions) =
drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
  File "/usr/local/samba/lib/python2.7/site-packages/samba/drs_utils.py", line
56, in drsuapi_connect
    raise drsException("DRS connection to %s failed: %s" % (server, e))

and the logs show:
[2018/02/07 16:02:50.440699,  1]
../source4/auth/gensec/gensec_gssapi.c:790(gensec_gssapi_update_internal)
  GSS server Update(krb5)(1) Update failed:  Miscellaneous failure (see text):
Decrypt integrity check failed for checksum type hmac-sha1-96-aes256, key type
aes256-cts-hmac-sha1-96
[2018/02/07 16:02:50.440729,  1]
../auth/gensec/spnego.c:411(gensec_spnego_parse_negTokenInit)
  SPNEGO(gssapi_krb5) NEG_TOKEN_INIT failed: NT_STATUS_LOGON_FAILURE

Replication is OK in the other direction on dc2:  ie
samba-tool drs replicate dc2 dc1 dc=samdom,dc=example,dc=com --full-sync

works ok.

samba-tool dbcheck --cross-ncs shows no errors when run on either machine.

Any help appreciated,
TIA
Roy