Error 8418: The replication operation failed because of a schema mismatch between the servers involved

[Sinelnikov Evgeniy]

Hello,

During the past two weeks, I was able to reproduce on Samba-4.3.4 SCHEMA_MISMATCH problem, which looks like this:

[root at dc02 ~]# samba-tool drs replicate dc01 dc02 dc=company3,dc=dd
ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync failed - drsException: DsReplicaSync failed (8418, 'WERR_DS_DRA_SCHEMA_MISMATCH')
  File "/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/drs.py", line 348, in run
    drs_utils.sendDsReplicaSync(self.drsuapi, self.drsuapi_handle, source_dsa_guid, NC, req_options)
  File "/usr/local/samba/lib64/python2.7/site-packages/samba/drs_utils.py", line 83, in sendDsReplicaSync
    raise drsException("DsReplicaSync failed %s" % estr)

[root at dc02 2016-02-08]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root at dc02 2016-02-08]# samba-tool ldapcmp ldap://dc01 ldap://dc02 --filter=whenChanged

* Comparing [DOMAIN] context...
    CN=Offline Address Book - /o\=Company3 Organisation/cn\=addrlists/cn\=,CN=Microsoft Exchange System Objects,DC=company3,DC=dd
    CN=Offline Address Book - /o\3DCompany3 Organisation/cn\3Daddrlists/cn\3D,CN=Microsoft Exchange System Objects,DC=company3,DC=dd

* Objects to be compared: 203

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 4517

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 2343

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 33

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 19

* Result for [DNSFOREST]: SUCCESS


Reproducable configuration includes Windows Domain Controller (Windows Server 2003, 64-bit)
with Exchange 2003 (Windows Server 2003, 32-bit, not DC) extended AD schema and CentOS-7.2 with
manually build Samba-4.3.4. This is simplified configuration of previously mailed configuration with same error:
https://lists.samba.org/archive/samba/2015-May/191635.html 

On Windows DC SCHEMA_MISMATCH problem looks like this (one of multiple types of events):

Event Type:	Error
Event Source:	NTDS Replication
Event Category:	Replication 
Event ID:	1791
Date:		27.01.2016
Time:		14:35:09
User:		NT AUTHORITY\ANONYMOUS LOGON
Computer:	DC01
Description:
Replication of Naming Context DC=company3,DC=dd from source a87941a1-9718-4f2a-91fe-bdb993dbd05b has been aborted. Replication requires consistent schema but last attempt to sync the schema had failed. It is crucial that schema replication functions properly. See previous errors for more diagnostics. If this issue persists, please contact Microsoft Product Support Services for assistance. Error 8418: The replication operation failed because of a schema mismatch between the servers involved..


I traced this SCHEMA_MISMATCH error with gdb and found that dcesrv_drsuapi_DsGetNCChanges() function
generates mismatched replicas for all name contexts except of cn=Schema,cn=Configuration,dc=company3,dc=dd.
All other NC's are mismatched during replication process from Samba DC to Windows DC, but not vice versa.

A this time I got decrypted DCERPC packets of DRSUAPI protocol using wireshark from metze's branch,
and his patched version of MIT Kerberos: https://wiki.samba.org/index.php/Wireshark_Keytab
Also decrypted packets successfully parsed with ndrdump utility.

I have next plans to debug this problem:
1. Try to find differences between mismatched and not mismatched of decrypted DRSUAPI packets:
  * https://goo.gl/bpTMKv (Error of replication WindowsDC from SambaDC)
  * https://goo.gl/nVDth9 (Success of replication SambaDC from WindowsDC)
2. Step by step send to Windows DC controlled list of replicas in fixed dcesrv_drsuapi_DsGetNCChanges()


I'll be glad to know about other debugging techniques of this problem. 
All methods that I could try looks too complex.