Error 8418: The replication operation failed because of a schema mismatch between the servers involved

Fri Feb 12 18:54:29 UTC 2016

On 02/12/2016 09:29 AM, Sinelnikov Evgeniy wrote:
> On Friday, February 12, 2016 10:58 AM, Matthieu Patou wrote:
>> On 02/08/2016 10:20 AM, Sinelnikov Evgeniy wrote:
>>> Hello,
>>>
>>> During the past two weeks, I was able to reproduce on Samba-4.3.4
>> SCHEMA_MISMATCH problem, which looks like this:
>>> [root at dc02 ~]# samba-tool drs replicate dc01 dc02 dc=company3,dc=dd
>>> ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync failed -
>> drsException: DsReplicaSync failed (8418,
>> 'WERR_DS_DRA_SCHEMA_MISMATCH')
>>>     File "/usr/local/samba/lib64/python2.7/site-
>> packages/samba/netcmd/drs.py", line 348, in run
>>>       drs_utils.sendDsReplicaSync(self.drsuapi, self.drsuapi_handle,
>> source_dsa_guid, NC, req_options)
>>>     File "/usr/local/samba/lib64/python2.7/site-
>> packages/samba/drs_utils.py", line 83, in sendDsReplicaSync
>>>       raise drsException("DsReplicaSync failed %s" % estr)
>> I assume that DC01 is Windows and DC02 is Linux, so you are asking Windows
>> to replicate with Linux and it fails.
> Yes, of course, I show it more detail in question about schema applied in samba@:
> https://lists.samba.org/archive/samba/2016-February/197691.html
>
>
>>> [root at dc02 2016-02-08]# cat /etc/redhat-release CentOS Linux release
>>> 7.2.1511 (Core)
>>> [root at dc02 2016-02-08]# samba-tool ldapcmp ldap://dc01 ldap://dc02
>>> --filter=whenChanged
>>>
>>> * Comparing [DOMAIN] context...
>>>       CN=Offline Address Book - /o\=Company3
>> Organisation/cn\=addrlists/cn\=,CN=Microsoft Exchange System
>> Objects,DC=company3,DC=dd
>>>       CN=Offline Address Book - /o\3DCompany3
>>> Organisation/cn\3Daddrlists/cn\3D,CN=Microsoft Exchange System
>>> [...]
>>> * Result for [DOMAIN]: SUCCESS
>>> [...]
>>> * Result for [CONFIGURATION]: SUCCESS
>>> [...]
>>> * Result for [SCHEMA]: SUCCESS
>>> [...]
>>> * Result for [DNSDOMAIN]: SUCCESS
>>> [...]
>>> * Result for [DNSFOREST]: SUCCESS
>> I think ldapcmp is lying, or there is another problem in the DRS code given the
>> error that you have.
> I don't understand yet how to know more right LDAP compares.
> But I tried to use ldbsearch utility for next three NC's:
>
> [root at dc02 Domain]# ./ldif.sh
> ldbsearch --paged -S -k yes -H ldap://dc01.company3.dd -b DC=company3,DC=dd (objectclass=*)
> ldbsearch --paged -S -k yes -H ldap://dc02.company3.dd -b DC=company3,DC=dd (objectclass=*)
> ldbsearch --paged -S -k yes -H ldap://dc03.company3.dd -b DC=company3,DC=dd (objectclass=*)
>
> [root at dc02 Schema]# ./ldif.sh
> ldbsearch --paged -S -k yes -H ldap://dc01.company3.dd -b CN=Schema,CN=Configuration,DC=company3,DC=dd (objectclass=*)
> ldbsearch --paged -S -k yes -H ldap://dc02.company3.dd -b CN=Schema,CN=Configuration,DC=company3,DC=dd (objectclass=*)
> ldbsearch --paged -S -k yes -H ldap://dc03.company3.dd -b CN=Schema,CN=Configuration,DC=company3,DC=dd (objectclass=*)
>
> [root at dc02 Configuration]# ./ldif.sh
> ldbsearch --paged -S -k yes -H ldap://dc01.company3.dd -b CN=Configuration,DC=company3,DC=dd (objectclass=*)
> ldbsearch --paged -S -k yes -H ldap://dc02.company3.dd -b CN=Configuration,DC=company3,DC=dd (objectclass=*)
> ldbsearch --paged -S -k yes -H ldap://dc03.company3.dd -b CN=Configuration,DC=company3,DC=dd (objectclass=*)
>
> where dc01 and dc03 - Windows 2003 DC's, and dc02 - Centos 7.2 with Samba-4.3.4
> Files could be found here - https://goo.gl/xuwNgP
>
> Main what I found:
> 1) For  Domain NC Referral looks on Samba DC different that on Windows DC's:
>
> # Referral
> -ref: ldap://company3.dd/CN=Configuration,DC=company3,DC=dd
> +ref: ldap://ForestDnsZones.company3.dd/DC=ForestDnsZones,DC=company3,DC=dd
> .
>   # Referral
> -ref: ldap://company3.dd/DC=DomainDnsZones,DC=company3,DC=dd
> +ref: ldap://DomainDnsZones.company3.dd/DC=DomainDnsZones,DC=company3,DC=dd
> .
>   # Referral
> -ref: ldap://company3.dd/DC=ForestDnsZones,DC=company3,DC=dd
> +ref: ldap://company3.dd/CN=Configuration,DC=company3,DC=dd
That's interesting but I'm pretty sure that it's irrelevant for your 
problem.

> 2) Strange difference on prefixMap attribute:
>
> --- dc02-Schema.ldif	2016-02-12 15:49:50.420899551 +0000
> +++ dc01-Schema.ldif	2016-02-12 15:49:48.324899579 +0000
> [...]
> @@ -74457,48 +74457,36 @@
>   objectClass: dMD
> objectGUID: 7a51a45f-0110-445f-977a-6e9dbe745abd
>   objectVersion: 30
> -prefixMap: 0:2.5.4;1:2.5.6;2:1.2.840.113556.1.2;3:1.2.840.113556.1.3;4:2.16.84
> - 0.1.101.2.2.1;5:2.16.840.1.101.2.2.3;6:2.16.840.1.101.2.1.5;7:2.16.840.1.101.
> - 2.1.4;8:2.5.5;9:1.2.840.113556.1.4;10:1.2.840.113556.1.5;19:0.9.2342.19200300
> - .100;20:2.16.840.1.113730.3;21:0.9.2342.19200300.100.1;22:2.16.840.1.113730.3
> - .1;23:1.2.840.113556.1.5.7000;24:2.5.21;25:2.5.18;26:2.5.20;11:1.2.840.113556
> - .1.4.260;12:1.2.840.113556.1.5.56;13:1.2.840.113556.1.4.262;14:1.2.840.113556
> - .1.5.57;15:1.2.840.113556.1.4.263;16:1.2.840.113556.1.5.58;17:1.2.840.113556.
> - 1.5.73;18:1.2.840.113556.1.4.305;27:1.3.6.1.4.1.1466.101.119;28:2.16.840.1.11
> - 3730.3.2;29:1.3.6.1.4.1.250.1;30:1.2.840.113549.1.9;31:0.9.2342.19200300.100.
> - 4;32:1.2.840.113556.1.6.23;33:1.2.840.113556.1.6.18.1;34:1.2.840.113556.1.6.1
> - 8.2;35:1.2.840.113556.1.6.13.3;36:1.2.840.113556.1.6.13.4;37:1.3.6.1.1.1.1;38
> - :1.3.6.1.1.1.2;4916:1.2.840.113556.1.4.7000.102;4939:1.2.840.113556.1.5.7000.
> - 62;4965:1.2.840.113556.1.4.7000.102:0x83;4968:1.2.840.113556.1.4.7000.102:0x8
> - 1;4994:1.2.840.113556.1.5.7000.62:0x81;5138:1.2.840.113556.1.5.7000.62:0x83;5
> - 540:1.2.840.113556.1.6.20.1;5546:1.2.840.113556.1.6.20.2
> -replUpToDateVector:: AgAAAAAAAAACAAAAAAAAAMRjlg9FLDxIh3fh4Gvcz4AjYQAAAAAAAA+Pz
> - gwDAAAABeIirkZf70ixZSffRGiEO2uhAAAAAAAAD4/ODAMAAAA=
> -repsFrom:: AQAAAAAAAAAMAQAAAAAAAA+PzgwDAAAAD4/ODAMAAAAAAAAA0AAAADwAAAB0AAAAERE
> +prefixMap:: CAAAAIIAAAA0EwsAKoZIhvcUAQS2WGZLEwsAKoZIhvcUAQW2WD5lEwwAKoZIhvcUAQ
> + S2WGaDaBMMACqGSIb3FAEEtlhmgYITDAAqhkiG9xQBBbZYPoESFAwAKoZIhvcUAQW2WD6DpBUKACq
> + GSIb3FAEGFAGqFQoAKoZIhvcUAQYUAg==
> +replUpToDateVector:: AgAAAAAAAAACAAAAAAAAAMRjlg9FLDxIh3fh4Gvcz4BFYQAAAAAAABaQz
> + gwDAAAAVt6GK0T8A0qD8Pt9uEDyZvIlAAAAAAAAAIA+1d6xnQE=
> +repsFrom:: AQAAAAAAAAAUAQAAAAAAABaQzgwDAAAAFpDODAMAAAAAAAAA2AAAADwAAABwAAAAERE
>    RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER
> - ERERERERERERERERERERERERERERERERAAAAAGahAAAAAAAAAAAAAAAAAABmoQAAAAAAAAXiIq5GX
> - +9IsWUn30RohDsF4iKuRl/vSLFlJ99EaIQ7AAAAAAAAAAAAAAAAAAAAADgAAABhZTIyZTIwNS01Zj
> - Q2LTQ4ZWYtYjE2NS0yN2RmNDQ2ODg0M2IuX21zZGNzLmNvbXBhbnkzLmRkAA==
> -repsFrom:: AQAAAAAAAAAMAQAAAAAAAA+PzgwDAAAAD4/ODAMAAAAAAAAA0AAAADwAAAB0AAAAERE
> + ERERERERERERERERERERERERERERERERAAAAAC4JAAAAAAAAAAAAAAAAAAAuCQAAAAAAAB+G2NML7
> + OZOlAG9gGFgPutW3oYrRPwDSoPw+324QPJmAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAAAAGQzZD
> + g4NjFmLWVjMGItNGVlNi05NDAxLWJkODA2MTYwM2VlYi5fbXNkY3MuY29tcGFueTMuZGQA
> +repsFrom:: AQAAAAAAAAAUAQAAAAAAABaQzgwDAAAAFpDODAMAAAAAAAAA2AAAADwAAABwAAAAERE
>    RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER
> - ERERERERERERERERERERERERERERERERAAAAACBhAAAAAAAAAAAAAAAAAAAgYQAAAAAAABJ7jOXKE
> - S1EvOdeKKAdgjLEY5YPRSw8SId34eBr3M+AAAAAAAAAAAAAAAAAAAAAADgAAABlNThjN2IxMi0xMW
> - NhLTQ0MmQtYmNlNy01ZTI4YTAxZDgyMzIuX21zZGNzLmNvbXBhbnkzLmRkAA==
> -repsTo:: AQAAAAAAAAAMAQAAAAAAANOJzgwDAAAA04nODAMAAAAAAAAA0AAAADwAAAAdAAAAAAAAA
> + ERERERERERERERERERERERERERERERERAAAAAMpgAAAAAAAAAAAAAAAAAADKYAAAAAAAABJ7jOXKE
> + S1EvOdeKKAdgjLEY5YPRSw8SId34eBr3M+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAAAAGU1OG
> + M3YjEyLTExY2EtNDQyZC1iY2U3LTVlMjhhMDFkODIzMi5fbXNkY3MuY29tcGFueTMuZGQA
> +repsTo:: AQAAAAAAAAAUAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2AAAADwAAAAQAAAAAAAAA
>    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABJ7jOXKES1
> - EvOdeKKAdgjIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgAAABlNThjN2IxMi0xMWNh
> - LTQ0MmQtYmNlNy01ZTI4YTAxZDgyMzIuX21zZGNzLmNvbXBhbnkzLmRkAA==
> -repsTo:: AQAAAAAAAAAMAQAAAAAAAEKKzgwDAAAAQorODAMAAAAAAAAA0AAAADwAAAAdAAAAAAAAA
> + AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB+G2NML7OZ
> + OlAG9gGFgPusAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAAAAGQzZDg4
> + NjFmLWVjMGItNGVlNi05NDAxLWJkODA2MTYwM2VlYi5fbXNkY3MuY29tcGFueTMuZGQA
> +repsTo:: AQAAAAAAAAAUAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2AAAADwAAAAQAAAAAAAAA
>    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
> - AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXiIq5GX+9
> - IsWUn30RohDsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgAAABhZTIyZTIwNS01ZjQ2
> - LTQ4ZWYtYjE2NS0yN2RmNDQ2ODg0M2IuX21zZGNzLmNvbXBhbnkzLmRkAA==
> + AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABJ7jOXKES1
> + EvOdeKKAdgjIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOAAAAGU1OGM3
> + YjEyLTExY2EtNDQyZC1iY2U3LTVlMjhhMDFkODIzMi5fbXNkY3MuY29tcGFueTMuZGQA
>   schemaInfo:: /wAACG4F4iKuRl/vSLFlJ99EaIQ7
>   showInAdvancedViewOnly: TRUE
> -uSNChanged: 8
> -uSNCreated: 8
> -whenChanged: 20160212135941.0Z
> +uSNChanged: 41101
> +uSNCreated: 4102
> +whenChanged: 20160212140942.0Z
>   whenCreated: 20160127092803.0Z

USN, which is a local serial number on the domain controller for the 
object is unique per NC per DC, it can vary accross DC so it's "normal" 
to have different values for USNCreated, USNChanged, whenChanged and 
whenCreated are also expected to be off.
repsTo, repsFrom is also expected to be different (DC01  replicates from 
DC02, and vice versa, as the GUID of the DSA object of the server with 
witch it is replicating is stored in those object, it's logicial to be 
different too).
prefixMap, needs more attention as one is binary and the other seems to 
be plain text.
I will look closer as the binary value to check if it's the equivalent 
of the plain text one.
>
>
>> [...]
>>> I traced this SCHEMA_MISMATCH error with gdb and found that
>>> dcesrv_drsuapi_DsGetNCChanges() function generates mismatched
>>> replicas for all name contexts except of
>>> cn=Schema,cn=Configuration,dc=company3,dc=dd.
>>> All other NC's are mismatched during replication process from Samba DC to
>>> Windows DC, but not vice versa.
>> More likely it's either dcesrv_drsuapi_is_reveal_secrets_request or
>> dcesrv_drsuapi_is_gc_pas_request It seems it's because we can't find in our
>> schema the requested attribute from the attids Can you:
> I can debug it, but I don't see any more at runtime in gdb, because error causing
> as DCERPC answer on fulfill DRSUAPI packet.
>
>> 1) Find out what is the real function that is causing this error code to be
>> returned, use something like ".DEBUG(0,(__location__ ": Entering function
>> xyz"))
> This is not our request on Samba DC. This error causing then Windows DC ask
> replication data from Samba DC in GetNCChanges RPC-request, which prepared
> in dcesrv_drsuapi_DsGetNCChanges()
I get this, from the command line you send a command to the windows DC 
to connect back to the Samba DC and replicate the specified NC.
Still it's Samba that is answering the RPC call from the windows DC, so 
in the RPC server code of Samba.
>
> So, next commands on Windows DC and Samba DC run same logic in final:
>
> C:\Documents and Settings\Administrator>repadmin /replicate dc01 dc02 dc=company3,dc=dd
> DsReplicaSync() failed with status 8418 (0x20e2):
>      The replication operation failed because of a schema mismatch between the servers involved.
>
> [root at dc02 samba.git]# samba-tool drs replicate dc01 dc02 dc=company3,dc=dd
> Start replicating for source GUID d3d8861f-ec0b-4ee6-9401-bd8061603eeb.
> ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync failed - drsException: DsReplicaSync failed (8418, 'WERR_DS_DRA_SCHEMA_MISMATCH')
>    File "/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/drs.py", line 349, in run
>      drs_utils.sendDsReplicaSync(self.drsuapi, self.drsuapi_handle, source_dsa_guid, NC, req_options)
>    File "/usr/local/samba/lib64/python2.7/site-packages/samba/drs_utils.py", line 83, in sendDsReplicaSync
>      raise drsException("DsReplicaSync failed %s" % estr)
>
> Samba DC send DsReplicaSync to Windows DC, which initiates replication process.
> Replication always works in pull mode.
>
> Send you samba_log_and_git_diff_for_SCHEMA_MISMATCH.txt with details.
>
>> 2)Print what is the the value of attid when the function returns
>> SCHEMA_MISMATCH
> This is really interesting. But how to debug error on Windows part?
> I got patched Wireshark and write packets in gdb trace mode.
>
> Main strange in decrypted response is:
> attid: UNKNOWN_ENUM_VALUE (0x200F4)
My script showattid for a 2010 exchange schema seems to indicate that 
it's homeMDB attribute:

scripts/showattid.py -s ~/workspace/samba/exchange2010/etc/smb.conf 0x200F4
Unknown parameter encountered: "dns recursive queries"
Ignoring unknown parameter "dns recursive queries"
CN=MSMQ-NT4-FLAGS,CN=SCHEMA,CN=CONFIGURATION,DC=EXCHANGE,DC=HOME,DC=MATWS,DC=NET
1.2.840.113556.1.2.244
Attid 0x200F4(131316) is attribute homeMDB

Can you check the definition of this attribute in the schema NC for 
Windows and Samba DC ?
>
>>> A this time I got decrypted DCERPC packets of DRSUAPI protocol using
>>> wireshark from metze's branch, and his patched version of MIT
>>> Kerberos: https://wiki.samba.org/index.php/Wireshark_Keytab
>>> Also decrypted packets successfully parsed with ndrdump utility.
>>>
>>> I have next plans to debug this problem:
>>> 1. Try to find differences between mismatched and not mismatched of
>> decrypted DRSUAPI packets:
>>>     * https://goo.gl/bpTMKv (Error of replication WindowsDC from SambaDC)
>>>     * https://goo.gl/nVDth9 (Success of replication SambaDC from
>>> WindowsDC) 2. Step by step send to Windows DC controlled list of
>>> replicas in fixed dcesrv_drsuapi_DsGetNCChanges()
>>>
>>>
>>> I'll be glad to know about other debugging techniques of this problem.
>>> All methods that I could try looks too complex.