Need urgent help with samba4 DC re-join

[Andreas Oster]

Am 02.07.2012 08:23, schrieb Andrew Bartlett:
> On Mon, 2012-07-02 at 07:34 +0200, Andreas Oster wrote:
>> Am 01.07.2012 22:44, schrieb Andrew Bartlett:
>>> On Thu, 2012-06-28 at 15:16 +0200, Andreas Oster wrote:
>>>> Am 28.06.2012 09:20, schrieb Andrew Bartlett:
>>>>> On Thu, 2012-06-28 at 07:26 +0200, Andreas Oster wrote:
>>>>>> Am 28.06.2012 00:00, schrieb Andrew Bartlett:
>>>>>>> On Wed, 2012-06-27 at 19:27 +0200, Andreas Oster wrote:
>>>>>>>> Am 27.06.2012 15:43, schrieb Andreas Oster:
>>>>>>>>> Am 27.06.2012 15:35, schrieb Andrew Bartlett:
>>>>>>>>>> On Wed, 2012-06-27 at 15:28 +0200, Andreas Oster wrote:
>>>>>>>>>>> Am 27.06.2012 15:21, schrieb Andrew Bartlett:
>>>>>>>>>>>> On Wed, 2012-06-27 at 15:09 +0200, Andreas Oster wrote:
>>>>>>>>>>>>> Hello Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>> i think the only differences when doing a "ldbsearch -H sam.ldb -s base
>>>>>>>>>>>>> -b DC=DomainDnsZones,DC=novanetwork,DC=loc" are:
>>>>>>>>>>>>>
>>>>>>>>>>>>> objectClass: domain
>>>>>>>>>>>>> objectClass: domainDNS
>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>> objectCategory: CN=Top,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not know if this was correct before demoting the second DC.
>>>>>>>>>>>>> It did not come into my mind to check for errors because everything
>>>>>>>>>>>>> worked like a charm and I was/am really happy with samba4.
>>>>>>>>>>>>>
>>>>>>>>>>>>> here the output of:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ../bin/ldbsearch -H sam.ldb -s base -b
>>>>>>>>>>>>> dc=domaindnszones,DC=novanetwork,DC=loc --reveal --show-binary
>>>>>>>>>>>>> replPropertyMetaData
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.  This gives us a very good clue as to what has gone on:
>>>>>>>>>>>>
>>>>>>>>>>>> I'm assuming that 61f36cfd-ba7d-4702-87d3-7e861bb32cfe is PDC and
>>>>>>>>>>>> fd9ca123-ed33-483a-a735-ff41940789a2 was the BDC?
>>>>>>>>>>>>
>>>>>>>>>>>> The key attributes changed that you mention are objectClass and
>>>>>>>>>>>> objectCategory.  Both need to be fixed.  The incorrect values seem to
>>>>>>>>>>>> have been written at Sun Apr 22 16:07:06 2012 CEST compared with Sun Apr
>>>>>>>>>>>> 22 16:03:41 2012 CEST for the good ones.
>>>>>>>>>>>>
>>>>>>>>>>>> My guess is that in attempting to replicate the DNS to the slave with
>>>>>>>>>>>> the samba-tool drs commands, and running samba_upgradedns on that
>>>>>>>>>>>> server, have somehow sent back a corrupted version of the same object.
>>>>>>>>>>>>
>>>>>>>>>>>> Andrew Bartlett
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hello Andrew,
>>>>>>>>>>>
>>>>>>>>>>> this is absolute possible. In a prior try to replicate the
>>>>>>>>>>> DomainDnsZones and ForestDnsZones I used the samba-tool drs command but
>>>>>>>>>>> this did not succeed and, if I do remember correct, quit with an error
>>>>>>>>>>> message. As everything kept on working as before, it did not come to my
>>>>>>>>>>> mind that it might have broken anything.
>>>>>>>>>>>
>>>>>>>>>>> Do you have an idea how to fix this ?
>>>>>>>>>>
>>>>>>>>>> ldbedit -H sam.ldb -s base -b dc=domaindnszones,DC=novanetwork,DC=loc
>>>>>>>>>>
>>>>>>>>>> Then set:
>>>>>>>>>>
>>>>>>>>>> objectClass: domainDNS 
>>>>>>>>>> objectCategory:
>>>>>>>>>> CN=Domain-DNS,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>>>>>>>>
>>>>>>>>>> That should fix it (I hope).
>>>>>>>>>>
>>>>>>>>>> This is the end for me for tonight, but I'll follow up tomorrow.
>>>>>>>>>> Hopefully others here can help you with any remaining details. 
>>>>>>>>>>
>>>>>>>>>> KEEP GOOD BACKUPS.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Andrew Bartlett
>>>>>>>>>>
>>>>>>>>> Hello Andrew,
>>>>>>>>>
>>>>>>>>> thank you very much for your help. I appreciate very much that you use
>>>>>>>>> your limited time to help guys like me.
>>>>>>>>>
>>>>>>>>> I will create a backup and do the proposed changes with ldbedit. I will
>>>>>>>>> report here if joining works again afterwards.
>>>>>>>>>
>>>>>>>>> best regards
>>>>>>>>>
>>>>>>>>> Andreas
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hello Andrew,
>>>>>>>>
>>>>>>>> unfortunately, I have been unable to modify/add the settings via
>>>>>>>> ldbedit. I got the following error message when committing the
>>>>>>>> modifications:
>>>>>>>>
>>>>>>>> ../bin/ldbedit -H sam.ldb -s base -b dc=domaindnszones,DC=novanetwork,DC=loc
>>>>>>>> failed to modify DC=DomainDnsZones,DC=novanetwork,DC=loc - cannot change
>>>>>>>> replicated attribute on partial replica at
>>>>>>>> ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:1408
>>>>>>>>
>>>>>>>> Any idea what could be causing it ?
>>>>>>>
>>>>>>> When Amitay first wrote samba_dnsupgrade, he misunderstood about the
>>>>>>> difference between a partial and a full replica.  A partition does not
>>>>>>> start as one, and then become another.  We will need to correct your
>>>>>>> database to record the DNS partition as being a full replica. 
>>>>>>>
>>>>>>>> Luckily, I did a vmware snapshot before demoting the second DC, I was so
>>>>>>>> upset that I forget about that. I have now reverted back to the old
>>>>>>>> snapshots and second DC is functional again.
>>>>>>>> I have done the tests with ldbsearch on the DomainDnsZones and
>>>>>>>> ForestDnsZones and realized, that the faulty entries already existed
>>>>>>>> before demoting. So I guess before I can demote the second DC again I
>>>>>>>> will have to fix those errors.
>>>>>>>
>>>>>>> It will also be required before any modifications can be made.  This may
>>>>>>> explain why DNS entries appear to be 'stuck' - Samba is refusing to
>>>>>>> change anything in that partition, because it wrongly believes that
>>>>>>> someone else is the master for that data. 
>>>>>>>
>>>>>>> Andrew Bartlett
>>>>>>>
>>>>>> Hello Andrew,
>>>>>>
>>>>>> do you have an idea what needs to be changed ? Is it only the
>>>>>> DomainDnsZones and ForestDnsZones part or are there other places where
>>>>>> changes need to be made ? Yesterday I have tried to change the
>>>>>> DomainDnsZones stuff but got an error message when trying to commit the
>>>>>> modifications.
>>>>>
>>>>> That is what I was trying to explain.  The fact that the NTDS Settings
>>>>> for your DC lists these as partialReplica partitions is the cause of the
>>>>> problem. 
>>>>>
>>>>> We need to correct that in your instance, and if we find that many folks
>>>>> have run the buggy version of the samba_dnsupgrade script, we may need
>>>>> to add a special case to dbcheck for this.  I'm already thinking a
>>>>> schema compliance check would be very worthwhile, so this can be found
>>>>> before modifications are made.
>>>>>
>>>>> Andrew Bartlett
>>>>>
>>>> Hello Andrew,
>>>>
>>>> so, how should I proceed on from here ? What can I do to fix those issues ?
>>>>
>>>> best regards
>>>>
>>>> Andreas
>>>
>>> Can you please run:
>>>
>>> ldbsearch -H sam.ldb -s sub --cross-ncs objectclass=ntdsdsa
>>>
>>> I need to see what is in your NTDS Setting entry for each DC so I can
>>> figure out how to fix this.
>>>
>>> Thanks,
>>>
>>> Andrew Bartlett
>>>
>>
>>
>> Hello Andrew,
>>
>> here is the output of  ../bin/ldbsearch -H sam.ldb -s sub --cross-ncs
>> objectclass=ntdsdsa
>>
>>
>> # record 1
>> dn: CN=NTDS
>> Settings,CN=NOVADC01,CN=Servers,CN=Standardname-des-ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
>> objectClass: top
>> objectClass: applicationSettings
>> objectClass: nTDSDSA
>> cn: NTDS Settings
>> instanceType: 4
>> whenCreated: 20120422134800.0Z
>> uSNCreated: 3212
>> dMDLocation: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> invocationId: 61f36cfd-ba7d-4702-87d3-7e861bb32cfe
>> showInAdvancedViewOnly: TRUE
>> name: NTDS Settings
>> objectGUID: c60bca82-df6e-409e-85c5-e2cc733691da
>> options: 1
>> systemFlags: 33554432
>> objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> msDS-Behavior-Version: 4
>> hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> hasMasterNCs: DC=novanetwork,DC=loc
>> msDS-HasDomainNCs: DC=novanetwork,DC=loc
>> msDS-HasInstantiatedNCs:
>> B:8:0000000D:DC=DomainDnsZones,DC=novanetwork,DC=loc
>> msDS-HasInstantiatedNCs:
>> B:8:0000000D:DC=ForestDnsZones,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=DomainDnsZones,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=ForestDnsZones,DC=novanetwork,DC=loc
>> whenChanged: 20120422140342.0Z
>> uSNChanged: 4066
>> distinguishedName: CN=NTDS
>> Settings,CN=NOVADC01,CN=Servers,CN=Standardname-des
>>  -ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
>>
>> # record 2
>> dn: CN=NTDS
>> Settings,CN=NOVADC02,CN=Servers,CN=Standardname-des-ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
>> objectClass: top
>> objectClass: applicationSettings
>> objectClass: nTDSDSA
>> cn: NTDS Settings
>> instanceType: 4
>> whenCreated: 20120503122809.0Z
>> hasMasterNCs: DC=novanetwork,DC=loc
>> hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> uSNCreated: 5326
>> dMDLocation: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> invocationId: b3ec35a6-d4c1-4f83-8ad6-1dcd330bd353
>> showInAdvancedViewOnly: TRUE
>> name: NTDS Settings
>> objectGUID: 94d1cf02-6aaf-41b7-928c-2292221525d8
>> options: 1
>> systemFlags: 33554432
>> objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> msDS-Behavior-Version: 4
>> msDS-HasDomainNCs: DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> whenChanged: 20120503124935.0Z
>> hasPartialReplicaNCs: DC=DomainDnsZones,DC=novanetwork,DC=loc
>> hasPartialReplicaNCs: DC=ForestDnsZones,DC=novanetwork,DC=loc
>> uSNChanged: 5435
>> distinguishedName: CN=NTDS
>> Settings,CN=NOVADC02,CN=Servers,CN=Standardname-des
>>  -ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
> 
> OK.  Now on which DC were you trying to do the edits?
> 
> You should have been trying to do the edits on DC01, so that we could
> then re-join DC02 from scratch.
> 
> Thanks,
> 
> Andrew Bartlett
> 
Hello Andrew,

I have tried to edit with ldbedit from novadc01 which failed with the
partital relication error message.

best regards

Andreas