Need urgent help with samba4 DC re-join

[Andreas Oster]

Am 02.07.2012 08:23, schrieb Andrew Bartlett:
> On Mon, 2012-07-02 at 07:34 +0200, Andreas Oster wrote:
>> Am 01.07.2012 22:44, schrieb Andrew Bartlett:
>>> On Thu, 2012-06-28 at 15:16 +0200, Andreas Oster wrote:
>>>> Am 28.06.2012 09:20, schrieb Andrew Bartlett:
>>>>> On Thu, 2012-06-28 at 07:26 +0200, Andreas Oster wrote:
>>>>>> Am 28.06.2012 00:00, schrieb Andrew Bartlett:
>>>>>>> On Wed, 2012-06-27 at 19:27 +0200, Andreas Oster wrote:
>>>>>>>> Am 27.06.2012 15:43, schrieb Andreas Oster:
>>>>>>>>> Am 27.06.2012 15:35, schrieb Andrew Bartlett:
>>>>>>>>>> On Wed, 2012-06-27 at 15:28 +0200, Andreas Oster wrote:
>>>>>>>>>>> Am 27.06.2012 15:21, schrieb Andrew Bartlett:
>>>>>>>>>>>> On Wed, 2012-06-27 at 15:09 +0200, Andreas Oster wrote:
>>>>>>>>>>>>> Hello Andrew,
>>>>>>>>>>>>>
>>>>>>>>>>>>> i think the only differences when doing a "ldbsearch -H sam.ldb -s base
>>>>>>>>>>>>> -b DC=DomainDnsZones,DC=novanetwork,DC=loc" are:
>>>>>>>>>>>>>
>>>>>>>>>>>>> objectClass: domain
>>>>>>>>>>>>> objectClass: domainDNS
>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>> objectCategory: CN=Top,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not know if this was correct before demoting the second DC.
>>>>>>>>>>>>> It did not come into my mind to check for errors because everything
>>>>>>>>>>>>> worked like a charm and I was/am really happy with samba4.
>>>>>>>>>>>>>
>>>>>>>>>>>>> here the output of:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ../bin/ldbsearch -H sam.ldb -s base -b
>>>>>>>>>>>>> dc=domaindnszones,DC=novanetwork,DC=loc --reveal --show-binary
>>>>>>>>>>>>> replPropertyMetaData
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.  This gives us a very good clue as to what has gone on:
>>>>>>>>>>>>
>>>>>>>>>>>> I'm assuming that 61f36cfd-ba7d-4702-87d3-7e861bb32cfe is PDC and
>>>>>>>>>>>> fd9ca123-ed33-483a-a735-ff41940789a2 was the BDC?
>>>>>>>>>>>>
>>>>>>>>>>>> The key attributes changed that you mention are objectClass and
>>>>>>>>>>>> objectCategory.  Both need to be fixed.  The incorrect values seem to
>>>>>>>>>>>> have been written at Sun Apr 22 16:07:06 2012 CEST compared with Sun Apr
>>>>>>>>>>>> 22 16:03:41 2012 CEST for the good ones.
>>>>>>>>>>>>
>>>>>>>>>>>> My guess is that in attempting to replicate the DNS to the slave with
>>>>>>>>>>>> the samba-tool drs commands, and running samba_upgradedns on that
>>>>>>>>>>>> server, have somehow sent back a corrupted version of the same object.
>>>>>>>>>>>>
>>>>>>>>>>>> Andrew Bartlett
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hello Andrew,
>>>>>>>>>>>
>>>>>>>>>>> this is absolute possible. In a prior try to replicate the
>>>>>>>>>>> DomainDnsZones and ForestDnsZones I used the samba-tool drs command but
>>>>>>>>>>> this did not succeed and, if I do remember correct, quit with an error
>>>>>>>>>>> message. As everything kept on working as before, it did not come to my
>>>>>>>>>>> mind that it might have broken anything.
>>>>>>>>>>>
>>>>>>>>>>> Do you have an idea how to fix this ?
>>>>>>>>>>
>>>>>>>>>> ldbedit -H sam.ldb -s base -b dc=domaindnszones,DC=novanetwork,DC=loc
>>>>>>>>>>
>>>>>>>>>> Then set:
>>>>>>>>>>
>>>>>>>>>> objectClass: domainDNS 
>>>>>>>>>> objectCategory:
>>>>>>>>>> CN=Domain-DNS,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>>>>>>>>
>>>>>>>>>> That should fix it (I hope).
>>>>>>>>>>
>>>>>>>>>> This is the end for me for tonight, but I'll follow up tomorrow.
>>>>>>>>>> Hopefully others here can help you with any remaining details. 
>>>>>>>>>>
>>>>>>>>>> KEEP GOOD BACKUPS.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Andrew Bartlett
>>>>>>>>>>
>>>>>>>>> Hello Andrew,
>>>>>>>>>
>>>>>>>>> thank you very much for your help. I appreciate very much that you use
>>>>>>>>> your limited time to help guys like me.
>>>>>>>>>
>>>>>>>>> I will create a backup and do the proposed changes with ldbedit. I will
>>>>>>>>> report here if joining works again afterwards.
>>>>>>>>>
>>>>>>>>> best regards
>>>>>>>>>
>>>>>>>>> Andreas
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hello Andrew,
>>>>>>>>
>>>>>>>> unfortunately, I have been unable to modify/add the settings via
>>>>>>>> ldbedit. I got the following error message when committing the
>>>>>>>> modifications:
>>>>>>>>
>>>>>>>> ../bin/ldbedit -H sam.ldb -s base -b dc=domaindnszones,DC=novanetwork,DC=loc
>>>>>>>> failed to modify DC=DomainDnsZones,DC=novanetwork,DC=loc - cannot change
>>>>>>>> replicated attribute on partial replica at
>>>>>>>> ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:1408
>>>>>>>>
>>>>>>>> Any idea what could be causing it ?
>>>>>>>
>>>>>>> When Amitay first wrote samba_dnsupgrade, he misunderstood about the
>>>>>>> difference between a partial and a full replica.  A partition does not
>>>>>>> start as one, and then become another.  We will need to correct your
>>>>>>> database to record the DNS partition as being a full replica. 
>>>>>>>
>>>>>>>> Luckily, I did a vmware snapshot before demoting the second DC, I was so
>>>>>>>> upset that I forget about that. I have now reverted back to the old
>>>>>>>> snapshots and second DC is functional again.
>>>>>>>> I have done the tests with ldbsearch on the DomainDnsZones and
>>>>>>>> ForestDnsZones and realized, that the faulty entries already existed
>>>>>>>> before demoting. So I guess before I can demote the second DC again I
>>>>>>>> will have to fix those errors.
>>>>>>>
>>>>>>> It will also be required before any modifications can be made.  This may
>>>>>>> explain why DNS entries appear to be 'stuck' - Samba is refusing to
>>>>>>> change anything in that partition, because it wrongly believes that
>>>>>>> someone else is the master for that data. 
>>>>>>>
>>>>>>> Andrew Bartlett
>>>>>>>
>>>>>> Hello Andrew,
>>>>>>
>>>>>> do you have an idea what needs to be changed ? Is it only the
>>>>>> DomainDnsZones and ForestDnsZones part or are there other places where
>>>>>> changes need to be made ? Yesterday I have tried to change the
>>>>>> DomainDnsZones stuff but got an error message when trying to commit the
>>>>>> modifications.
>>>>>
>>>>> That is what I was trying to explain.  The fact that the NTDS Settings
>>>>> for your DC lists these as partialReplica partitions is the cause of the
>>>>> problem. 
>>>>>
>>>>> We need to correct that in your instance, and if we find that many folks
>>>>> have run the buggy version of the samba_dnsupgrade script, we may need
>>>>> to add a special case to dbcheck for this.  I'm already thinking a
>>>>> schema compliance check would be very worthwhile, so this can be found
>>>>> before modifications are made.
>>>>>
>>>>> Andrew Bartlett
>>>>>
>>>> Hello Andrew,
>>>>
>>>> so, how should I proceed on from here ? What can I do to fix those issues ?
>>>>
>>>> best regards
>>>>
>>>> Andreas
>>>
>>> Can you please run:
>>>
>>> ldbsearch -H sam.ldb -s sub --cross-ncs objectclass=ntdsdsa
>>>
>>> I need to see what is in your NTDS Setting entry for each DC so I can
>>> figure out how to fix this.
>>>
>>> Thanks,
>>>
>>> Andrew Bartlett
>>>
>>
>>
>> Hello Andrew,
>>
>> here is the output of  ../bin/ldbsearch -H sam.ldb -s sub --cross-ncs
>> objectclass=ntdsdsa
>>
>>
>> # record 1
>> dn: CN=NTDS
>> Settings,CN=NOVADC01,CN=Servers,CN=Standardname-des-ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
>> objectClass: top
>> objectClass: applicationSettings
>> objectClass: nTDSDSA
>> cn: NTDS Settings
>> instanceType: 4
>> whenCreated: 20120422134800.0Z
>> uSNCreated: 3212
>> dMDLocation: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> invocationId: 61f36cfd-ba7d-4702-87d3-7e861bb32cfe
>> showInAdvancedViewOnly: TRUE
>> name: NTDS Settings
>> objectGUID: c60bca82-df6e-409e-85c5-e2cc733691da
>> options: 1
>> systemFlags: 33554432
>> objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> msDS-Behavior-Version: 4
>> hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> hasMasterNCs: DC=novanetwork,DC=loc
>> msDS-HasDomainNCs: DC=novanetwork,DC=loc
>> msDS-HasInstantiatedNCs:
>> B:8:0000000D:DC=DomainDnsZones,DC=novanetwork,DC=loc
>> msDS-HasInstantiatedNCs:
>> B:8:0000000D:DC=ForestDnsZones,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=DomainDnsZones,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=ForestDnsZones,DC=novanetwork,DC=loc
>> whenChanged: 20120422140342.0Z
>> uSNChanged: 4066
>> distinguishedName: CN=NTDS
>> Settings,CN=NOVADC01,CN=Servers,CN=Standardname-des
>>  -ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
>>
>> # record 2
>> dn: CN=NTDS
>> Settings,CN=NOVADC02,CN=Servers,CN=Standardname-des-ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
>> objectClass: top
>> objectClass: applicationSettings
>> objectClass: nTDSDSA
>> cn: NTDS Settings
>> instanceType: 4
>> whenCreated: 20120503122809.0Z
>> hasMasterNCs: DC=novanetwork,DC=loc
>> hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> uSNCreated: 5326
>> dMDLocation: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> invocationId: b3ec35a6-d4c1-4f83-8ad6-1dcd330bd353
>> showInAdvancedViewOnly: TRUE
>> name: NTDS Settings
>> objectGUID: 94d1cf02-6aaf-41b7-928c-2292221525d8
>> options: 1
>> systemFlags: 33554432
>> objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> msDS-Behavior-Version: 4
>> msDS-HasDomainNCs: DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
>> msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>> whenChanged: 20120503124935.0Z
>> hasPartialReplicaNCs: DC=DomainDnsZones,DC=novanetwork,DC=loc
>> hasPartialReplicaNCs: DC=ForestDnsZones,DC=novanetwork,DC=loc
>> uSNChanged: 5435
>> distinguishedName: CN=NTDS
>> Settings,CN=NOVADC02,CN=Servers,CN=Standardname-des
>>  -ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
> 
> OK.  Now on which DC were you trying to do the edits?
> 
> You should have been trying to do the edits on DC01, so that we could
> then re-join DC02 from scratch.
> 
> Thanks,
> 
> Andrew Bartlett
> 


Hello Andrew,

as I have written, I have managed to restore the system to the state
before my disastrous attempt to demote my BDC (novadc02). Currently both
servers operate normal but still the problems with objectClass and
objectCategory of the DomainDnsZones and ForestDnsZones exists.

Would it make sense to, after taking a proper backup, demote the second
DC again or should the faulty DB entries be fixed first ?

best regards

Andreas