Need urgent help with samba4 DC re-join

[Andreas Oster]

Am 01.07.2012 22:44, schrieb Andrew Bartlett:
> On Thu, 2012-06-28 at 15:16 +0200, Andreas Oster wrote:
>> Am 28.06.2012 09:20, schrieb Andrew Bartlett:
>>> On Thu, 2012-06-28 at 07:26 +0200, Andreas Oster wrote:
>>>> Am 28.06.2012 00:00, schrieb Andrew Bartlett:
>>>>> On Wed, 2012-06-27 at 19:27 +0200, Andreas Oster wrote:
>>>>>> Am 27.06.2012 15:43, schrieb Andreas Oster:
>>>>>>> Am 27.06.2012 15:35, schrieb Andrew Bartlett:
>>>>>>>> On Wed, 2012-06-27 at 15:28 +0200, Andreas Oster wrote:
>>>>>>>>> Am 27.06.2012 15:21, schrieb Andrew Bartlett:
>>>>>>>>>> On Wed, 2012-06-27 at 15:09 +0200, Andreas Oster wrote:
>>>>>>>>>>> Hello Andrew,
>>>>>>>>>>>
>>>>>>>>>>> i think the only differences when doing a "ldbsearch -H sam.ldb -s base
>>>>>>>>>>> -b DC=DomainDnsZones,DC=novanetwork,DC=loc" are:
>>>>>>>>>>>
>>>>>>>>>>> objectClass: domain
>>>>>>>>>>> objectClass: domainDNS
>>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>>>>>
>>>>>>>>>>> objectCategory: CN=Top,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I do not know if this was correct before demoting the second DC.
>>>>>>>>>>> It did not come into my mind to check for errors because everything
>>>>>>>>>>> worked like a charm and I was/am really happy with samba4.
>>>>>>>>>>>
>>>>>>>>>>> here the output of:
>>>>>>>>>>>
>>>>>>>>>>> ../bin/ldbsearch -H sam.ldb -s base -b
>>>>>>>>>>> dc=domaindnszones,DC=novanetwork,DC=loc --reveal --show-binary
>>>>>>>>>>> replPropertyMetaData
>>>>>>>>>>
>>>>>>>>>> Thanks.  This gives us a very good clue as to what has gone on:
>>>>>>>>>>
>>>>>>>>>> I'm assuming that 61f36cfd-ba7d-4702-87d3-7e861bb32cfe is PDC and
>>>>>>>>>> fd9ca123-ed33-483a-a735-ff41940789a2 was the BDC?
>>>>>>>>>>
>>>>>>>>>> The key attributes changed that you mention are objectClass and
>>>>>>>>>> objectCategory.  Both need to be fixed.  The incorrect values seem to
>>>>>>>>>> have been written at Sun Apr 22 16:07:06 2012 CEST compared with Sun Apr
>>>>>>>>>> 22 16:03:41 2012 CEST for the good ones.
>>>>>>>>>>
>>>>>>>>>> My guess is that in attempting to replicate the DNS to the slave with
>>>>>>>>>> the samba-tool drs commands, and running samba_upgradedns on that
>>>>>>>>>> server, have somehow sent back a corrupted version of the same object.
>>>>>>>>>>
>>>>>>>>>> Andrew Bartlett
>>>>>>>>>>
>>>>>>>>
>>>>>>>>> Hello Andrew,
>>>>>>>>>
>>>>>>>>> this is absolute possible. In a prior try to replicate the
>>>>>>>>> DomainDnsZones and ForestDnsZones I used the samba-tool drs command but
>>>>>>>>> this did not succeed and, if I do remember correct, quit with an error
>>>>>>>>> message. As everything kept on working as before, it did not come to my
>>>>>>>>> mind that it might have broken anything.
>>>>>>>>>
>>>>>>>>> Do you have an idea how to fix this ?
>>>>>>>>
>>>>>>>> ldbedit -H sam.ldb -s base -b dc=domaindnszones,DC=novanetwork,DC=loc
>>>>>>>>
>>>>>>>> Then set:
>>>>>>>>
>>>>>>>> objectClass: domainDNS 
>>>>>>>> objectCategory:
>>>>>>>> CN=Domain-DNS,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>>>>>>
>>>>>>>> That should fix it (I hope).
>>>>>>>>
>>>>>>>> This is the end for me for tonight, but I'll follow up tomorrow.
>>>>>>>> Hopefully others here can help you with any remaining details. 
>>>>>>>>
>>>>>>>> KEEP GOOD BACKUPS.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Andrew Bartlett
>>>>>>>>
>>>>>>> Hello Andrew,
>>>>>>>
>>>>>>> thank you very much for your help. I appreciate very much that you use
>>>>>>> your limited time to help guys like me.
>>>>>>>
>>>>>>> I will create a backup and do the proposed changes with ldbedit. I will
>>>>>>> report here if joining works again afterwards.
>>>>>>>
>>>>>>> best regards
>>>>>>>
>>>>>>> Andreas
>>>>>>>
>>>>>>>
>>>>>> Hello Andrew,
>>>>>>
>>>>>> unfortunately, I have been unable to modify/add the settings via
>>>>>> ldbedit. I got the following error message when committing the
>>>>>> modifications:
>>>>>>
>>>>>> ../bin/ldbedit -H sam.ldb -s base -b dc=domaindnszones,DC=novanetwork,DC=loc
>>>>>> failed to modify DC=DomainDnsZones,DC=novanetwork,DC=loc - cannot change
>>>>>> replicated attribute on partial replica at
>>>>>> ../source4/dsdb/samdb/ldb_modules/repl_meta_data.c:1408
>>>>>>
>>>>>> Any idea what could be causing it ?
>>>>>
>>>>> When Amitay first wrote samba_dnsupgrade, he misunderstood about the
>>>>> difference between a partial and a full replica.  A partition does not
>>>>> start as one, and then become another.  We will need to correct your
>>>>> database to record the DNS partition as being a full replica. 
>>>>>
>>>>>> Luckily, I did a vmware snapshot before demoting the second DC, I was so
>>>>>> upset that I forget about that. I have now reverted back to the old
>>>>>> snapshots and second DC is functional again.
>>>>>> I have done the tests with ldbsearch on the DomainDnsZones and
>>>>>> ForestDnsZones and realized, that the faulty entries already existed
>>>>>> before demoting. So I guess before I can demote the second DC again I
>>>>>> will have to fix those errors.
>>>>>
>>>>> It will also be required before any modifications can be made.  This may
>>>>> explain why DNS entries appear to be 'stuck' - Samba is refusing to
>>>>> change anything in that partition, because it wrongly believes that
>>>>> someone else is the master for that data. 
>>>>>
>>>>> Andrew Bartlett
>>>>>
>>>> Hello Andrew,
>>>>
>>>> do you have an idea what needs to be changed ? Is it only the
>>>> DomainDnsZones and ForestDnsZones part or are there other places where
>>>> changes need to be made ? Yesterday I have tried to change the
>>>> DomainDnsZones stuff but got an error message when trying to commit the
>>>> modifications.
>>>
>>> That is what I was trying to explain.  The fact that the NTDS Settings
>>> for your DC lists these as partialReplica partitions is the cause of the
>>> problem. 
>>>
>>> We need to correct that in your instance, and if we find that many folks
>>> have run the buggy version of the samba_dnsupgrade script, we may need
>>> to add a special case to dbcheck for this.  I'm already thinking a
>>> schema compliance check would be very worthwhile, so this can be found
>>> before modifications are made.
>>>
>>> Andrew Bartlett
>>>
>> Hello Andrew,
>>
>> so, how should I proceed on from here ? What can I do to fix those issues ?
>>
>> best regards
>>
>> Andreas
> 
> Can you please run:
> 
> ldbsearch -H sam.ldb -s sub --cross-ncs objectclass=ntdsdsa
> 
> I need to see what is in your NTDS Setting entry for each DC so I can
> figure out how to fix this.
> 
> Thanks,
> 
> Andrew Bartlett
> 


Hello Andrew,

here is the output of  ../bin/ldbsearch -H sam.ldb -s sub --cross-ncs
objectclass=ntdsdsa


# record 1
dn: CN=NTDS
Settings,CN=NOVADC01,CN=Servers,CN=Standardname-des-ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
objectClass: top
objectClass: applicationSettings
objectClass: nTDSDSA
cn: NTDS Settings
instanceType: 4
whenCreated: 20120422134800.0Z
uSNCreated: 3212
dMDLocation: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
invocationId: 61f36cfd-ba7d-4702-87d3-7e861bb32cfe
showInAdvancedViewOnly: TRUE
name: NTDS Settings
objectGUID: c60bca82-df6e-409e-85c5-e2cc733691da
options: 1
systemFlags: 33554432
objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
msDS-Behavior-Version: 4
hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
hasMasterNCs: DC=novanetwork,DC=loc
msDS-HasDomainNCs: DC=novanetwork,DC=loc
msDS-HasInstantiatedNCs:
B:8:0000000D:DC=DomainDnsZones,DC=novanetwork,DC=loc
msDS-HasInstantiatedNCs:
B:8:0000000D:DC=ForestDnsZones,DC=novanetwork,DC=loc
msDS-hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
msDS-hasMasterNCs: DC=novanetwork,DC=loc
msDS-hasMasterNCs: DC=DomainDnsZones,DC=novanetwork,DC=loc
msDS-hasMasterNCs: DC=ForestDnsZones,DC=novanetwork,DC=loc
whenChanged: 20120422140342.0Z
uSNChanged: 4066
distinguishedName: CN=NTDS
Settings,CN=NOVADC01,CN=Servers,CN=Standardname-des
 -ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc

# record 2
dn: CN=NTDS
Settings,CN=NOVADC02,CN=Servers,CN=Standardname-des-ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc
objectClass: top
objectClass: applicationSettings
objectClass: nTDSDSA
cn: NTDS Settings
instanceType: 4
whenCreated: 20120503122809.0Z
hasMasterNCs: DC=novanetwork,DC=loc
hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
uSNCreated: 5326
dMDLocation: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
invocationId: b3ec35a6-d4c1-4f83-8ad6-1dcd330bd353
showInAdvancedViewOnly: TRUE
name: NTDS Settings
objectGUID: 94d1cf02-6aaf-41b7-928c-2292221525d8
options: 1
systemFlags: 33554432
objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
msDS-Behavior-Version: 4
msDS-HasDomainNCs: DC=novanetwork,DC=loc
msDS-hasMasterNCs: DC=novanetwork,DC=loc
msDS-hasMasterNCs: CN=Configuration,DC=novanetwork,DC=loc
msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
whenChanged: 20120503124935.0Z
hasPartialReplicaNCs: DC=DomainDnsZones,DC=novanetwork,DC=loc
hasPartialReplicaNCs: DC=ForestDnsZones,DC=novanetwork,DC=loc
uSNChanged: 5435
distinguishedName: CN=NTDS
Settings,CN=NOVADC02,CN=Servers,CN=Standardname-des
 -ersten-Standorts,CN=Sites,CN=Configuration,DC=novanetwork,DC=loc

# returned 2 records
# 2 entries
# 0 referrals

Thanks

best regards

Andreas