[Samba] Invalid data for index DN=@INDEX:OBJECTCLASS:DNSNODE

Fri May 13 14:41:53 UTC 2016

On 13/05/16 14:49, ash-samba at comtek.co.uk wrote:
>
> We have a Samba primary domain controller "empire", which seems to have
> DNS update issues. We can seem to query all records on empire just fine,
> and we can modify IPs for existing records, but it will not delete or
> add new records. Attempting to delete via the AD tools shows "Local
> security authority database contains an internal inconsistency". Adding
> a record on the command line shows:
>
>> samba-tool  dns add empire chester-dc.example.com p-bats A 10.4.4.141
> -U ash
>> Password for [CHESTER-DC\ash]:
>> ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR')
>>   File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
> line 175, in _run
>>     return self.run(*args, **kwargs)
>>   File "/usr/lib/python2.7/dist-packages/samba/netcmd/dns.py", line
> 1067, in run
>>     0, server, zone, name, add_rec_buf, None)
>
> We have two other DCs (hawaii and alaska), but we are reluctant to
> switch to them, since they are located in another country, and have an
> unreliable high latency link. The other two DCs accept DNS record
> additions/deletions.
>
> Our plan was to set up a 4th DC locally (v-ward), and ultimately make
> that the primary server. Unfortunately, this results in:
>
>>
>> samba-tool domain join chester-dc.example.com DC -Uash
> --realm=CHESTER-DC.EXAMPLE.COM
>> Finding a writeable DC for domain 'chester-dc.example.com'
>> Found DC empire.chester-dc.example.com
>> Password for [CHESTER-DC\ash]:
>> workgroup is CHESTER-DC
>> realm is chester-dc.example.com
>> checking sAMAccountName
>> Adding CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=example,DC=com
>> Adding
> CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com 
>
>> Adding CN=NTDS
> Settings,CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com 
>
>> Adding SPNs to CN=V-WARD,OU=Domain
> Controllers,DC=chester-dc,DC=example,DC=com
>> Setting account password for V-WARD$
>> Enabling account
>> Calling bare provision
>> No IPv6 address will be assigned
>> Provision OK for domain DN DC=chester-dc,DC=example,DC=com
>> Starting replication
>> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[402/1550] linked_values[0/0]
>> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[804/1550] linked_values[0/0]
>> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[1206/1550] linked_values[0/0]
>> Schema-DN[CN=Schema,CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[1550/1550] linked_values[0/0]
>> Analyze and apply schema objects
>> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[402/1634] linked_values[0/0]
>> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[804/1634] linked_values[0/0]
>> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[1206/1634] linked_values[0/0]
>> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[1608/1634] linked_values[0/0]
>> Partition[CN=Configuration,DC=chester-dc,DC=example,DC=com]
> objects[1634/1634] linked_values[53/0]
>> Replicating critical objects from the base DN of the domain
>> Partition[DC=chester-dc,DC=example,DC=com] objects[100/100]
> linked_values[39/0]
>> Partition[DC=chester-dc,DC=example,DC=com] objects[502/723]
> linked_values[0/0]
>> Partition[DC=chester-dc,DC=example,DC=com] objects[823/723]
> linked_values[988/0]
>> Done with always replicated NC (base, config, schema)
>> Replicating DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[402/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[804/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[1206/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[1608/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[2010/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[2412/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[2814/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[3216/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[3618/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[4020/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[4422/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[4824/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[5226/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[5628/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[6030/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[6432/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[6834/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[7236/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[7638/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[8040/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[8442/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[8844/9093] linked_values[0/0]
>> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[9093/9093] linked_values[0/0]
>> Replicating DC=ForestDnsZones,DC=chester-dc,DC=example,DC=com
>> Partition[DC=ForestDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[27/27] linked_values[0/0]
>> Partition[DC=ForestDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[54/27] linked_values[0/0]
>> Committing SAM database
>> descriptor_modify: Could not find SD for
> DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com
>>
>> Join failed - cleaning up
>> checking sAMAccountName
>> Deleted CN=V-WARD,OU=Domain Controllers,DC=chester-dc,DC=example,DC=com
>> Deleted CN=NTDS
> Settings,CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com 
>
>> Deleted
> CN=V-WARD,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=chester-dc,DC=example,DC=com 
>
>> ERROR(ldb): uncaught exception - operations error at
> ../source4/dsdb/samdb/ldb_modules/descriptor.c:1147
>>   File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
> line 175, in _run
>>     return self.run(*args, **kwargs)
>>   File "/usr/lib/python2.7/dist-packages/samba/netcmd/domain.py",
> line 555, in run
>>     machinepass=machinepass, use_ntvfs=use_ntvfs,
> dns_backend=dns_backend)
>>   File "/usr/lib/python2.7/dist-packages/
>
> I have noticed that the the DNS ldb file is rather large (300M):
>
>> total 347988
>> -rw------- 1 root root  10383360 May 13 14:13
> CN%3DCONFIGURATION,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
>> -rw------- 1 root root  10383360 May 13 14:13
> CN%3DSCHEMA,CN%3DCONFIGURATION,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
>> -rw------- 1 root root  17158144 May 13 14:13
> DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
>> -rw------- 1 root root 313745408 May 13 14:13
> DC%3DDOMAINDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
>> -rw------- 1 root root   4247552 May 13 14:13
> DC%3DFORESTDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb
>> -rw-r----- 1 root root    421888 May 13 14:09 metadata.tdb
>
> Investigating further:
>
>> 0 root at empire:~[0] /usr/bin/samba-tool drs replicate
> empire.chester-dc.example.com alaska.chester-dc.example.com
> DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com --local
> Partition[DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com]
> objects[402/15688] linked_values[0/0]
>> Invalid data for index DN=@INDEX:OBJECTCLASS:DNSNODE
>>
>> replmd_replicated_request rename
> DC=DEELR013\0ADEL:08ae6b71-9b11-4003-9daf-f2e2ed3a58be,CN=Deleted
> Objects,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com =>
> DC=DEELR013\0ACNF:08ae6b71-9b11-4003-9daf-f2e2ed3a58be\0ADEL:08ae6b71-9b11-4003-9daf-f2e2ed3a58be,CN=Deleted 
>
> Objects,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com failed -
> ldb_wait: > Operations error (1)
>>
>> Failed to apply records: ldb_wait: Operations error (1): Other
>> Failed to commit objects: WERR_GENERAL_FAILURE
>> ERROR(<type 'exceptions.TypeError'>): Error replicating DN
> DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com - Failed to process
> chunk: NT_STATUS_UNSUCCESSFUL
>>   File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
> 269, in drs_local_replicate
>>     repl.replicate(NC, source_dsa_invocation_id, destination_dsa_guid)
>>   File "/usr/lib/python2.7/dist-packages/samba/drs_utils.py", line
> 256, in replicate
>>     schema=schema, req_level=req_level, req=req)
>
> This pointed us at the DEELR013 record, so, I tried:
>
>> 0 root at empire:~[0] ldbdel -H
> /var/lib/samba/private/sam.ldb.d/DC%3DDOMAINDNSZONES,DC%3DCHESTER-DC,DC%3DEXAMPLE,DC%3DCOM.ldb 
>
> DC=DEELR013,DC=chester-dc.example.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com 
>
>> Invalid data for index DN=@INDEX:OBJECTCLASS:DNSNODE
>>
>> delete of
> 'DC=DEELR013,DC=chester-dc.example.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=chester-dc,DC=example,DC=com' 
>
> failed - (Operations error) ldb_wait: Operations error (1)
>>
>
> Finally, stumbling around blindly I ran tdbbackup on the DOMAINDNSZONES
> ldb file (which shrunk a few megabytes - no errors though), and I
> managed to ldbedit and delete the file index, then it allowed me to
> ldbdel. I Copied the newly modified file on top of the original one,
> restarted Samba, and at that point I realised that the file was now over
> 700mb. Samba had hung and stopped accepting connections (I couldn't even
> get a share list with smbclient). Unfortunately I can't give accurate
> detail about this paragraph, because I rolled back to last night's LXC
> snapshot.
>
> Can anybody please give us advice on how to proceed from here?
>
>> 0 root at empire:~[0] samba-tool -V
>> 4.1.11-Debian
>> 0 root at empire:~[0] dpkg -s samba |grep ^Ver
>> Version: 2:4.1.11+dfsg-1
>> 0 root at empire:~[0] uname -a
>> Linux empire 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
> x86_64 GNU/Linux
>
>

First things first, is there anyway you can update Samba ?
The 4.1.X series is now EOL and wasn't patched for badlock, depending on 
what version of debian you are running, you should be able to upgrade 
easily.

Please do not alter the ldb under sam.ldb.d directly, only modify the 
sam.ldb file (this contains everything in sam.ldb.d)

With AD, there is no such thing as a primary domain controller, all DCs 
are equal, the only difference is in which DC has the FSMO roles and 
these do not need to be all on the same DC. I mention this because it 
can get confusing when/if somebody asks a question about an NT-style PDC 
problem.

Your domain zone growing in size is probably down to tombstone objects, 
try searching on 'samba tombstone' for help on this.

Have you tried running 'samba-tool dbcheck' ??

Rowland