[Samba] samba4 replication issues | sam.ldb inconsistency

[Andrew Bartlett]

On Tue, 2014-07-08 at 17:58 +0200, mourik jan heupink - merit wrote:
> Hi all,
> 
> We seem to have some issues with our samba4 ad setup. I posted about 
> this last week already but had received no replies at all so far. :-(

If you urgently need help, please contact a Samba commercial support
provider with experience in the AD DC:

https://www.samba.org/samba/support/globalsupport.html

> What is our situation:
> 
> two domain controllers (dc1 and dc2), one (separate) fileserver, all 
> running sernet-4.1.7. From the workstations perspective, everything is 
> running as it should, there appear to be no issues.
> 
> However: something in my replication has gone wrong... on dc2:
> 
> ==== INBOUND NEIGHBORS ====
> 
> DC=DomainDnsZones,DC=samba,DC=company,DC=com
>          Default-First-Site-Name\DC1 via RPC
>                  DSA object GUID: 81a27497-bdfb-4977-9874-675bbfba490f
>                  Last attempt @ Tue Jul  8 17:12:09 2014 CEST failed, 
> result 8442 (WERR_DS_DRA_INTERNAL_ERROR)
>                  3252 consecutive failure(s).
>                  Last success @ Tue Jul  1 16:34:57 2014 CEST
> 
> CN=Configuration,DC=samba,DC=company,DC=com
>          Default-First-Site-Name\DC1 via RPC
>                  DSA object GUID: 81a27497-bdfb-4977-9874-675bbfba490f
>                  Last attempt @ Tue Jul  8 17:12:10 2014 CEST was successful
>                  0 consecutive failure(s).
>                  Last success @ Tue Jul  8 17:12:10 2014 CEST
> (the rest all replicates succesfully)
> 
> Then, to verify integrity of DC=DomainDnsZones on dc1, I type:
> 
> root at dc1:/var/log/samba# samba-tool dbcheck --cross-ncs
> ltdb: 
> tdb(/var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SAMBA,DC=COMPANY,DC=COM.ldb): 
> tdb_rec_read bad magic 0x198 at offset=1044437120
> ERROR(ldb): uncaught exception - Indexed and full searches both failed!

This implies very serious corruption of this tdb (ldb) file. 

> On dc2 the same "samba-tool dbcheck cross-ncs" says: "checking 187478 
> objects". Has been running for many hours now, I have no idea how far it 
> is. The server is pretty buzy doing it.

This is quite likely, as dbcheck is fairly intensive and the internal
DNS bug regarding deleted objects means we get a *lot* of records.  It
probably is still making progress however. 

Perhaps see the suggestions elsewhere on this list for purging the DNS
records after only 1 month. 

> So, my working conclusion is that on DC1 the 
> DC=DomainDnsZones,DC=samba,DC=company,DC=com has become corrupted, and 
> therefore fails to replicate to dc2.
> 
> Does the list agree with this?

Yes.

> I hope that dc2 is still having the correct DC=DomainDnsZones. But, 
> since replication seems to be only from dc1 TO dc2, I'm unsure how to 
> import the healthy dc2 database into dc1.
> 
> Does the above make any sense? How to make both dc's happy and fully 
> functional again?
> 
> Any help would be VERY much appreciated... Hopefully I'll get some 
> replies this time!

This is a difficult situation.  Ideally you would get the 'good' DC to
replicate to a new installation, and work from there.

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba