[Samba] Initial replication halts with "The handle is invalid." (msDS-NC-Replica-Locations corrupted?)

[Andrew Bartlett]

On Mon, 2017-01-16 at 14:03 -0500, Adam Tauno Williams via samba wrote:
> Quoting Adam Tauno Williams via samba <samba at lists.samba.org>:
> > Quoting Rowland Penny via samba <samba at lists.samba.org>:
> > > > samba-tool's dbcheck finds only two errors in cn=Configuration,
> > > > but it
> > > > does not repair them.  These appear to be references to an
> > > > original,
> > > > long since demoted, DC.  But these values appear in neither the
> > > > results of an ldbsearch or via ADSI edit.
> > > 
> > > Have a look at this bug report:
> > > https://bugzilla.samba.org/show_bug.cgi?id=12385
> > 
> > Thanks.  I have checked out master and built it [**1];  Bug#12297  
> > [**2] indicates I can run samba-tool from the build of the
> > checkout.  
> >  I assume tombstone expunge, even for CN=Configuration, runs
> > against  
> > sam.ldb???  So the correct syntax is -
> > [ADC28 samba]# bin/samba-tool domain tombstones expunge -H  
> > /var/lib/samba/private/sam.ldb
> 
> 
> This does not appear to have reaped any links.
> 
> 
> [root at larkin28 samba]# bin/samba-tool domain tombstones expunge -d
> 10  
> -H /var/lib/samba/private/sam.ldb
> ...
> lpcfg_servicenumber: couldn't find ldb
> Initial schema load needed, as we have no existing schema, seq_num:
> 64
> schema_fsmo_init: we are master[no] updates allowed[no]
> Initial schema load needed, as we have no existing schema, seq_num:
> 64
> schema_fsmo_init: we are master[no] updates allowed[no]
> gendb_search_v: CN=Directory Service,CN=Windows  
> NT,CN=Services,CN=Configuration,DC=micore,DC=us  
> objectClass=nTDSService -> 1
> Doing a full scan on CN=Configuration,DC=micore,DC=us and looking
> for  
> deleted objects
> Doing a full scan on DC=micore,DC=us and looking for deleted objects
> Doing a full scan on DC=DomainDnsZones,DC=micore,DC=us and looking
> for  
> deleted objects
> Doing a full scan on DC=ForestDnsZones,DC=micore,DC=us and looking
> for  
> deleted objects
> Removed 0 objects and 0 links successfully

Thanks.  When you run 
bin/samba-tool dbcheck -H /var/lib/samba/private/sam.ldb --cross-ncs
is it less scary now?  We tried to remove some of the scare factor for
this one, it is 'mostly harmless'.

In your earlier mail you said the output was:

[root at larkin27 ~]# samba-tool dbcheck --cross-ncs --fix --yes
> Checking 8411 objects
> ERROR: no target object found for GUID component for msDS-NC-Replica-
> Locations in object CN=3ad6381a-9725-4e28-8157-
> a5a3fde68a43,CN=Partitions,CN=Configuration,DC=micore,DC=us -
> <GUID=7d3f95a5cdfa1246b1fb2fcd16e5f877>;<RMD_ADDTIME=1300000651000000
> 00
> > ;<RMD_CHANGETIME=130000065100000000>;<RMD_FLAGS=0>;<RMD_INVOCID=01d
> > b57> 
> fd8d4ddd469aee9cbd36abb3e1>;<RMD_LOCAL_USN=5160>;<RMD_ORIGINATING_USN
> =3
> 630>;<RMD_VERSION=0>;CN=NTDS
> Settings,CN=BARBEL,CN=Servers,CN=Default-
> First-Site-Name,CN=Sites,CN=Configuration,DC=micore,DC=us
> Not removing dangling forward link
> ERROR: no target object found for GUID component for msDS-NC-Replica-
> Locations in object CN=55b4d7f1-b1b1-4843-ae00-
> 7908adf44ffa,CN=Partitions,CN=Configuration,DC=micore,DC=us -
> <GUID=7d3f95a5cdfa1246b1fb2fcd16e5f877>;<RMD_ADDTIME=1300000651000000
> 00
> > ;<RMD_CHANGETIME=130000065100000000>;<RMD_FLAGS=0>;<RMD_INVOCID=01d
> > b57> 
> fd8d4ddd469aee9cbd36abb3e1>;<RMD_LOCAL_USN=5134>;<RMD_ORIGINATING_USN
> =3
> 629>;<RMD_VERSION=0>;CN=NTDS
> Settings,CN=BARBEL,CN=Servers,CN=Default-
> First-Site-Name,CN=Sites,CN=Configuration,DC=micore,DC=us
> Not removing dangling forward link

So, what is happening here is that we have a non-deleted link to an
object that does not exist.  We have so far been very conservative in
removing such links, because logic does exist to remove them when the
target object is deleted.  We are concerned they may point at something
that is yet to be replicated (eg if you replicated with --critical-
objects-only).  They may point outside our current replicated partition
(once we get subdomains), and in general they are not something we are
quite as clear as being totally redundant.  (ie, we don't want to
automatically delete possible information)

The tombstones expunge command above is related to a slightly different
issue, which is that in the past, when we deleted an object, we deleted
(marked with RMD_FLAGS=1, rather than just removed) links to the
deleted object (which it self was marked with isDeleted=TRUE and had
\0DEL:GUID put in the name).

Time would pass and in 180 days the target object would be deleted, but
until 4.5.2 the link wouldn't be removed.  Also with 4.5.0, we started
checking more of these links, not just ignoring them.  So the reason
the tombstones expunge was suggested is that IF it was RMD_FLAGS=1, and
deleted at the same time as the target, the expunge would also find it
180 days old, and tidy the link away... 

Now, the fact that you have replication failures suggests this is a
very real problem for you.  It may or may not actually be this object,
but clearly you want to try.  I'll try and get you a command to remove
the links with the 'vanish links' control, so you can test it out on a
backup.  

Thanks,,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba