[Samba] Replication failures

Wed May 1 15:16:23 UTC 2019

Hai Mason, 

I had a look at the debug output. 

on 1) why around 15-16 second, that i really dont know. im trying to figure that out. 
on 2) if DNS is inconsistance, that everything is unrelayable. 
This is really the first the that needs fixing. 
then we look again at the replication. 

The debug output still shows several messages about zones in flat files. 
I still do believe also that this has impact on your problem. 

Your bind config is still not correct, set it exactly as i''ve dont in the howto. 
first get everything running error free, then, add you own setting to them. 

for example: You ad-dc managed zones still need  auth-nxdomain yes;

And  i'll have a look at the debug script again since i see it fails at the end. 

Im  failing to see the big picture here, how your setup is done .. and that does not happen often. :-/ 
(hint https://www.diagrameditor.com/ )  

What i suggest, or what i would do.  Verify all needed dns records per server, per host. 

I'll sleep a night over this and maybe i can come up with some more. 
Your problem is not your samba, but DNS settings and maybe an inheratence from the past.. 

Greetz, 

Louis

Van: M B [mailto:mmx at exm0.net] 
Verzonden: woensdag 1 mei 2019 14:44
Aan: L.P.H. van Belle; samba at lists.samba.org
Onderwerp: Re: [Samba] Replication failures

New observations: 1. "samba: task[kccsrv]" always goes to PANIC around 15-16 seconds after samba starts
2. I have three sites and the automatic" NTDS Settings" links between sites are not being generated consistently. I’ve had to manually create some NTDS Seting replication links, especially after I demote/rejoin any DC. I’m guessing the “kccsrv” process should manage these links automatically but it’s crashing so it can not create appropriate links. It seems that links within a site are created automatically, but not necessarily links between sites. I’ve seen links created automatically in some newly re-joined DCs, but not in existing DCs back to the newly re-joined DCs

samba-check-db-repl.sh output pasted below. I pasted results from only one DC. All others are similar. I do get some replication inconsistencies in DNS, but those go away if I run the script again as the differences get resolved

On May 1, 2019, at 2:25 AM, L.P.H. van Belle via samba <samba at lists.samba.org> wrote:

Hai Mason, 

-----Oorspronkelijk bericht-----
Van: M B [ MailScanner heeft een e-mail met mogelijk een poging tot fraude gevonden van "exm0.net" mailto:mmx at exm0.net] 
Verzonden: dinsdag 30 april 2019 20:42
Aan: L.P.H. van Belle; samba at lists.samba.org
Onderwerp: Re: [Samba] Replication failures

Hi Louis,

In the past few days I’ve removed all bind flat file configs 
from my environment, and I’ve checked carefully that all DCs 
are replicating and that all changes on any DC eventually 
replicate cleanly to all other DCs

Ok, so to confirm, your replication is ok now? 
If you think yes, then get en review the setting in this script. 
wget https://raw.githubusercontent.com/thctlo/samba4/master/samba-check-db-repl.sh 
Run it from every dc and post the outputs. 

I’ve checked resolv.conf on all the DCs as well and they all 
have at least two other IPs of other DC in them. I believe 
you said that the first IP should be the IP of the local 
host, but I haven’t done that on every server yet.

Yes, but you change that after the join and after you check replication is ok. 
What i always do is, join, reboot, check replication, change dns, reboot, and verify replication again. 
This order. 

I’m running dc4 on Ubuntu 18.04 using your samba packages. 
All other samba DCs are running 4.9.3 that I’ve compiled 
previously on Ubuntu 16.04. This same 4.9.3 package is 
running without any kcc errors or process PANICs on another 
site I manage.
Also, one DC is Windows 2008 R2 (WDC1)

Every time I start samba AD DC on 18.04 with your packages or 
on 16.04 with my own packages, the samba kccsvr ( ??????6615 
samba: task[kccsrv]  )  task starts with all other samba 
components and runs for about 10-12 seconds and then goes to 
PANIC and crashes as shown in the logs below. After that 
‘samba-tool drs showrepl’ always fails.

On the server, set log level = 10 
A pain yes, but i dont see directly whats wrong here. 
Before a log level 10 post, run on the DC with my packages this again.
https://github.com/thctlo/samba4/blob/master/samba-collect-debug-info.sh 
Pm me the unmodified output, i'll re-check that. 

What i suspect is a damaged AD or DNS or both. 
It's just hard to find, but if AD is replication now, it must be something in the DNS.
I can't tell jet. 

I don’t know how to tell if I’m using talloc/tdb from Samba 
source or from the OS. I believe it’s from source because I 
always compile on a new, clean system and I don’t install any 
talloc/tdb or samba packages to prepare the system for compile.

I’ve checked versions as you’ve requested. This version list 
is from DC4, with your packages.

This looks ok. 

This is from DC5 with my packages. You’ll note that this list 
shows "samba-common   2:4.3.11+dfsg-0ubuntu0.16.04.12” but 
this is only the folder structure and file structure created 
by 4.3.11 Ubuntu package. I found out the hard way that if I 
purge that package, it deletes my entire /var/lib/samba 
directory, so I had to re-build one of my DC’s from scratch. :(

Au, yes, the other option was to run : apt dist-upgrade 
What should have upgraded that package. 
Hard, but this way we learn quicker, and.. I know you feeling ;-) 

Here also left overs. In samba packages. 
The sources build does include tallec/tevent/tdb/ldb so you dont see these in the list. 
And i dont know how you create your samba 4.9.3 package so this is a bit hard to tell. 

I suggest, 
Stop samba, backup you /var/{lib,cache}/samba/  and /etc/samba 
apt remove --purge samba-common samba --autoremove 
And install the 4.9.3 back. 
Or, upgrade to ubuntu 18.04 and setup my 4.9 repo. 
Or use my repo and rebuild the packages for your own. 

Greetz, 

Louis

Typical output from script:

Running with with console output
Checking the DC_With_FSMO (dc1) with SAMBA DC: dc5.my.company.tld
dc4.my.company.tld
dc7.my.company.tld
dc6.my.company.tld
dc2.my.company.tld
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc5.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1713

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc4.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1713

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc7.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1713

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc6.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1714

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc2.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1714

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
.. Next check.. 
Running : samba-tool drs showrepl
grep -c "failed" /tmp/samba_drs_showrepl
grep -c "successful" /tmp/samba_drs_showrepl
         failures don't match
        successes don't match
         failures don't match
        successes don't match
         failures don't match
        successes don't match
         failures don't match
        successes don't match
         failures don't match
        successes don't match

if [ "${EMAIL_REPORT_ALWAYS}" = "yes" ] && [ -n "${EMAIL_REPORT_ADDRESS}" ]; then
    #cat /tmp/samba_drs_showrepl | ${SET_MAILTOOL} -s "SAMBA CHECK DB : showrepl results" $EMAIL_REPORT_ADDRESS
    ${SET_MAILTOOL} -s "SAMBA CHECK DB : showrepl results" $EMAIL_REPORT_ADDRESS < /tmp/samba_drs_showrepl
    #cat /tmp/samba_ldapcmp_checkdb | ${SET_MAILTOOL} -s "SAMBA CHECK DB : ldapcmp results" $EMAIL_REPORT_ADDRESS
    ${SET_MAILTOOL} -s "SAMBA CHECK DB : ldapcmp results" $EMAIL_REPORT_ADDRESS < /tmp/samba_ldapcmp_checkdb
fi

if [ "${SETREMOVELOG}" = "yes" ]; then
    if [ -f /tmp/samba_ldapcmp_checkdb ]; then
        rm /tmp/samba_ldapcmp_checkdb
    fi
    if [ -f /tmp/samba_drs_showrepl ]; then
        rm /tmp/samba_drs_showrepl
    fi
fi