[Samba] Replication failures

M B mmx at exm0.net
Wed May 1 12:43:31 UTC 2019 

New observations:
1. "samba: task[kccsrv]" always goes to PANIC around 15-16 seconds after samba starts
2. I have three sites and the automatic" NTDS Settings" links between sites are not being generated consistently. I’ve had to manually create some NTDS Seting replication links, especially after I demote/rejoin any DC. I’m guessing the “kccsrv” process should manage these links automatically but it’s crashing so it can not create appropriate links. It seems that links within a site are created automatically, but not necessarily links between sites. I’ve seen links created automatically in some newly re-joined DCs, but not in existing DCs back to the newly re-joined DCs

samba-check-db-repl.sh output pasted below. I pasted results from only one DC. All others are similar. I do get some replication inconsistencies in DNS, but those go away if I run the script again as the differences get resolved


> On May 1, 2019, at 2:25 AM, L.P.H. van Belle via samba <samba at lists.samba.org> wrote:
> 
> Hai Mason, 
> 
> 
>> -----Oorspronkelijk bericht-----
>> Van: M B [mailto:mmx at exm0.net <mailto:mmx at exm0.net>] 
>> Verzonden: dinsdag 30 april 2019 20:42
>> Aan: L.P.H. van Belle; samba at lists.samba.org <mailto:samba at lists.samba.org>
>> Onderwerp: Re: [Samba] Replication failures
>> 
>> Hi Louis,
>> 
>> In the past few days I’ve removed all bind flat file configs 
>> from my environment, and I’ve checked carefully that all DCs 
>> are replicating and that all changes on any DC eventually 
>> replicate cleanly to all other DCs
> 
> Ok, so to confirm, your replication is ok now? 
> If you think yes, then get en review the setting in this script. 
> wget https://raw.githubusercontent.com/thctlo/samba4/master/samba-check-db-repl.sh <https://raw.githubusercontent.com/thctlo/samba4/master/samba-check-db-repl.sh> 
> Run it from every dc and post the outputs. 
> 
>> 
>> I’ve checked resolv.conf on all the DCs as well and they all 
>> have at least two other IPs of other DC in them. I believe 
>> you said that the first IP should be the IP of the local 
>> host, but I haven’t done that on every server yet.
> 
> Yes, but you change that after the join and after you check replication is ok. 
> What i always do is, join, reboot, check replication, change dns, reboot, and verify replication again. 
> This order. 
> 
>> 
>> I’m running dc4 on Ubuntu 18.04 using your samba packages. 
>> All other samba DCs are running 4.9.3 that I’ve compiled 
>> previously on Ubuntu 16.04. This same 4.9.3 package is 
>> running without any kcc errors or process PANICs on another 
>> site I manage.
>> Also, one DC is Windows 2008 R2 (WDC1)
>> 
>> Every time I start samba AD DC on 18.04 with your packages or 
>> on 16.04 with my own packages, the samba kccsvr ( ??????6615 
>> samba: task[kccsrv]  )  task starts with all other samba 
>> components and runs for about 10-12 seconds and then goes to 
>> PANIC and crashes as shown in the logs below. After that 
>> ‘samba-tool drs showrepl’ always fails.
> 
> On the server, set log level = 10 
> A pain yes, but i dont see directly whats wrong here. 
> Before a log level 10 post, run on the DC with my packages this again.
> https://github.com/thctlo/samba4/blob/master/samba-collect-debug-info.sh <https://github.com/thctlo/samba4/blob/master/samba-collect-debug-info.sh> 
> Pm me the unmodified output, i'll re-check that. 
> 
> What i suspect is a damaged AD or DNS or both. 
> It's just hard to find, but if AD is replication now, it must be something in the DNS.
> I can't tell jet. 
> 
>> 
>> I don’t know how to tell if I’m using talloc/tdb from Samba 
>> source or from the OS. I believe it’s from source because I 
>> always compile on a new, clean system and I don’t install any 
>> talloc/tdb or samba packages to prepare the system for compile.
>> 
>> I’ve checked versions as you’ve requested. This version list 
>> is from DC4, with your packages.
>> 
>> ubuntu at dc4:~$ dpkg -l |egrep 
>> "samba|winbin|?db|tevent|talloc|nss|wrapper"
>> ii  dbus                                  1.12.2-1ubuntu1     
>>              amd64        simple interprocess messaging 
>> system (daemon and utilities)
>> .... Shorted this a bit. 
>> 2018.05.09-0ubuntu1~18.04.1       all          wireless 
>> regulatory database
>> 
> 
> This looks ok. 
> 
>> 
>> This is from DC5 with my packages. You’ll note that this list 
>> shows "samba-common   2:4.3.11+dfsg-0ubuntu0.16.04.12” but 
>> this is only the folder structure and file structure created 
>> by 4.3.11 Ubuntu package. I found out the hard way that if I 
>> purge that package, it deletes my entire /var/lib/samba 
>> directory, so I had to re-build one of my DC’s from scratch. :(
> 
> Au, yes, the other option was to run : apt dist-upgrade 
> What should have upgraded that package. 
> Hard, but this way we learn quicker, and.. I know you feeling ;-) 
> 
> 
>> ==
>> ubuntu at dc5:~$ dpkg -l |egrep 
>> "samba|winbin|?db|tevent|talloc|nss|wrapper"
>> ii  dbus                                  1.10.6-1ubuntu3.3   
>>                       amd64        simple interprocess 
> ....
>> 2018.05.09-0ubuntu1~16.04.1                all          
>> wireless regulatory database
> 
> Here also left overs. In samba packages. 
> The sources build does include tallec/tevent/tdb/ldb so you dont see these in the list. 
> And i dont know how you create your samba 4.9.3 package so this is a bit hard to tell. 
> 
> I suggest, 
> Stop samba, backup you /var/{lib,cache}/samba/  and /etc/samba 
> apt remove --purge samba-common samba --autoremove 
> And install the 4.9.3 back. 
> Or, upgrade to ubuntu 18.04 and setup my 4.9 repo. 
> Or use my repo and rebuild the packages for your own. 
> 
> 
> 
> Greetz, 
> 
> Louis
> 
> 
> 
Typical output from script:

Running with with console output
Checking the DC_With_FSMO (dc1) with SAMBA DC: dc5.my.company.tld
dc4.my.company.tld
dc7.my.company.tld
dc6.my.company.tld
dc2.my.company.tld
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc5.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1713

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc4.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1713

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc7.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1713

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc6.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1714

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld ldap://dc2.my.company.tld 
Please wait.. this can take a while..
cat /tmp/samba_ldapcmp_checkdb

* Comparing [DOMAIN] context...

* Objects to be compared: 1321

* Result for [DOMAIN]: SUCCESS

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1714

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1550

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 1691

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 49

* Result for [DNSFOREST]: SUCCESS
.. Next check.. 
Running : samba-tool drs showrepl
grep -c "failed" /tmp/samba_drs_showrepl
grep -c "successful" /tmp/samba_drs_showrepl
         failures don't match
        successes don't match
         failures don't match
        successes don't match
         failures don't match
        successes don't match
         failures don't match
        successes don't match
         failures don't match
        successes don't match

if [ "${EMAIL_REPORT_ALWAYS}" = "yes" ] && [ -n "${EMAIL_REPORT_ADDRESS}" ]; then
    #cat /tmp/samba_drs_showrepl | ${SET_MAILTOOL} -s "SAMBA CHECK DB : showrepl results" $EMAIL_REPORT_ADDRESS
    ${SET_MAILTOOL} -s "SAMBA CHECK DB : showrepl results" $EMAIL_REPORT_ADDRESS < /tmp/samba_drs_showrepl
    #cat /tmp/samba_ldapcmp_checkdb | ${SET_MAILTOOL} -s "SAMBA CHECK DB : ldapcmp results" $EMAIL_REPORT_ADDRESS
    ${SET_MAILTOOL} -s "SAMBA CHECK DB : ldapcmp results" $EMAIL_REPORT_ADDRESS < /tmp/samba_ldapcmp_checkdb
fi

if [ "${SETREMOVELOG}" = "yes" ]; then
    if [ -f /tmp/samba_ldapcmp_checkdb ]; then
        rm /tmp/samba_ldapcmp_checkdb
    fi
    if [ -f /tmp/samba_drs_showrepl ]; then
        rm /tmp/samba_drs_showrepl
    fi
fi

More information about the samba mailing list