[Samba] Replication failures

Fri May 3 07:30:41 UTC 2019

Hai Mason, 

Good to see you found something here. 
I've lookup that part .. and asked for a small update on that bug report. 

> Apparently this zone scavenging is not compatible with my setup. 
No, as said, first you need to fix/setup your dns. 

Ofcourse thats up to you, but it can bit you in a the long run, if you dont change it. 
Also, the zone scavaening is only supported on new setups domains/zones. 

I've tested this in my production also, and yep, we have a bug here. 
Same crashes, but i have also old zones. 

I'll go search if there is an bugreport on this. 

Greetz, 

Louis

> -----Oorspronkelijk bericht-----
> Van: M B [mailto:mmx at exm0.net] 
> Verzonden: vrijdag 3 mei 2019 8:23
> Aan: L.P.H. van Belle
> CC: samba at lists.samba.org
> Onderwerp: Re: [Samba] Replication failures
> 
> This line in smb.conf was causing "samba: task[kccsrv]” to 
> PANIC and crash after 15-16 seconds.
> 
> When I remove this line, the kccsrv is stable again and 
> “samba-tool drs showrepl” gives the normal output.
> 
> “dns zone scavenging = yes”
> 
> Apparently this zone scavenging is not compatible with my setup. 
> 
> This also partially answers my other post about zone 
> scavenging on domains that were set up prior to samba version 4.9
> 
> 
> 
> 
> > On May 1, 2019, at 8:16 AM, L.P.H. van Belle via samba 
> <samba at lists.samba.org> wrote:
> > 
> > Hai Mason, 
> >  
> > I had a look at the debug output. 
> >  
> > on 1) why around 15-16 second, that i really dont know. im 
> trying to figure that out. 
> > on 2) if DNS is inconsistance, that everything is unrelayable. 
> > This is really the first the that needs fixing. 
> > then we look again at the replication. 
> >  
> > The debug output still shows several messages about zones 
> in flat files. 
> > I still do believe also that this has impact on your problem. 
> >  
> > Your bind config is still not correct, set it exactly as 
> i''ve dont in the howto. 
> > first get everything running error free, then, add you own 
> setting to them. 
> >  
> > for example: You ad-dc managed zones still need  auth-nxdomain yes;
> >  
> > And  i'll have a look at the debug script again since i see 
> it fails at the end. 
> >  
> > Im  failing to see the big picture here, how your setup is 
> done .. and that does not happen often. :-/ 
> > (hint https://www.diagrameditor.com/ )  
> > 
> >  
> > What i suggest, or what i would do.  Verify all needed dns 
> records per server, per host. 
> >  
> > I'll sleep a night over this and maybe i can come up with 
> some more. 
> > Your problem is not your samba, but DNS settings and maybe 
> an inheratence from the past.. 
> >  
> > Greetz, 
> >  
> > Louis
> >  
> >  
> >  
> > 
> > Van: M B [mailto:mmx at exm0.net] 
> > Verzonden: woensdag 1 mei 2019 14:44
> > Aan: L.P.H. van Belle; samba at lists.samba.org
> > Onderwerp: Re: [Samba] Replication failures
> > 
> > 
> > 
> > New observations: 1. "samba: task[kccsrv]" always goes to 
> PANIC around 15-16 seconds after samba starts
> > 2. I have three sites and the automatic" NTDS Settings" 
> links between sites are not being generated consistently. 
> I’ve had to manually create some NTDS Seting replication 
> links, especially after I demote/rejoin any DC. I’m guessing 
> the “kccsrv” process should manage these links automatically 
> but it’s crashing so it can not create appropriate links. It 
> seems that links within a site are created automatically, but 
> not necessarily links between sites. I’ve seen links created 
> automatically in some newly re-joined DCs, but not in 
> existing DCs back to the newly re-joined DCs
> > 
> > 
> > samba-check-db-repl.sh output pasted below. I pasted 
> results from only one DC. All others are similar. I do get 
> some replication inconsistencies in DNS, but those go away if 
> I run the script again as the differences get resolved
> > 
> > 
> > On May 1, 2019, at 2:25 AM, L.P.H. van Belle via samba 
> <samba at lists.samba.org> wrote:
> > 
> > Hai Mason, 
> > 
> > 
> > -----Oorspronkelijk bericht-----
> > Van: M B [ MailScanner heeft een e-mail met mogelijk een 
> poging tot fraude gevonden van "exm0.net" mailto:mmx at exm0.net] 
> > Verzonden: dinsdag 30 april 2019 20:42
> > Aan: L.P.H. van Belle; samba at lists.samba.org
> > Onderwerp: Re: [Samba] Replication failures
> > 
> > Hi Louis,
> > 
> > In the past few days I’ve removed all bind flat file configs 
> > from my environment, and I’ve checked carefully that all DCs 
> > are replicating and that all changes on any DC eventually 
> > replicate cleanly to all other DCs
> > 
> > Ok, so to confirm, your replication is ok now? 
> > If you think yes, then get en review the setting in this script. 
> > wget 
> https://raw.githubusercontent.com/thctlo/samba4/master/samba-c
> heck-db-repl.sh 
> > Run it from every dc and post the outputs. 
> > 
> > 
> > I’ve checked resolv.conf on all the DCs as well and they all 
> > have at least two other IPs of other DC in them. I believe 
> > you said that the first IP should be the IP of the local 
> > host, but I haven’t done that on every server yet.
> > 
> > Yes, but you change that after the join and after you check 
> replication is ok. 
> > What i always do is, join, reboot, check replication, 
> change dns, reboot, and verify replication again. 
> > This order. 
> > 
> > 
> > I’m running dc4 on Ubuntu 18.04 using your samba packages. 
> > All other samba DCs are running 4.9.3 that I’ve compiled 
> > previously on Ubuntu 16.04. This same 4.9.3 package is 
> > running without any kcc errors or process PANICs on another 
> > site I manage.
> > Also, one DC is Windows 2008 R2 (WDC1)
> > 
> > Every time I start samba AD DC on 18.04 with your packages or 
> > on 16.04 with my own packages, the samba kccsvr ( ??????6615 
> > samba: task[kccsrv]  )  task starts with all other samba 
> > components and runs for about 10-12 seconds and then goes to 
> > PANIC and crashes as shown in the logs below. After that 
> > ‘samba-tool drs showrepl’ always fails.
> > 
> > On the server, set log level = 10 
> > A pain yes, but i dont see directly whats wrong here. 
> > Before a log level 10 post, run on the DC with my packages 
> this again.
> > 
> https://github.com/thctlo/samba4/blob/master/samba-collect-deb
> ug-info.sh 
> > Pm me the unmodified output, i'll re-check that. 
> > 
> > What i suspect is a damaged AD or DNS or both. 
> > It's just hard to find, but if AD is replication now, it 
> must be something in the DNS.
> > I can't tell jet. 
> > 
> > 
> > I don’t know how to tell if I’m using talloc/tdb from Samba 
> > source or from the OS. I believe it’s from source because I 
> > always compile on a new, clean system and I don’t install any 
> > talloc/tdb or samba packages to prepare the system for compile.
> > 
> > I’ve checked versions as you’ve requested. This version list 
> > is from DC4, with your packages.
> > 
> > ubuntu at dc4:~$ dpkg -l |egrep 
> > "samba|winbin|?db|tevent|talloc|nss|wrapper"
> > ii  dbus                                  1.12.2-1ubuntu1     
> >              amd64        simple interprocess messaging 
> > system (daemon and utilities)
> > .... Shorted this a bit. 
> > 2018.05.09-0ubuntu1~18.04.1       all          wireless 
> > regulatory database
> > 
> > 
> > This looks ok. 
> > 
> > 
> > This is from DC5 with my packages. You’ll note that this list 
> > shows "samba-common   2:4.3.11+dfsg-0ubuntu0.16.04.12” but 
> > this is only the folder structure and file structure created 
> > by 4.3.11 Ubuntu package. I found out the hard way that if I 
> > purge that package, it deletes my entire /var/lib/samba 
> > directory, so I had to re-build one of my DC’s from scratch. :(
> > 
> > Au, yes, the other option was to run : apt dist-upgrade 
> > What should have upgraded that package. 
> > Hard, but this way we learn quicker, and.. I know you feeling ;-) 
> > 
> > 
> > ==
> > ubuntu at dc5:~$ dpkg -l |egrep 
> > "samba|winbin|?db|tevent|talloc|nss|wrapper"
> > ii  dbus                                  1.10.6-1ubuntu3.3   
> >                       amd64        simple interprocess 
> > ....
> > 2018.05.09-0ubuntu1~16.04.1                all          
> > wireless regulatory database
> > 
> > Here also left overs. In samba packages. 
> > The sources build does include tallec/tevent/tdb/ldb so you 
> dont see these in the list. 
> > And i dont know how you create your samba 4.9.3 package so 
> this is a bit hard to tell. 
> > 
> > I suggest, 
> > Stop samba, backup you /var/{lib,cache}/samba/  and /etc/samba 
> > apt remove --purge samba-common samba --autoremove 
> > And install the 4.9.3 back. 
> > Or, upgrade to ubuntu 18.04 and setup my 4.9 repo. 
> > Or use my repo and rebuild the packages for your own. 
> > 
> > 
> > 
> > Greetz, 
> > 
> > Louis
> > 
> > 
> > 
> > 
> > Typical output from script:
> > 
> > 
> > Running with with console output
> > Checking the DC_With_FSMO (dc1) with SAMBA DC: dc5.my.company.tld
> > dc4.my.company.tld
> > dc7.my.company.tld
> > dc6.my.company.tld
> > dc2.my.company.tld
> > Running : /usr/bin/samba-tool ldapcmp 
> --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld 
> ldap://dc5.my.company.tld 
> > Please wait.. this can take a while..
> > cat /tmp/samba_ldapcmp_checkdb
> > 
> > 
> > * Comparing [DOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1321
> > 
> > 
> > * Result for [DOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [CONFIGURATION] context...
> > 
> > 
> > * Objects to be compared: 1713
> > 
> > 
> > * Result for [CONFIGURATION]: SUCCESS
> > 
> > 
> > * Comparing [SCHEMA] context...
> > 
> > 
> > * Objects to be compared: 1550
> > 
> > 
> > * Result for [SCHEMA]: SUCCESS
> > 
> > 
> > * Comparing [DNSDOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1691
> > 
> > 
> > * Result for [DNSDOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [DNSFOREST] context...
> > 
> > 
> > * Objects to be compared: 49
> > 
> > 
> > * Result for [DNSFOREST]: SUCCESS
> > Running : /usr/bin/samba-tool ldapcmp 
> --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld 
> ldap://dc4.my.company.tld 
> > Please wait.. this can take a while..
> > cat /tmp/samba_ldapcmp_checkdb
> > 
> > 
> > * Comparing [DOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1321
> > 
> > 
> > * Result for [DOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [CONFIGURATION] context...
> > 
> > 
> > * Objects to be compared: 1713
> > 
> > 
> > * Result for [CONFIGURATION]: SUCCESS
> > 
> > 
> > * Comparing [SCHEMA] context...
> > 
> > 
> > * Objects to be compared: 1550
> > 
> > 
> > * Result for [SCHEMA]: SUCCESS
> > 
> > 
> > * Comparing [DNSDOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1691
> > 
> > 
> > * Result for [DNSDOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [DNSFOREST] context...
> > 
> > 
> > * Objects to be compared: 49
> > 
> > 
> > * Result for [DNSFOREST]: SUCCESS
> > Running : /usr/bin/samba-tool ldapcmp 
> --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld 
> ldap://dc7.my.company.tld 
> > Please wait.. this can take a while..
> > cat /tmp/samba_ldapcmp_checkdb
> > 
> > 
> > * Comparing [DOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1321
> > 
> > 
> > * Result for [DOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [CONFIGURATION] context...
> > 
> > 
> > * Objects to be compared: 1713
> > 
> > 
> > * Result for [CONFIGURATION]: SUCCESS
> > 
> > 
> > * Comparing [SCHEMA] context...
> > 
> > 
> > * Objects to be compared: 1550
> > 
> > 
> > * Result for [SCHEMA]: SUCCESS
> > 
> > 
> > * Comparing [DNSDOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1691
> > 
> > 
> > * Result for [DNSDOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [DNSFOREST] context...
> > 
> > 
> > * Objects to be compared: 49
> > 
> > 
> > * Result for [DNSFOREST]: SUCCESS
> > Running : /usr/bin/samba-tool ldapcmp 
> --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld 
> ldap://dc6.my.company.tld 
> > Please wait.. this can take a while..
> > cat /tmp/samba_ldapcmp_checkdb
> > 
> > 
> > * Comparing [DOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1321
> > 
> > 
> > * Result for [DOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [CONFIGURATION] context...
> > 
> > 
> > * Objects to be compared: 1714
> > 
> > 
> > * Result for [CONFIGURATION]: SUCCESS
> > 
> > 
> > * Comparing [SCHEMA] context...
> > 
> > 
> > * Objects to be compared: 1550
> > 
> > 
> > * Result for [SCHEMA]: SUCCESS
> > 
> > 
> > * Comparing [DNSDOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1691
> > 
> > 
> > * Result for [DNSDOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [DNSFOREST] context...
> > 
> > 
> > * Objects to be compared: 49
> > 
> > 
> > * Result for [DNSFOREST]: SUCCESS
> > Running : /usr/bin/samba-tool ldapcmp 
> --filter="whenChanged,dc,DC,cn,CN" ldap://dc1.my.company.tld 
> ldap://dc2.my.company.tld 
> > Please wait.. this can take a while..
> > cat /tmp/samba_ldapcmp_checkdb
> > 
> > 
> > * Comparing [DOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1321
> > 
> > 
> > * Result for [DOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [CONFIGURATION] context...
> > 
> > 
> > * Objects to be compared: 1714
> > 
> > 
> > * Result for [CONFIGURATION]: SUCCESS
> > 
> > 
> > * Comparing [SCHEMA] context...
> > 
> > 
> > * Objects to be compared: 1550
> > 
> > 
> > * Result for [SCHEMA]: SUCCESS
> > 
> > 
> > * Comparing [DNSDOMAIN] context...
> > 
> > 
> > * Objects to be compared: 1691
> > 
> > 
> > * Result for [DNSDOMAIN]: SUCCESS
> > 
> > 
> > * Comparing [DNSFOREST] context...
> > 
> > 
> > * Objects to be compared: 49
> > 
> > 
> > * Result for [DNSFOREST]: SUCCESS
> > .. Next check.. 
> > Running : samba-tool drs showrepl
> > grep -c "failed" /tmp/samba_drs_showrepl
> > grep -c "successful" /tmp/samba_drs_showrepl
> >          failures don't match
> >         successes don't match
> >          failures don't match
> >         successes don't match
> >          failures don't match
> >         successes don't match
> >          failures don't match
> >         successes don't match
> >          failures don't match
> >         successes don't match
> > 
> > 
> > if [ "${EMAIL_REPORT_ALWAYS}" = "yes" ] && [ -n 
> "${EMAIL_REPORT_ADDRESS}" ]; then
> >     #cat /tmp/samba_drs_showrepl | ${SET_MAILTOOL} -s 
> "SAMBA CHECK DB : showrepl results" $EMAIL_REPORT_ADDRESS
> >     ${SET_MAILTOOL} -s "SAMBA CHECK DB : showrepl results" 
> $EMAIL_REPORT_ADDRESS < /tmp/samba_drs_showrepl
> >     #cat /tmp/samba_ldapcmp_checkdb | ${SET_MAILTOOL} -s 
> "SAMBA CHECK DB : ldapcmp results" $EMAIL_REPORT_ADDRESS
> >     ${SET_MAILTOOL} -s "SAMBA CHECK DB : ldapcmp results" 
> $EMAIL_REPORT_ADDRESS < /tmp/samba_ldapcmp_checkdb
> > fi
> > 
> > 
> > if [ "${SETREMOVELOG}" = "yes" ]; then
> >     if [ -f /tmp/samba_ldapcmp_checkdb ]; then
> >         rm /tmp/samba_ldapcmp_checkdb
> >     fi
> >     if [ -f /tmp/samba_drs_showrepl ]; then
> >         rm /tmp/samba_drs_showrepl
> >     fi
> > fi
> > 
> > 
> > 
> > 
> > 
> > -- 
> > To unsubscribe from this list go to the following URL and read the
> > instructions:  https://lists.samba.org/mailman/options/samba
> 
> 