[Samba] DLZ Backend DNS Hosed

Sun Jun 23 23:35:05 UTC 2019

I wanted to update you on my solution, and tell you about the new issue that came out of my solution.

 - The idea of "upgrading" to internal DNS failed to help. In fact, it broke DNS for me. I quickly re-upgraded my DNS to BIND_DLZ.
 - The idea of simply deleting and recreating my zone file (samdom.mycompany.net) appears to have been the ticket. I was afraid all kinds of terribly things would happen if I deleted it, but I was wrong. Everything went smoothly and I was able to re-add my NS records and update the SOA and add A records for my DCs, etc. So at that point, all my DCs were talking to each other very nicely...

...except for one painful mystery:

I have a single DC (we'll call it, "DC1") that simply will not take my password when I run this command:

#samba-tool ldapcmp ldap://dc2 ldap://dc3 -Uadministrator

Or this command:

#samba-tool ldapcmp ldap://dc1 ldap://dc2 -Uadministrator

I basically get this:

> Password for [SAMDOM\administrator]:
> Password for [SAMDOM\administrator]:
> Password for [SAMDOM\administrator]:
> Failed to bind - LDAP error 49 LDAP_INVALID_CREDENTIALS -  <8009030C: LdapErr: DSID-0C0904DC, comment: AcceptSecurityContext error, data 52e, > v1db1> <>
> Failed to connect to 'ldap://dc2' with backend 'ldap': LDAP error 49 LDAP_INVALID_CREDENTIALS -  <8009030C: LdapErr: DSID-0C0904DC, comment: AcceptSecurityContext error, data 52e, v1db1> <>
> ERROR(ldb): uncaught exception - LDAP error 49 LDAP_INVALID_CREDENTIALS -  <8009030C: LdapErr: DSID-0C0904DC, comment: AcceptSecurityContext error, data 52e, v1db1> <>
>   File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line 185, in _run
>     return self.run(*args, **kwargs)
>   File "/usr/lib/python3/dist-packages/samba/netcmd/ldapcmp.py", line 942, in run
>     outf=self.outf, errf=self.errf, skip_missing_dn=skip_missing_dn)
>   File "/usr/lib/python3/dist-packages/samba/netcmd/ldapcmp.py", line 64, in __init__
>     options=ldb_options)
>   File "/usr/lib/python3/dist-packages/samba/__init__.py", line 115, in __init__
>     self.connect(url, flags, options)

It *will* authenticate when I run this command, which implies that DC2 is the one who doesn't like my password, but only when it comes from DC1:

#samba-tool ldapcmp ldap://dc1 ldap://dc3 -Uadministrator

>From DC2 and DC3, I can run all three of those commands with success.

What could cause one of my DCs (DC2) to hate my password only when it comes from one of my other DCs (DC1)? And, by the way, under that circumstance, it seems to hate every username and password combination I have that I could think to try ("-Umatthewdelfino", for example).

What have I already tried? I've demoted and re-promoted DC2, which didn't make things any better. Passwords still fail in the same manner, but now every time I do an ldapcmp from samba-tool, I see a complaint about "serverReferenceBL," either as an attribute that doesn't exist in DC1 for 'CN=DC2,OU=DOMAIN CONTROLLERS,DC=SAMDOM,DC=MYCOMPANY,DC=NET', or as an uncaught exception like this:

> ERROR(<class 'KeyError'>): uncaught exception - 'serverReferenceBL'
>   File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line 185, in _run
>     return self.run(*args, **kwargs)
>   File "/usr/lib/python3/dist-packages/samba/netcmd/ldapcmp.py", line 957, in run
>     if b1.diff(b2):
>   File "/usr/lib/python3/dist-packages/samba/netcmd/ldapcmp.py", line 781, in diff
>     if object1 == object2:
>   File "/usr/lib/python3/dist-packages/samba/netcmd/ldapcmp.py", line 549, in __eq__
>     return self.cmp_attrs(other)
>   File "/usr/lib/python3/dist-packages/samba/netcmd/ldapcmp.py", line 590, in cmp_attrs
>     if isinstance(self.attributes[x], list) and isinstance(other.attributes[x], list):

For me, it seems, layers of an onion. Or a bad game of whack-a-mole. Can anyone help?

Thanks,
Matthew

 From:   Matthew Delfino via samba <samba at lists.samba.org> 
 To:   L.P.H. van Belle <belle at bazuin.nl>, <dcardon at tranquil.it>, Rowland Penny <rpenny at samba.org>, "samba at lists.samba.org" <samba at lists.samba.org> 
 Sent:   6/21/2019 10:05 AM 
 Subject:   Re: [Samba] DLZ Backend DNS Hosed 

Louis, 

I appreciate your efforts with my predicament. I'm very sorry to say that your advice hasn't gotten me to a solution. After updating my /etc/network/interfaces to put my localhost IP address first (192.168.3.201, for example), saving, restarting services, rebooting, running "samba_upgradedns --dns-backend=BIND9_DLZ", saving, rebooting, etc., I still cannot add, edit or remove records from the samdom.mycompany.net zone. 

# samba_dnsupdate --verbose 
IPs: ['192.168.3.201'] 
Looking for DNS entry A umbriel.samdom.mycompany.net 192.168.3.201 as umbriel.samdom.mycompany.net. 
Looking for DNS entry NS samdom.mycompany.net umbriel.samdom.mycompany.net as samdom.mycompany.net. 
Traceback (most recent call last): 
  File "/usr/sbin/samba_dnsupdate", line 320, in check_dns_name 
    ans = check_one_dns_name(normalised_name, d.type, d) 
  File "/usr/sbin/samba_dnsupdate", line 296, in check_one_dns_name 
    ans = resolver.query(name, name_type) 
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 821, in query 
    raise NoNameservers 
dns.resolver.NoNameservers 

During handling of the above exception, another exception occurred: 

Traceback (most recent call last): 
  File "/usr/sbin/samba_dnsupdate", line 851, in <module> 
    elif not check_dns_name(d): 
  File "/usr/sbin/samba_dnsupdate", line 324, in check_dns_name 
    raise Exception("Unable to contact a working DNS server while looking for %s as %s" % (d, normalised_name)) 
Exception: Unable to contact a working DNS server while looking for NS samdom.mycompany.net umbriel.samdom.mycompany.net as samdom.mycompany.net. 

Denis, 

I appreciate this email you sent. Running this command results in the following: 

#samba-tool dns add umbriel samdom.mycompany.net @ NS umbriel.samdom.mycompany.net -P 

ERROR(runtime): uncaught exception - (1383, 'WERR_INTERNAL_DB_ERROR') 
  File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line 185, in _run 
    return self.run(*args, **kwargs) 
  File "/usr/lib/python3/dist-packages/samba/netcmd/dns.py", line 944, in run 
    raise e 
  File "/usr/lib/python3/dist-packages/samba/netcmd/dns.py", line 940, in run 
    0, server, zone, name, add_rec_buf, None) 

It just seems that my zone "samdom.mycompany.net" is completely hosed. 

Rowland, 

I always appreciate your tireless support. You made some comments, asked some questions. Here are my responses: 

>>  
>>  
>> Able to be edited: _msdcs.samdom.mycompany.net  
>> NOT able to be edited: samdom.mycompany.net  
> 
> I read the output of Louis's script you posted and my first thought was,  
> 'why has he got dns domains that have nothing to do with AD ?'  

On the subject of Samba: Advice and how-tos on the Internet focus on telling us what to do, often with only the most minimal context. It has taken me time to learn and discover how the configurations advised can work best for my environment. There are bits of configuration representing the legacy of that evolution and there you saw one. 

> In my opinion, you should only have the dns records for your Samba AD > domain in AD, this should include any reverse zones.  

So, do be clear, the bare minimum in my case would be: 

samdom.mycompany.net 
_msdcs.samdom.mycompany.net 
3.168.192.in-addr.arpa 

Do you concur? 

>> I'm feeling lonely here.  
>  
> Do you have a backup ?  

Of course. But I'm confident the backups don't go far enough back to get me past whatever hosed my zone. I suspect all this started with me being ignorant of the change which moved many BIND DLZ backend stuff from ../samba/private/ to ../samba/bind-dns/. We make so few changes to the DNS on the DCs that I didn't notice that bind had been unable to talk to samba for goodness knows how long. Between my first email and my second email to the list, I discovered that erroneous configuration and addressed it, which helped immensely in terms of end user experience. 

> I have had something similar happen, but with the reverse zone and I  
> just deleted the zone and recreated it with samba-tool and then let the  
> records be recreated. In your case, I would be tempted to 'upgrade' to  
> the internal dns server and then 'upgrade' to the Bind9 server. This  
> should recreate all the required zones and records.  

This is the only thing I have not tried and it's the one that seems to make the most sense to me. But I don't want to do it during business hours. I'll need to stick around late tonight to give it a shot (including backups before I attempt all this). I've never done a maneuver like that before - I will report my results in several hours. 

>> search samdom.mycompany.net mycompany.net mycompany.com  
> I would remove any domains that are not the Samba dns domain  

Done. 

>>   server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate  
>>   workgroup = SAMDOM  
>>   idmap_ldb:use rfc2307 = yes  
>>   #dns forwarder = 8.8.4.4  
>>   #dns forwarder = 8.8.8.8  
>>   allow dns updates = disabled  
> Bad move, something needs to be able to upgrade your dns records.  
>>   dsdb:schema update allowed = true  
> Remove this, it is only needed to extend the schema and can be used on  
> the ldbmodify command line.  

Done and done. 

> Please set your dns up correctly, first remove the 3 blocks above, > forward anything outside your Samba dns domain to another external dns  
> server and inform any other, non AD dns servers that you have, where  
> your AD domain is 

I removed the 3 blocks above as they are unnecessary. Forwards to external DNS servers were already there, the other DNS servers know about samdom.mycompany.net by way of glue records in db.mycompany.net like this: 

;;;;;;;;;;;;;;;;;; 
;; Glue Records ;; For the SAMBA Active Directory Domain Controllers 
;;;;;;;;;;;;;;;;;; 

$ORIGIN samdom.mycompany.net. 
@ 86400 IN NS samdom.mycompany.net.  
 86400 IN NS samdom.mycompany.net.  
 86400 IN NS samdom.mycompany.net. 
cordelia.samdom.mycompany.net. IN A 192.168.3.201  
hyperion.samdom.mycompany.net. IN A 192.168.3.202  
umbriel.samdom.mycompany.net. IN A 192.168.3.203  

Is this what you meant by "inform any other, non AD dns servers that you have, where your AD domain is?" 

Thanks again, there are no further comments from me below. 

Matthew 

© 2019 KNOCK, inc. All rights reserved. KNOCK is a registered trademark of KNOCK, inc. This message and any attachments contain information, which is confidential and/or privileged. If you are not the intended recipient, please refrain from any disclosure, copying, distribution or use of this information. Please be aware that such actions are prohibited. If you have received this transmission in error, kindly notify the sender by e-mail. Your cooperation is appreciated.