[Samba] After upgrade to 4.9.4, internal DNS no longer working

Wed Dec 26 18:03:11 UTC 2018

Hi Rowland,

I went back to the logs from before the upgrade to see if we didn't miss
anything.

If I understand the logs correctly, in the early morning of December 21
(3am), the server started running out of memory as was confirmed in the
logs with several Samba components complaining about NT_STATUS_NO_MEMORY.
Here just one line as an example.

[2018/12/21 09:19:38.518542,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_NO_MEMORY

By the time I was able to work on this (around 9:30), the machine could no
longer be accessed and had to be rebooted via hard reset.

However, Samba started after this hard reset without any errors. Samba
actually remains error free for the next 1.5 days.

In the evening of the next day, I started the upgrade, at 19:10 to be
exact. And the following lines stands out:

[2018/12/22 19:13:44.916007,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: Traceback (most recent call last):
[2018/12/22 19:13:44.925489,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File "/usr/bin/samba_kcc", line 45, in <module>
[2018/12/22 19:13:44.925556,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     from samba import getopt as options
[2018/12/22 19:13:44.925595,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File
"/usr/lib/python2.7/site-packages/samba/__init__.py", line 29, in <module>
[2018/12/22 19:13:44.999014,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     import samba.param
[2018/12/22 19:13:44.999099,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: ImportError: /usr/lib/samba/libreplace-samba4.so:
version `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
[2018/12/22 19:13:45.004497,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_ACCESS_DENIED

These and some SIGTERM messages are the last lines in the Samba log before
the server is rebooted, and Samba is then started in version 4.9.4 - but
now with several errors.

While I really don't know if the lines above have any relevance, I believe
we may need to correct the assumption that something was wrong before the
update, it now much rather looks as if the update broke my DC. Do you
agree?

On Wed, 26 Dec 2018 at 16:29, Viktor Trojanovic <viktor at troja.ch> wrote:

>
>
> On Wed, 26 Dec 2018 at 15:04, Rowland Penny via samba <
> samba at lists.samba.org> wrote:
>
>> On Wed, 26 Dec 2018 14:40:10 +0100
>> Viktor Trojanovic <viktor at troja.ch> wrote:
>>
>> > Hi Rowland,
>> >
>> > Thanks for taking an interest.
>> >
>> > On Wed, 26 Dec 2018 at 14:27, Rowland Penny via samba
>> > <samba at lists.samba.org> wrote:
>> >
>> > > On Wed, 26 Dec 2018 11:43:37 +0100
>> > > Viktor Trojanovic <viktor at troja.ch> wrote:
>> > >
>> > > > I could really use some support with this. I understand it's
>> > > > always possible to just restore from a backup but the more
>> > > > interesting question is if something can be done with the data at
>> > > > hand. Basically, I'm trying to understand how it's possible that
>> > > > a dbcheck shows no errors, an ldbsearch is successful, and yet
>> > > > it's not possible to start the AD properly. What else is there
>> > > > that could be corrupted, and is there a way to repair it?
>> > > >
>> > >
>> > > OK, I have been reviewing all the posts in this thread and I have a
>> > > few questions ;-)
>> > >
>> > > You posted that you are running the DC using the internal DNS
>> > > server, but you also posted this:
>> > >
>> > > All checks on the flat files work fine
>> > >
>> > > What 'flat files' ?
>> > > Are you referring to Bind9 flat files ?
>> > >
>> > >
>> > No. I meant the *.tdb and *.ldb files in the samba directory. I'm
>> > using the internal DNS server, I don't have Bind installed, never had.
>>
>> Good, 'flat files' usually refers to the Bind9 zones conf files.
>>
>> >
>> >
>> > > There is also this:
>> > >
>> > > kdc_task_init: Cannot determine if we are an RODC: operations error
>> > > at ../source4/dsdb/common/util.c:3534
>> > > task_server_terminate: task_server_terminate: [kdc:
>> > > krb5_init_context samdb RODC connect failed]
>> > >
>> > > Why is 'RODC' getting mentioned ?
>> > >
>> > >
>> > I don't even know what RODC is. :) This is a regular AD DC install as
>> > it is described on the Wiki, I didn't do anything exotic there.
>>
>> A normal DC is an RWDC (Read Write Domain Controller), an RODC is a
>> 'Read Only Domain Controller'
>>
>> For some reason, Your Samba is getting confused
>>
>>
> I wouldn't know why, I didn't make any changes in this respect.
>
>
>> >
>> >
>> > > Can you post the contents of:
>> > >
>> > > /etc/hostname
>> > >
>> >
>> > DC1
>> >
>> >
>> > > /etc/nsswitch.conf
>> > >
>> > >
>> > passwd: files winbind mymachines systemd
>> > group: files winbind mymachines systemd
>> > shadow: files
>> >
>> > publickey: files
>> >
>> > hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns
>>
>> Try the above line like this:
>>
>> hosts: files dns
>>
>> > networks: files
>> >
>> > protocols: files
>> > services: files
>> > ethers: files
>> > rpc: files
>> >
>> > netgroup: files
>> >
>> >
>> > Could the separate Samba daemons (smbd, nmbd, winbind) be being
>> > started
>> > > instead of/as well as the 'samba' daemon ?
>> > >
>> > >
>> > No, it's really just the "Samba AD Daemon" (samba.service). smbd
>> > (smb.service), nmbd (nmb.service), as well as winbindd
>> > (winbind.service) are not started. As you could see from the log, the
>> > binaries such as smbd and winbindd are indeed started but I guess
>> > that is done by Samba automatically?
>>
>> Yes, the 'samba' daemon does start smbd & winbind, but it was just a
>> thought.
>>
>> Check if systemd is starting any other dns server (resolved ?)
>>
>
> Yes, it is starting resolved but that was always the case. Resolved reads
> from the network file and adds the correct nameserver (itself) and search
> string (samdom.example.com).
>
> Just to rule out resolved as the culprit, I now disabled resolved entirely
> and created a static resolv.conf file, then rebooted the server.
>
> After doing all of this, this is the Samba log:
>
> [2018/12/26 16:28:57.872726,  0]
> ../source4/smbd/server.c:510(binary_smbd_main)
>   samba version 4.9.4 started.
>   Copyright Andrew Tridgell and the Samba Team 1992-2018
> [2018/12/26 16:28:58.179432,  0]
> ../source4/smbd/server.c:696(binary_smbd_main)
>   binary_smbd_main: samba: using 'standard' process model
> [2018/12/26 16:28:58.192497,  0]
> ../source4/dsdb/common/util.c:1815(samdb_reference_dn_is_our_ntdsa)
>   Failed to find object DC=samdom,DC=example,DC=com for attribute
> fsmoRoleOwner - Cannot find DN DC=samdom,DC=example,DC=com to get attribute
> fsmoRoleOwner for reference dn: No such Base DN: DC=samdom,DC=example,DC=com
> [2018/12/26 16:28:58.208645,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
> [2018/12/26 16:28:58.211103,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
>   task_server_terminate: task_server_terminate: [kdc: krb5_init_context
> samdb RODC connect failed]
>   task_server_terminate: task_server_terminate: [dreplsrv: Failed to
> connect to local samdb: WERR_DS_UNAVAILABLE
>   ]
> [2018/12/26 16:28:58.235875,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
> [2018/12/26 16:28:58.241918,  0]
> ../source4/dsdb/dns/dns_update.c:127(dnsupdate_rebuild)
>   task_server_terminate: task_server_terminate: [kccsrv: Failed to connect
> to local samdb: WERR_DS_UNAVAILABLE
>   ]
> [2018/12/26 16:28:58.274730,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
>   task_server_terminate: task_server_terminate: [dns: failed to load DNS
> zones]
>
>
>
>
>
>