[Samba] After upgrade to 4.9.4, internal DNS no longer working

Sun Dec 23 12:12:04 UTC 2018

The two systems I'm referring to are on different domains, different
locations. No connection whatsoever except me being the one administering
both.

According to oEMInformation the server was provisioned by 4.3.3. I just
checked the Samba log file and it seems that version 4.7.4 was running
before the update.

I'm now looking at the logs right before the upgrade and indeed, there have
been errors so it looks like we found the reason for my issues. As a matter
of fact, just a day before the upgrade, we had an out of memory error on
the server and got into the situation that we had to make a hard reset. As
I can see now, Samba didn't go through this chain of events unharmed. It
was careless of me not to check but since the clients seemed to be able to
talk to the AD, I just didn't think there is an issue with Samba.

Here are the respective log entries. In the beginning, they refer to
version 4.7.4.:

[... prior log entry from a month earlier, no issues. Errors and warning
start after out of memory issue on the server ...]

[2018/12/21 03:06:02.974778,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:16:03.413186,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:16:03.741426,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
110
[2018/12/21 03:26:03.562027,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:26:03.831110,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
110
[2018/12/21 03:27:58.903344,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_IO_TIMEOUT
[2018/12/21 03:32:59.630451,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_IO_TIMEOUT

  [... same errors repeat about 100 times, deleted for better legibility
...]

[2018/12/21 09:17:59.389008,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
12
[2018/12/21 09:19:38.518542,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_NO_MEMORY
[2018/12/21 09:22:52.899142,  0]
../source4/smbd/server.c:450(binary_smbd_main)
  samba version 4.7.4 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2017
[2018/12/21 09:22:55.544887,  0]
../source4/smbd/server.c:622(binary_smbd_main)
  samba: using 'standard' process model
[2018/12/21 09:22:55.565451,  0]
../lib/util/become_daemon.c:124(daemon_ready)
  STATUS=daemon 'samba' finished starting up and ready to serve connections
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.

[... warning above repeats about 10 times ...]

[2018/12/22 19:13:44.916007,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: Traceback (most recent call last):
[2018/12/22 19:13:44.925489,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File "/usr/bin/samba_kcc", line 45, in <module>
[2018/12/22 19:13:44.925556,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     from samba import getopt as options
[2018/12/22 19:13:44.925595,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File
"/usr/lib/python2.7/site-packages/samba/__init__.py", line 29, in <module>
[2018/12/22 19:13:44.999014,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     import samba.param
[2018/12/22 19:13:44.999099,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: ImportError: /usr/lib/samba/libreplace-samba4.so:
version `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
[2018/12/22 19:13:45.004497,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_ACCESS_DENIED
[2018/12/22 19:31:48.283948,  0]
../source4/smbd/process_standard.c:86(sigterm_signal_handler)
  Exiting pid 391 on SIGTERM

  [... error above repeats about 10 times ...]

  [2018/12/22 19:31:48.446449,  0] ../source4/smbd/server.c:131(sig_term)
  SIGTERM: killing children
[2018/12/22 19:31:48.447387,  0] ../source4/smbd/server.c:131(sig_term)
  SIGTERM: killing children
[2018/12/22 19:31:48.447428,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 367 on SIGTERM
[2018/12/22 19:31:48.464044,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 391 on SIGTERM
[2018/12/22 19:31:48.806419,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 394 on SIGTERM
[2018/12/22 19:31:48.806490,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 395 on SIGTERM
[2018/12/22 19:31:48.806540,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 399 on SIGTERM
[2018/12/22 19:31:48.806590,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 392 on SIGTERM
[2018/12/22 19:31:48.806634,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 458 on SIGTERM
[2018/12/22 19:31:48.806679,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 393 on SIGTERM
[2018/12/22 19:31:48.811314,  0] ../source4/smbd/server.c:135(sig_term)
[2018/12/22 19:31:48.811348,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 401 on SIGTERM
  Exiting pid 396 on SIGTERM
[2018/12/22 19:31:48.811404,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 400 on SIGTERM
[2018/12/22 19:31:48.811451,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 402 on SIGTERM
[2018/12/22 19:31:48.811511,  0] ../source4/smbd/server.c:135(sig_term)
[2018/12/22 19:31:48.811546,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 398 on SIGTERM
[2018/12/22 19:31:48.811605,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 390 on SIGTERM
  Exiting pid 397 on SIGTERM
[2018/12/22 19:35:38.400284,  0]
../source4/smbd/server.c:510(binary_smbd_main)

[... clearly, I made an upgrade at this point, not aware I had problems.
what follows are issues I already described ...]

  samba version 4.9.4 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2018
[2018/12/22 19:35:40.471526,  0]
../source4/smbd/server.c:696(binary_smbd_main)
  binary_smbd_main: samba: using 'standard' process model
[2018/12/22 19:35:40.647282,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
[2018/12/22 19:35:40.647380,  0]
../source4/dsdb/common/util.c:1815(samdb_reference_dn_is_our_ntdsa)
  Failed to find object DC=samdom,DC=example,DC=com for attribute
fsmoRoleOwner - Cannot find DN DC=samdom,DC=example,DC=com to get attribute
fsmoRoleOwner for reference dn: No such Base DN: DC=samdom,DC=example,DC=com
[2018/12/22 19:35:40.648352,  0]
../source4/dsdb/dns/dns_update.c:127(dnsupdate_rebuild)
  task_server_terminate: task_server_terminate: [kdc: krb5_init_context
samdb RODC connect failed]
[2018/12/22 19:35:40.674777,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
[2018/12/22 19:35:40.674823,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
  task_server_terminate: task_server_terminate: [dreplsrv: Failed to
connect to local samdb: WERR_DS_UNAVAILABLE

It looks like the memory issue corrupted something in Samba. But what and
why? I'm sure I didn't make it better by upgrading, I assume this
reindexing you referred to didn't really happen if Samba was already in a
degraded state. On the other hand, if the ldb-files can be queried
properly, where is the corruption? When it comes to these details, Samba is
still pretty much a black box to me.

Thanks a lot for bearing with me so far. I hope this can be solved somehow.
I guess the good thing is that, if worse comes to worst, I could just
downgrade back to 4.7. and restore a backup from before the memory issue.
Would it be alright, in that worst case, if I just overwrite all contents
of /var/lib/samba with the files I have in the backup?

On Sun, 23 Dec 2018 at 11:36, Rowland Penny via samba <samba at lists.samba.org>
wrote:

> On Sun, 23 Dec 2018 11:19:37 +0100
> Viktor Trojanovic <viktor at troja.ch> wrote:
>
> > I was too quick to send my previous email. It can't really be a
> > problem of the package since I have, as mentioned, another system
> > with the exact same setup and the exact same package versions and it
> > all works there...
> >
>
> Upgrading a running Samba server on an OS should work without error,
> unless there was an underlying problem already.
>
> Is your other system running as a DC in the same domain, or is it a
> totally separate domain ?
>
> It might help if you could remember what version you upgraded from, was
> it the version it was provisioned at ?
> If so, there is an attribute in the base 'dc=samdom,dc=example,dc',
> this attribute 'oEMInformation' contains the Samba version at provision.
>
> Rowland
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba