[Samba] PANIC running Samba as Domain Controller

Wed Aug 19 07:26:00 UTC 2020

We've been having issues with the samba process PANICing for some time
now (approx 24 months, though the log files have rotated so I can't
give an exact date).
For most of this time it was an annoyance - it occurred perhaps once a
fortnight and didn't cause great disruption.
Around the end of 2019 the PANIC's became more frequent. When they
occur machines attempting to authenticate seem to hang for a couple of
minutes, unaware the process they were talking to on the DC has died.
This is a particular issue for our the Windows 2012 R2 machine doing
RADIUS for our wireless controller as noone can get on the wifi until
that 2012 machine comes out of the timeout.
At their worst they have occurred every 5 minutes for about 90
minutes, though I only witnessed that one morning week commencing 3rd
August.

The PANICs occur on all DCs, but much more frequently on the one at
our hub site which has the most users, wireless controller, IIS server
authenticating webapps - presumably due to load. I'm assuming that is
why the PANICs have increased over time - increasing load.

Originally the DCs ran CentOS 6 with samba compiled from source. The
only thread I could find for this issue was
https://lists.samba.org/archive/samba/2020-January/227768.html which
appears to be the same. The advice given was that the dependencies
were likely out of date and the best option was to run packages from
apt.van-belle.nl rather than compiling from source.

I worked towards this goal slowly first replacing the hub site DC with
a CentOS 8 machine (to obtain up to date dependencies) and compiling
from source. This issue persisted. I tried initially version 4.12.5,
then rebuilt the DC with 4.9.18 (hoping the issue was introduced in
4.10) but the PANICs continued under both versions.

I then rebuilt the DC with ubuntu 20.04 using the stock 4.11.6 samba
packages, but still got the panics. Last night I added the
apt.van-belle.nl repository and upgraded to the 4.12.6 packages from
there, but this morning I have been greeted with another PANIC.

Its backtrace is below. Please advise what further troubleshooting
steps I can take to find out what is causing this issue and resolve
it.

The domain was created in 2012 with Samba 4.1.12 on CentOS 6 and
upgraded over time. There are 6 DCs, one at each site.

Many thanks,

Will

[2020/08/19 07:01:23.527871,  0]
../../source4/lib/cmdline/popt_common.c:74(popt_s4_talloc_log_fn)
  Bad talloc magic value - unknown value
[2020/08/19 07:01:23.531455,  0] ../../lib/util/fault.c:131(smb_panic_default)
  smb_panic_default: PANIC (pid 1188): Bad talloc magic value - unknown value
[2020/08/19 07:01:23.565451,  0] ../../lib/util/fault.c:264(log_stack_trace)
  BACKTRACE: 58 stack frames:
   #0 /lib/x86_64-linux-gnu/libsamba-util.so.0(log_stack_trace+0x34)
[0x7ff8ea21e624]
   #1 /lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x51) [0x7ff8ea21e741]
   #2 /lib/x86_64-linux-gnu/libtalloc.so.2(+0x3622) [0x7ff8e9f1b622]
   #3 /usr/lib/x86_64-linux-gnu/samba/libdsdb-module.so.0(dsdb_module_am_system+0x2b)
[0x7ff8e659a8bb]
   #4 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/acl.so(+0x63f5)
[0x7ff8e5ceb3f5]
   #5 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170) [0x7ff8e9eebe80]
   #6 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/descriptor.so(+0x40fb)
[0x7ff8e5cbe0fb]
   #7 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170) [0x7ff8e9eebe80]
   #8 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/extended_dn_in.so(+0x3060)
[0x7ff8e5c8f060]
   #9 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170) [0x7ff8e9eebe80]
   #10 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/asq.so(+0x170b)
[0x7ff8e61b170b]
   #11 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #12 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/server_sort.so(+0x1ae3)
[0x7ff8e59e0ae3]
   #13 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #14 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/anr.so(+0x1bfa)
[0x7ff8e5cd9bfa]
   #15 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #16 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/ranged_results.so(+0x1a9c)
[0x7ff8e5bf2a9c]
   #17 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #18 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/vlv.so(+0x3323)
[0x7ff8e59f1323]
   #19 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #20 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/paged_results.so(+0x314b)
[0x7ff8e5c1e14b]
   #21 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #22 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/dirsync.so(+0x2beb)
[0x7ff8e5cb2beb]
   #23 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #24 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/lazy_commit.so(+0x15cb)
[0x7ff8e5c635cb]
   #25 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #26 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/dsdb_notification.so(+0x1643)
[0x7ff8e5c9d643]
   #27 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #28 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/rootdse.so(+0x7ed3)
[0x7ff8e5b51ed3]
   #29 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #30 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/resolve_oids.so(+0x20f3)
[0x7ff8e5b610f3]
   #31 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #32 /lib/x86_64-linux-gnu/libldb.so.2(+0x231cb) [0x7ff8e9f041cb]
   #33 /lib/x86_64-linux-gnu/libldb.so.2(ldb_request+0x1d6) [0x7ff8e9f034a6]
   #34 /usr/lib/x86_64-linux-gnu/samba/libsamdb-common.so.0(dsdb_search+0x17f)
[0x7ff8e9f6326f]
   #35 /lib/x86_64-linux-gnu/libsamdb.so.0(+0xd7b4) [0x7ff8ea1c77b4]
   #36 /lib/x86_64-linux-gnu/libsamdb.so.0(DsCrackNameOneName+0x3e3)
[0x7ff8ea1c6923]
   #37 /lib/x86_64-linux-gnu/libsamdb.so.0(dcesrv_drsuapi_CrackNamesByNameFormat+0xc0)
[0x7ff8ea1c8ef0]
   #38 /lib/x86_64-linux-gnu/libdcerpc-server.so.0(+0x28d28) [0x7ff8e65edd28]
   #39 /lib/x86_64-linux-gnu/libdcerpc-server-core.so.0(+0x82e5)
[0x7ff8e65ba2e5]
   #40 /lib/x86_64-linux-gnu/libdcerpc-binding.so.0(+0xbd6f) [0x7ff8e9756d6f]
   #41 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so.0(+0x8f0b)
[0x7ff8e978df0b]
   #42 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so.0(+0x7c8e)
[0x7ff8e978cc8e]
   #43 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_invoke_immediate_handler+0x14a)
[0x7ff8e9f31c4a]
   #44 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_immediate+0x1e)
[0x7ff8e9f31c6e]
   #45 /lib/x86_64-linux-gnu/libtevent.so.0(+0xdab0) [0x7ff8e9f37ab0]
   #46 /lib/x86_64-linux-gnu/libtevent.so.0(+0xbe3b) [0x7ff8e9f35e3b]
   #47 /lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x98)
[0x7ff8e9f30ec8]
   #48 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b)
[0x7ff8e9f3116b]
   #49 /lib/x86_64-linux-gnu/libtevent.so.0(+0xbdcb) [0x7ff8e9f35dcb]
   #50 /usr/lib/x86_64-linux-gnu/samba/process_model/prefork.so(+0x2d7b)
[0x7ff8e6688d7b]
   #51 /usr/lib/x86_64-linux-gnu/samba/process_model/prefork.so(+0x322a)
[0x7ff8e668922a]
   #52 /usr/lib/x86_64-linux-gnu/samba/process_model/prefork.so(+0x3484)
[0x7ff8e6689484]
   #53 /usr/lib/x86_64-linux-gnu/samba/libservice.so.0(task_server_startup+0x60)
[0x7ff8ea2bc2f0]
   #54 /usr/lib/x86_64-linux-gnu/samba/libservice.so.0(server_service_startup+0x9a)
[0x7ff8ea2bab9a]
   #55 samba: task[rpc] pre-forked worker(0)(+0x5f05) [0x559a98fdef05]
   #56 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7ff8e9cd90b3]
   #57 samba: task[rpc] pre-forked worker(0)(_start+0x2e) [0x559a98fdde2e]