Anyone seen this crash in winbindd?

[Richard Sharpe]

On Mon, Aug 21, 2017 at 7:39 AM, Richard Sharpe
<realrichardsharpe at gmail.com> wrote:
> Hi folks,
>
> I did a bit of searching on the web but couldn't come up with
> anything. It looks like the memory has been deallocated out from under
> the code that was trying to take the allrecord_mutex_lock:
>
> #0  0x00007f1831fb61d7 in __GI_raise (sig=sig at entry=6)
>     at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00007f1831fb78c8 in __GI_abort () at abort.c:90
> #2  0x00007f18351f963b in dump_core () at ../source3/lib/dumpcore.c:322
> #3  0x00007f18351eacf7 in smb_panic_s3 (why=<optimized out>)
>     at ../source3/lib/util.c:814
> #4  0x00007f183891b12f in smb_panic (why=why at entry=0x7f18389686ab
> "internal error")
>     at ../lib/util/fault.c:166
> #5  0x00007f183891b346 in fault_report (sig=<optimized out>) at
> ../lib/util/fault.c:83
> #6  sig_fault (sig=<optimized out>) at ../lib/util/fault.c:94
> #7  <signal handler called>
> #8  0x00007f183a374e70 in __pthread_mutex_trylock (mutex=0x7f183a5a90a8)
>     at pthread_mutex_trylock.c:141
> #9  0x00007f18386f68d5 in allrecord_mutex_lock (m=0x7f183a5a9000,
>     waitflag=<optimized out>) at ../lib/tdb/common/mutex.c:203
> #10 0x00007f18386f6dfc in tdb_mutex_allrecord_lock
> (tdb=tdb at entry=0x7f183bf6e950,
>     ltype=ltype at entry=1, flags=flags at entry=TDB_LOCK_NOWAIT)
>     at ../lib/tdb/common/mutex.c:374
> #11 0x00007f18386efd8e in tdb_allrecord_lock (tdb=0x7f183bf6e950, ltype=1,
>     flags=TDB_LOCK_NOWAIT, upgradable=<optimized out>) at
> ../lib/tdb/common/lock.c:646
> #12 0x00007f18386efe7e in tdb_lockall_nonblock (tdb=<optimized out>)
>     at ../lib/tdb/common/lock.c:771
> #13 0x00007f1835200b6f in gencache_stabilize () at ../source3/lib/gencache.c:667
> #14 0x00007f183a7dd6b9 in terminate (is_parent=<optimized out>)
>     at ../source3/winbindd/winbindd.c:247
> #15 0x00007f183a7dd79a in winbindd_sig_term_handler (ev=<optimized out>,
>     se=<optimized out>, signum=15, count=<optimized out>,
> siginfo=<optimized out>,
>     private_data=<optimized out>) at ../source3/winbindd/winbindd.c:280
> #16 0x00007f1837e83ef7 in tevent_common_check_signal (ev=0x7f183bf5b8e0)
>     at ../lib/tevent/tevent_signal.c:461
> #17 0x00007f1837e85bec in epoll_event_loop (tvalp=0x7fff2f95b770,
>     epoll_ev=0x7f183bf5d2e0) at ../lib/tevent/tevent_epoll.c:647
> #18 epoll_event_loop_once (ev=<optimized out>, location=<optimized out>)
>     at ../lib/tevent/tevent_epoll.c:926
> #19 0x00007f1837e84137 in std_event_loop_once (ev=0x7f183bf5b8e0,
>     location=0x7f183a874c70 "../source3/winbindd/winbindd_dual.c:1592")
>     at ../lib/tevent/tevent_standard.c:114
> #20 0x00007f1837e8038d in _tevent_loop_once (ev=0x7f183bf5b8e0,
>     location=location at entry=0x7f183a874c70
> "../source3/winbindd/winbindd_dual.c:1592")
>     at ../lib/tevent/tevent.c:533
> #21 0x00007f183a8084f8 in fork_domain_child (
>     child=0x7f183aabfd00 <static_idmap_child>)
>     at ../source3/winbindd/winbindd_dual.c:1592
> #22 0x00007f183a808bc5 in wb_child_request_trigger (req=0x7f183bf6f250,
>     private_data=<optimized out>) at ../source3/winbindd/winbindd_dual.c:173
> #23 0x00007f1837e80bb4 in tevent_common_loop_immediate
> (ev=ev at entry=0x7f183bf5b8e0)
>     at ../lib/tevent/tevent_immediate.c:135
> #24 0x00007f1837e85a2e in epoll_event_loop_once (ev=0x7f183bf5b8e0,
>     location=<optimized out>) at ../lib/tevent/tevent_epoll.c:907
> #25 0x00007f1837e84137 in std_event_loop_once (ev=0x7f183bf5b8e0,
>     location=0x7f183a85ec48 "../source3/winbindd/winbindd.c:1809")
>     at ../lib/tevent/tevent_standard.c:114
> #26 0x00007f1837e8038d in _tevent_loop_once (ev=0x7f183bf5b8e0,
>     location=location at entry=0x7f183a85ec48
> "../source3/winbindd/winbindd.c:1809")
>     at ../lib/tevent/tevent.c:533
> #27 0x00007f183a7cecfc in main (argc=<optimized out>, argv=<optimized out>)
>     at ../source3/winbindd/winbindd.c:1809
> #9  0x00007f18386f68d5 in allrecord_mutex_lock (m=0x7f183a5a9000,
>     waitflag=<optimized out>) at ../lib/tdb/common/mutex.c:203
> (gdb) frame 9
> 203                     ret = pthread_mutex_trylock(&m->allrecord_mutex);
> (gdb) p m
> $2 = (struct tdb_mutexes *) 0x7f183a5a9000
> (gdb) p *m
> Cannot access memory at address 0x7f183a5a9000

More info. It's a shutdown race by the look of things:

[2017/08/20 17:11:01.927595,  0]
../source3/winbindd/winbindd.c:279(winbindd_sig_term_handler)
  Got sig[15] terminate (is_parent=0)
[2017/08/20 17:11:01.936107,  0] ../lib/util/fault.c:78(fault_report)
  ===============================================================
[2017/08/20 17:11:01.936136,  0] ../lib/util/fault.c:79(fault_report)
  INTERNAL ERROR: Signal 7 in pid 18845 (4.5.11)
  Please read the Trouble-Shooting section of the Samba HOWTO
[2017/08/20 17:11:01.936151,  0] ../lib/util/fault.c:81(fault_report)
  ===============================================================
[2017/08/20 17:11:01.936162,  0] ../source3/lib/util.c:791(smb_panic_s3)
  PANIC (pid 18845): internal error
[2017/08/20 17:11:01.936479,  0] ../source3/lib/util.c:902(log_stack_trace)
  BACKTRACE: 25 stack frames:
   #0 /lib64/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f18351eabba]
   #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f18351eac90]
   #2 /lib64/libsamba-util.so.0(smb_panic+0x2f) [0x7f183891b12f]
   #3 /lib64/libsamba-util.so.0(+0x1c346) [0x7f183891b346]


-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)