"smbd" core-dump as a result of changing "inherit permissions" to "no" in "smb.conf".

Fri Sep 27 10:07:35 MDT 2013

Interesting Samba issue for the gurus:

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
16:19:32.718301,  0] ../lib/util/fault.c:72(fault_report)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]
===============================================================

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
16:19:32.718388,  0] ../lib/util/fault.c:73(fault_report)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]   INTERNAL
ERROR: Signal 6 in pid 2392 (4.2.0pre1-GIT-89200c2)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]   Please read
the Trouble-Shooting section of the Samba HOWTO

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
16:19:32.718502,  0] ../lib/util/fault.c:75(fault_report)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]
===============================================================

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
16:19:32.718589,  0] ../source3/lib/util.c:785(smb_panic_s3)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]   PANIC (pid
2392): internal error

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
16:19:32.786239,  0] ../source3/lib/util.c:896(log_stack_trace)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]   BACKTRACE: 24
stack frames:

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #0
/usr/local/samba/lib/libsmbconf.so.0'log_stack_trace+0x2b [0xfd223996]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #1
/usr/local/samba/lib/libsmbconf.so.0'smb_panic_s3+0x80 [0xfd2237fd]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #2
/usr/local/samba/lib/libsamba-util.so.0.0.1'smb_panic+0x2a [0xfe5aab1e]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #3
/usr/local/samba/lib/libsamba-util.so.0.0.1'sig_fault+0x0 [0xfe5aa7f7]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #4
/usr/local/samba/lib/libsamba-util.so.0.0.1'sig_fault+0x11 [0xfe5aa808]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #5
/lib/libc.so.1'__sighndlr+0x15 [0xfe6a47d5]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #6
/lib/libc.so.1'call_user_handler+0x2d2 [0xfe697dae]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #7
/lib/libc.so.1'_lwp_kill+0x15 [0xfe6a9265]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #8
/lib/libc.so.1'raise+0x25 [0xfe650fed]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #9
/lib/libc.so.1'abort+0x77 [0xfe6286df]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #10
/lib/libc.so.1'0x48add [0xfe628add]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #11
/usr/local/samba/lib/libsmbconf.so.0'msg_channel_trigger+0xc5 [0xfd232b70]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #12
/usr/local/samba/lib/private/libtevent.so.0.9.19'tevent_common_loop_immediat
e+0x1a4 [0xfe324

e8d]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #13
/usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x4a [0xfd23fb1c]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #14
/usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x66 [0xfd2400fd]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #15
/usr/local/samba/lib/private/libtevent.so.0.9.19'_tevent_loop_once+0xf3
[0xfe32422a]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #16
/usr/local/samba/lib/private/libsmbd_base.so'smbd_process+0x13ff
[0xfdf5f283]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #17
/usr/local/samba/sbin/smbd'smbd_accept_connection+0x435 [0x805a21d]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #18
/usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x483 [0xfd23ff55]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #19
/usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x18a [0xfd240221]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #20
/usr/local/samba/lib/private/libtevent.so.0.9.19'_tevent_loop_once+0xf3
[0xfe32422a]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #21
/usr/local/samba/sbin/smbd'smbd_parent_loop+0x8c [0x805afdc]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #22
/usr/local/samba/sbin/smbd'main+0x1951 [0x805ca93]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]    #23
/usr/local/samba/sbin/smbd'_start+0x83 [0x8055823]

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
16:19:32.820147,  0] ../source3/lib/dumpcore.c:317(dump_core)

Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]   dumping core
in /var/samba/log/cores/smbd

This started yesterday (which was why I tried pulling the latest snapshot)
after I made a single change to our "smb.conf" file:

$ diff smb.conf
/.zfs/snapshot/__PeriodicSnapshot__2013.269.-09-26_15:00/etc/sfw/samba/smb.c
onf

216,217c216

<       inherit permissions = no

< #     inherit permissions = yes

---

>       inherit permissions = yes

$

All seems to work fine until I stress the Samba server - in this case by
streaming about 8Gb to a file.

I changed the "inherit permissions" setting to try and get to the bottom of
a really annoying problem whereby we keep getting spurious permissions
errors on files on Samba shares (can't write some files, can't execute
others).  Changing the setting apparently resolved those problems (though I
hadn't finished verifying that), but causes "smbd" to core-dump instead.

It's not a problem in interpreting the setting from "smb.conf"; the default
value for this option is "no" and if I just comment out both "inherit
permissions" lines (so the default behaviour applies) then "smbd" still
core-dumps as soon as the server gets stressed (poor little server .).
Anyway, the configuration file parsing code is surely pretty mature by now -
I wouldn't expect any problems of that sort.

I would poke the core dump with "gdb", but unfortunately the "gdb" supplied
with "Solaris 11" is terminally brain-damaged and always reports any error
as being at the first line of code in the source file.  I've not yet managed
to compile up any later "gdb" (to see whether that fixes it) - that's about
item number 49 on my current "urgent" to-do list :(.  So that stack trace is
about as close as I can get, unless a disassembly dump of the code around
"/usr/local/samba/lib/libsmbconf.so.0'msg_channel_trigger+0xc5 [0xfd232b70]"
would be of any use?  However the core dumps I was getting *yesterday* are
*slightly* less unhelpful:

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
20:18:51.589417,  0] ../source3/smbd/open.c:1527(defer_open_done)

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]   PANIC: assert
failed at ../source3/smbd/open.c(1527): ret

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
20:18:51.589593,  0] ../source3/lib/util.c:810(smb_panic_s3)

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]   PANIC (pid
2005): assert failed: ret

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
20:18:51.638743,  0] ../source3/lib/util.c:921(log_stack_trace)

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]   BACKTRACE: 19
stack frames:

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #0
/usr/local/samba/lib/libsmbconf.so.0'log_stack_trace+0x2b [0xfd232aac]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #1
/usr/local/samba/lib/libsmbconf.so.0'smb_panic_s3+0x80 [0xfd232913]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #2
/usr/local/samba/lib/libsamba-util.so.0.0.1'smb_panic+0x2a [0xfe5aab1e]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #3
/usr/local/samba/lib/private/libsmbd_base.so'defer_open_done+0x203
[0xfdf2cbdd]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #4
/usr/local/samba/lib/private/libtevent.so.0.9.18'_tevent_req_notify_callback
+0x5c [0xfe3257ff

]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #5
/usr/local/samba/lib/private/libtevent.so.0.9.18'tevent_req_finish+0x2d
[0xfe325832]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #6
/usr/local/samba/lib/private/libtevent.so.0.9.18'tevent_req_timedout+0x69
[0xfe325b6c]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #7
/usr/local/samba/lib/private/libtevent.so.0.9.18'tevent_common_loop_timer_de
lay+0x229 [0xfe32

9c03]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #8
/usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x70 [0xfd24d78e]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #9
/usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x18a [0xfd24de6d]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #10
/usr/local/samba/lib/private/libtevent.so.0.9.18'_tevent_loop_once+0xf3
[0xfe3241ea]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #11
/usr/local/samba/lib/private/libsmbd_base.so'smbd_process+0x13ff
[0xfdf57625]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #12
/usr/local/samba/sbin/smbd'smbd_accept_connection+0x435 [0x8059f6d]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #13
/usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x483 [0xfd24dba1]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #14
/usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x18a [0xfd24de6d]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #15
/usr/local/samba/lib/private/libtevent.so.0.9.18'_tevent_loop_once+0xf3
[0xfe3241ea]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #16
/usr/local/samba/sbin/smbd'smbd_parent_loop+0x8c [0x805ad2c]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #17
/usr/local/samba/sbin/smbd'main+0x1951 [0x805c7e3]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]    #18
/usr/local/samba/sbin/smbd'_start+0x83 [0x8055783]

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
20:18:51.671743,  0] ../source3/lib/dumpcore.c:317(dump_core)

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]   dumping core
in /var/samba/log/cores/smbd

Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]

So that's actually showing an assertion failure at line 1527 of
"source3/smbd/open.c" (in "defer_open_done()"):

ret = schedule_deferred_open_message_smb(state->sconn, state->mid);

SMB_ASSERT(ret);

Now the two stack traces aren't particularly similar, though both seemed to
be processing queued event callbacks.  If I set "inherit permissions" to
"yes", although Samba does seem to have difficulties (all access to the
server locks up for a while, then processes report that that network share
has gone away, "Windows Explorer" gets typically completely bemused and
confused, .), there's no core-dump.

Anything else I can do/provide/. to help diagnose & fix this?

Cheers folks!

Tris.