"smbd" core-dump as a result of changing "inherit permissions" to "no" in "smb.conf".
Jeremy Allison
jra at samba.org
Fri Sep 27 10:15:32 MDT 2013
On Fri, Sep 27, 2013 at 05:07:35PM +0100, Tris Mabbs wrote:
> Interesting Samba issue for the gurus:
>
>
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
> 16:19:32.718301, 0] ../lib/util/fault.c:72(fault_report)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]
> ===============================================================
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
> 16:19:32.718388, 0] ../lib/util/fault.c:73(fault_report)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] INTERNAL
> ERROR: Signal 6 in pid 2392 (4.2.0pre1-GIT-89200c2)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] Please read
> the Trouble-Shooting section of the Samba HOWTO
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
> 16:19:32.718502, 0] ../lib/util/fault.c:75(fault_report)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error]
> ===============================================================
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
> 16:19:32.718589, 0] ../source3/lib/util.c:785(smb_panic_s3)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] PANIC (pid
> 2392): internal error
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
> 16:19:32.786239, 0] ../source3/lib/util.c:896(log_stack_trace)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] BACKTRACE: 24
> stack frames:
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #0
> /usr/local/samba/lib/libsmbconf.so.0'log_stack_trace+0x2b [0xfd223996]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #1
> /usr/local/samba/lib/libsmbconf.so.0'smb_panic_s3+0x80 [0xfd2237fd]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #2
> /usr/local/samba/lib/libsamba-util.so.0.0.1'smb_panic+0x2a [0xfe5aab1e]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #3
> /usr/local/samba/lib/libsamba-util.so.0.0.1'sig_fault+0x0 [0xfe5aa7f7]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #4
> /usr/local/samba/lib/libsamba-util.so.0.0.1'sig_fault+0x11 [0xfe5aa808]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #5
> /lib/libc.so.1'__sighndlr+0x15 [0xfe6a47d5]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #6
> /lib/libc.so.1'call_user_handler+0x2d2 [0xfe697dae]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #7
> /lib/libc.so.1'_lwp_kill+0x15 [0xfe6a9265]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #8
> /lib/libc.so.1'raise+0x25 [0xfe650fed]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #9
> /lib/libc.so.1'abort+0x77 [0xfe6286df]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #10
> /lib/libc.so.1'0x48add [0xfe628add]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #11
> /usr/local/samba/lib/libsmbconf.so.0'msg_channel_trigger+0xc5 [0xfd232b70]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #12
> /usr/local/samba/lib/private/libtevent.so.0.9.19'tevent_common_loop_immediat
> e+0x1a4 [0xfe324
>
> e8d]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #13
> /usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x4a [0xfd23fb1c]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #14
> /usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x66 [0xfd2400fd]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #15
> /usr/local/samba/lib/private/libtevent.so.0.9.19'_tevent_loop_once+0xf3
> [0xfe32422a]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #16
> /usr/local/samba/lib/private/libsmbd_base.so'smbd_process+0x13ff
> [0xfdf5f283]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #17
> /usr/local/samba/sbin/smbd'smbd_accept_connection+0x435 [0x805a21d]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #18
> /usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x483 [0xfd23ff55]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #19
> /usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x18a [0xfd240221]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #20
> /usr/local/samba/lib/private/libtevent.so.0.9.19'_tevent_loop_once+0xf3
> [0xfe32422a]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #21
> /usr/local/samba/sbin/smbd'smbd_parent_loop+0x8c [0x805afdc]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #22
> /usr/local/samba/sbin/smbd'main+0x1951 [0x805ca93]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] #23
> /usr/local/samba/sbin/smbd'_start+0x83 [0x8055823]
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] [2013/09/27
> 16:19:32.820147, 0] ../source3/lib/dumpcore.c:317(dump_core)
>
> Sep 27 16:19:32 Gateway smbd[2392]: [ID 702911 daemon.error] dumping core
> in /var/samba/log/cores/smbd
>
>
>
> This started yesterday (which was why I tried pulling the latest snapshot)
> after I made a single change to our "smb.conf" file:
>
>
>
> $ diff smb.conf
> /.zfs/snapshot/__PeriodicSnapshot__2013.269.-09-26_15:00/etc/sfw/samba/smb.c
> onf
>
> 216,217c216
>
> < inherit permissions = no
>
> < # inherit permissions = yes
>
> ---
>
> > inherit permissions = yes
>
> $
>
>
>
> All seems to work fine until I stress the Samba server - in this case by
> streaming about 8Gb to a file.
>
>
>
> I changed the "inherit permissions" setting to try and get to the bottom of
> a really annoying problem whereby we keep getting spurious permissions
> errors on files on Samba shares (can't write some files, can't execute
> others). Changing the setting apparently resolved those problems (though I
> hadn't finished verifying that), but causes "smbd" to core-dump instead.
>
>
>
> It's not a problem in interpreting the setting from "smb.conf"; the default
> value for this option is "no" and if I just comment out both "inherit
> permissions" lines (so the default behaviour applies) then "smbd" still
> core-dumps as soon as the server gets stressed (poor little server .).
> Anyway, the configuration file parsing code is surely pretty mature by now -
> I wouldn't expect any problems of that sort.
>
>
>
> I would poke the core dump with "gdb", but unfortunately the "gdb" supplied
> with "Solaris 11" is terminally brain-damaged and always reports any error
> as being at the first line of code in the source file. I've not yet managed
> to compile up any later "gdb" (to see whether that fixes it) - that's about
> item number 49 on my current "urgent" to-do list :(. So that stack trace is
> about as close as I can get, unless a disassembly dump of the code around
> "/usr/local/samba/lib/libsmbconf.so.0'msg_channel_trigger+0xc5 [0xfd232b70]"
> would be of any use? However the core dumps I was getting *yesterday* are
> *slightly* less unhelpful:
>
>
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
> 20:18:51.589417, 0] ../source3/smbd/open.c:1527(defer_open_done)
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] PANIC: assert
> failed at ../source3/smbd/open.c(1527): ret
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
> 20:18:51.589593, 0] ../source3/lib/util.c:810(smb_panic_s3)
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] PANIC (pid
> 2005): assert failed: ret
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
> 20:18:51.638743, 0] ../source3/lib/util.c:921(log_stack_trace)
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] BACKTRACE: 19
> stack frames:
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #0
> /usr/local/samba/lib/libsmbconf.so.0'log_stack_trace+0x2b [0xfd232aac]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #1
> /usr/local/samba/lib/libsmbconf.so.0'smb_panic_s3+0x80 [0xfd232913]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #2
> /usr/local/samba/lib/libsamba-util.so.0.0.1'smb_panic+0x2a [0xfe5aab1e]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #3
> /usr/local/samba/lib/private/libsmbd_base.so'defer_open_done+0x203
> [0xfdf2cbdd]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #4
> /usr/local/samba/lib/private/libtevent.so.0.9.18'_tevent_req_notify_callback
> +0x5c [0xfe3257ff
>
> ]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #5
> /usr/local/samba/lib/private/libtevent.so.0.9.18'tevent_req_finish+0x2d
> [0xfe325832]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #6
> /usr/local/samba/lib/private/libtevent.so.0.9.18'tevent_req_timedout+0x69
> [0xfe325b6c]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #7
> /usr/local/samba/lib/private/libtevent.so.0.9.18'tevent_common_loop_timer_de
> lay+0x229 [0xfe32
>
> 9c03]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #8
> /usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x70 [0xfd24d78e]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #9
> /usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x18a [0xfd24de6d]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #10
> /usr/local/samba/lib/private/libtevent.so.0.9.18'_tevent_loop_once+0xf3
> [0xfe3241ea]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #11
> /usr/local/samba/lib/private/libsmbd_base.so'smbd_process+0x13ff
> [0xfdf57625]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #12
> /usr/local/samba/sbin/smbd'smbd_accept_connection+0x435 [0x8059f6d]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #13
> /usr/local/samba/lib/libsmbconf.so.0'run_events_poll+0x483 [0xfd24dba1]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #14
> /usr/local/samba/lib/libsmbconf.so.0's3_event_loop_once+0x18a [0xfd24de6d]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #15
> /usr/local/samba/lib/private/libtevent.so.0.9.18'_tevent_loop_once+0xf3
> [0xfe3241ea]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #16
> /usr/local/samba/sbin/smbd'smbd_parent_loop+0x8c [0x805ad2c]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #17
> /usr/local/samba/sbin/smbd'main+0x1951 [0x805c7e3]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] #18
> /usr/local/samba/sbin/smbd'_start+0x83 [0x8055783]
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] [2013/09/26
> 20:18:51.671743, 0] ../source3/lib/dumpcore.c:317(dump_core)
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error] dumping core
> in /var/samba/log/cores/smbd
>
> Sep 26 20:18:51 Gateway smbd[2005]: [ID 702911 daemon.error]
>
>
>
> So that's actually showing an assertion failure at line 1527 of
> "source3/smbd/open.c" (in "defer_open_done()"):
>
>
>
> ret = schedule_deferred_open_message_smb(state->sconn, state->mid);
>
> SMB_ASSERT(ret);
>
>
>
> Now the two stack traces aren't particularly similar, though both seemed to
> be processing queued event callbacks. If I set "inherit permissions" to
> "yes", although Samba does seem to have difficulties (all access to the
> server locks up for a while, then processes report that that network share
> has gone away, "Windows Explorer" gets typically completely bemused and
> confused, .), there's no core-dump.
>
>
>
> Anything else I can do/provide/. to help diagnose & fix this?
Add the line:
panic action = /bin/sleep 9999999
in the [global] section of your smb.conf file.
Build with symbols, and when it crashes attach
to the parent process of the sleep with a debugger
and it should be frozen at the right place.
Having said this this looks like something we
fixed w.r.t. dos filenames hitting a race
condition with open requests.
But the above proceedure should give us the
info we need to track it down.
Jeremy.
More information about the samba-technical
mailing list