FW: Sporadic permission denied error in Samba 4.3 and 4.4, 4.2 OK

Tue Nov 15 18:37:23 UTC 2016

Thanks Jeremy and Volker.

I tested messaging using unix_msg_test_drain and unix_msg_test_source, there's nothing out of ordinary.

I dtraced sendmsg() and here are 3 error codes that it returns during the testing:

13: EACCES
2: ENOENT
11: EAGAIN

What concerns me is the error code EACCES, its stack looks like the following:

              libc.so.1`__so_sendmsg+0xa
              libsocket.so.1`sendmsg+0x1e
              libmessages-dgm-samba4.so`unix_dgram_send_job+0x4e
              libmessages-dgm-samba4.so`pthreadpool_server+0x17e
              libc.so.1`_thrp_setup+0x8a
              libc.so.1`_lwp_start

And indeed its euid is not 0(root) when EACCES was returned.

I tried to set euid and egid to 0 inside unix_dgram_send_job() at the beginning and the restore them at the end, then some of the smbd processes crashed with the following stack trace:

  BACKTRACE: 38 stack frames:
   #0 /tmw-nas-3p/samba/lib/libsmbconf.so.0'log_stack_trace+0x1f [0xffffbf7fb11585b0]
   #1 /tmw-nas-3p/samba/lib/libsmbconf.so.0'smb_panic_s3+0x6f [0xffffbf7fb1158424]
   #2 /tmw-nas-3p/samba/lib/libsamba-util.so.0'smb_panic+0x28 [0xffffbf7fb11ed750]
   #3 /tmw-nas-3p/samba/lib/private/libsamba3-util-samba4.so'assert_uid+0xb8 [0xffffbf7fb0d44935]
   #4 /tmw-nas-3p/samba/lib/private/libsamba3-util-samba4.so'set_effective_uid+0x29 [0xffffbf7fb0d44aa2]
   #5 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'become_uid+0x8b [0xffffbf7fb0ab4bcb]
   #6 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'become_id+0x26 [0xffffbf7fb0ab4c8a]
   #7 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'set_unix_security_ctx+0x58 [0xffffbf7fb0ab5397]
   #8 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'pop_sec_ctx+0x1e5 [0xffffbf7fb0ab59b1]
   #9 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'smbd_unbecome_root+0x9 [0xffffbf7fb0a9bd9a]
   #10 /tmw-nas-3p/samba/lib/private/libsmbd-shim-samba4.so'unbecome_root+0x19 [0xffffbf7fb21016e8]
   #11 /tmw-nas-3p/samba/lib/libsmbconf.so.0'messaging_send_iov_from+0x252 [0xffffbf7fb11656f0]
   #12 /tmw-nas-3p/samba/lib/libsmbconf.so.0'messaging_send_iov+0x74 [0xffffbf7fb116578a]
   #13 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'notify_add+0x214 [0xffffbf7fb0b37129]
   #14 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'change_notify_create+0x2aa [0xffffbf7fb0b3583c]
   #15 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'smbd_smb2_notify_send+0x33a [0xffffbf7fb0b13593]
   #16 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'smbd_smb2_request_process_notify+0x3a5 [0xffffbf7fb0b12e96]
   #17 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'smbd_smb2_request_dispatch+0x2601 [0xffffbf7fb0af2ee4]
   #18 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'smbd_smb2_request_dispatch_immediate+0xed [0xffffbf7fb0af4700]
   #19 /tmw-nas-3p/samba/lib/private/libtevent.so.0'tevent_common_loop_immediate+0x1f5 [0xffffbf7fb337a2c0]
   #20 /tmw-nas-3p/samba/lib/private/libtevent.so.0'epoll_event_loop_once+0x76 [0xffffbf7fb3382a4e]
   #21 /tmw-nas-3p/samba/lib/private/libtevent.so.0'std_event_loop_once+0x5c [0xffffbf7fb337f9a0]
   #22 /tmw-nas-3p/samba/lib/private/libtevent.so.0'_tevent_loop_once+0xf9 [0xffffbf7fb33793f8]
   #23 /tmw-nas-3p/samba/lib/private/libtevent.so.0'tevent_common_loop_wait+0x25 [0xffffbf7fb3379647]
   #24 /tmw-nas-3p/samba/lib/private/libtevent.so.0'std_event_loop_wait+0x5c [0xffffbf7fb337fa42]
   #25 /tmw-nas-3p/samba/lib/private/libtevent.so.0'_tevent_loop_wait+0x2b [0xffffbf7fb337971b]
   #26 /tmw-nas-3p/samba/lib/private/libsmbd-base-samba4.so'smbd_process+0xbbb [0xffffbf7fb0ad7b5a]
   #27 /tmw-nas-3p/samba/sbin/smbd'smbd_accept_connection+0x3c1 [0x40ef40]
   #28 /tmw-nas-3p/samba/lib/private/libtevent.so.0'epoll_event_loop+0x3a5 [0xffffbf7fb33824ca]
   #29 /tmw-nas-3p/samba/lib/private/libtevent.so.0'epoll_event_loop_once+0x115 [0xffffbf7fb3382aed]
   #30 /tmw-nas-3p/samba/lib/private/libtevent.so.0'std_event_loop_once+0x5c [0xffffbf7fb337f9a0]
   #31 /tmw-nas-3p/samba/lib/private/libtevent.so.0'_tevent_loop_once+0xf9 [0xffffbf7fb33793f8]
   #32 /tmw-nas-3p/samba/lib/private/libtevent.so.0'tevent_common_loop_wait+0x25 [0xffffbf7fb3379647]
   #33 /tmw-nas-3p/samba/lib/private/libtevent.so.0'std_event_loop_wait+0x5c [0xffffbf7fb337fa42]
   #34 /tmw-nas-3p/samba/lib/private/libtevent.so.0'_tevent_loop_wait+0x2b [0xffffbf7fb337971b]
   #35 /tmw-nas-3p/samba/sbin/smbd'smbd_parent_loop+0x9a [0x40fcbb]
   #36 /tmw-nas-3p/samba/sbin/smbd'main+0x17f2 [0x411669]
   #37 /tmw-nas-3p/samba/sbin/smbd'_start+0x6c [0x40946c]

For now I don't know how to fix the EACCES error. Please advise.

Thanks,

--Youzhong

-----Original Message-----
From: Jeremy Allison [mailto:jra at samba.org] 
Sent: Monday, November 14, 2016 5:05 PM
To: Volker Lendecke <vl at samba.org>
Cc: Youzhong Yang <Youzhong.Yang at mathworks.com>; samba-technical at lists.samba.org
Subject: Re: FW: Sporadic permission denied error in Samba 4.3 and 4.4, 4.2 OK

On Mon, Nov 14, 2016 at 10:54:57PM +0100, Volker Lendecke wrote:
> On Mon, Nov 14, 2016 at 01:50:01PM -0800, Jeremy Allison wrote:
> > On Mon, Nov 14, 2016 at 09:42:53PM +0000, Youzhong Yang wrote:
> > > Yes, it's illumos/SmartOS system running Samba. It's possible there's some kind of bug either in kernel or in user-land programming. If you guys can point out a few critical points to put debug statements, I would be more than happy to debug the issue here.
> > 
> > There's no magic debug that will show a kernel bug. If you're 
> > reporting it right, the issue is on 4.5.x on your kernel a sendmsg() 
> > call on a connected UNIX domain socket is dropping the message.
> > 
> > That simply should not happen. For a lease break message this isn't 
> > anything fancy such as fd-passing, this should be a simple datagram 
> > message.
> > 
> > Look at the test program in:
> > 
> > source3/lib/unix_msg/test_source.c
> > 
> > Ensure this works correctly on your kernel.
> > 
> > Can you run directly out of master on a test system, so we know 
> > we're looking at identical code ?
> 
> Master has changed vastly, unix_msg.c is gone.

Yeah I know, I was giving him advice on looking at his current source :-). Ensuring that test works should be task #1 IMHO.

But if there really is a problem, I'd like to reproduce on master to fix going forward :-).

> Also, I still have some FreeBSD-specific patches for an overload
> situation: FreeBSD does not block, but instead returns ENOBUFS. We 
> need to a) treat ENOBUFS the same as EWOULDBLOCK in the nonblocking 
> path, and b) do send-polling in the thread until EWOULDBLOCK vanishes. 
> I could imagine something similar in Solaris too.

Oh that's possible ! Youzhong, can you
use dtrace to find out if your kernel is returning ENOBUFS to the sendmsg() in the case where the message is being dropped ?

Jeremy.