smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf

[Youzhong Yang]

Hi All,

I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:


  *   sendfile() first returns 35 (EAGAIN):

# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
23  64274                  sendfile:return
              libc.so.7`__sys_sendfile+0xa
              libsmbd-base-samba4.so`vfswrap_sendfile+0x127
              libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
              libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
              libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35


  *   then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:

# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16  63748                    writev:return
              libc.so.7`_writev+0xa
              libsys-rw-samba4.so`sys_writev+0x21
              libsys-rw-samba4.so`write_data_iov+0x88
              libsys-rw-samba4.so`write_data+0x39
              libsmbd-base-samba4.so`fake_sendfile+0x16e
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35


  *   and this is the stack trace of smbd:

# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped

Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
  * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
    frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
    frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
    frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
    frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
    frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
    frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
    frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
    frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
    frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
    frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
    frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
    frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
    frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
    frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
    frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
    frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
    frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
    frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
    frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
    frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
    frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
    frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
    frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
    frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
    frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
    frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
    frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
    frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
    frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
    frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
    frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
    frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
    frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
    frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
    frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
    frame #36: 0x0000000001027cf0 smbd`_start + 384

In my smb.conf, I have socket options set as follows:

socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5

By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?

Thanks,

--Youzhong