smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Youzhong Yang
Youzhong.Yang at mathworks.com
Fri Nov 3 20:47:09 UTC 2017
Don't know if it's relevant, the manpage of sendfile() says:
HISTORY
The sendfile() system call first appeared in FreeBSD 3.0. This manual
page first appeared in FreeBSD 3.1. In FreeBSD 10 support for sending
shared memory descriptors had been introduced. In FreeBSD 11 a non-
blocking implementation had been introduced.
From: Youzhong Yang
Sent: Friday, November 03, 2017 4:28 PM
To: samba-technical at lists.samba.org
Subject: RE: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Sorry, I meant:
if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened.
From: Youzhong Yang
Sent: Friday, November 03, 2017 4:27 PM
To: samba-technical at lists.samba.org<mailto:samba-technical at lists.samba.org>
Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Hi All,
I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:
* sendfile() first returns 35 (EAGAIN):
# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU ID FUNCTION:NAME
23 64274 sendfile:return
libc.so.7`__sys_sendfile+0xa
libsmbd-base-samba4.so`vfswrap_sendfile+0x127
libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
libtalloc.so.2.1.9`_tc_free_internal+0x152
libtalloc.so.2.1.9`_tc_free_children_internal+0xac
libtalloc.so.2.1.9`_tc_free_internal+0x331
libtalloc.so.2.1.9`_talloc_free_internal+0xb2
libtalloc.so.2.1.9`_talloc_free+0x114
libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35
* then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:
# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16 63748 writev:return
libc.so.7`_writev+0xa
libsys-rw-samba4.so`sys_writev+0x21
libsys-rw-samba4.so`write_data_iov+0x88
libsys-rw-samba4.so`write_data+0x39
libsmbd-base-samba4.so`fake_sendfile+0x16e
libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
libtalloc.so.2.1.9`_tc_free_internal+0x152
libtalloc.so.2.1.9`_tc_free_children_internal+0xac
libtalloc.so.2.1.9`_tc_free_internal+0x331
libtalloc.so.2.1.9`_talloc_free_internal+0xb2
libtalloc.so.2.1.9`_talloc_free+0x114
libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35
* and this is the stack trace of smbd:
# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped
Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
* frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
frame #36: 0x0000000001027cf0 smbd`_start + 384
In my smb.conf, I have socket options set as follows:
socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5
By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?
Thanks,
--Youzhong
More information about the samba-technical
mailing list