smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Youzhong Yang
Youzhong.Yang at mathworks.com
Fri Nov 3 20:28:20 UTC 2017
Sorry, I meant:
if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened.
From: Youzhong Yang
Sent: Friday, November 03, 2017 4:27 PM
To: samba-technical at lists.samba.org
Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Hi All,
I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:
* sendfile() first returns 35 (EAGAIN):
# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU ID FUNCTION:NAME
23 64274 sendfile:return
libc.so.7`__sys_sendfile+0xa
libsmbd-base-samba4.so`vfswrap_sendfile+0x127
libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
libtalloc.so.2.1.9`_tc_free_internal+0x152
libtalloc.so.2.1.9`_tc_free_children_internal+0xac
libtalloc.so.2.1.9`_tc_free_internal+0x331
libtalloc.so.2.1.9`_talloc_free_internal+0xb2
libtalloc.so.2.1.9`_talloc_free+0x114
libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35
* then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:
# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16 63748 writev:return
libc.so.7`_writev+0xa
libsys-rw-samba4.so`sys_writev+0x21
libsys-rw-samba4.so`write_data_iov+0x88
libsys-rw-samba4.so`write_data+0x39
libsmbd-base-samba4.so`fake_sendfile+0x16e
libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
libtalloc.so.2.1.9`_tc_free_internal+0x152
libtalloc.so.2.1.9`_tc_free_children_internal+0xac
libtalloc.so.2.1.9`_tc_free_internal+0x331
libtalloc.so.2.1.9`_talloc_free_internal+0xb2
libtalloc.so.2.1.9`_talloc_free+0x114
libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35
* and this is the stack trace of smbd:
# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped
Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
* frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
frame #36: 0x0000000001027cf0 smbd`_start + 384
In my smb.conf, I have socket options set as follows:
socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5
By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?
Thanks,
--Youzhong
More information about the samba-technical
mailing list