smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Youzhong Yang
Youzhong.Yang at mathworks.com
Mon Nov 6 00:51:30 UTC 2017
Hi Timur,
That’s too bad. Yes, I am using zfs on freebsd.
This one explains why:
https://wiki.freebsd.org/DevSummit/20170907/ZFS
“ZFS currently does not play nicely with sendfile(2). The idea with sendfile(2) is to share pages between the buffer cache and the mbuf, to prevent copying. The call results in a range of bytes from an open file descriptor being faulted into the buffer cache, and then those pages being shared all the way through to the network stack until the bytes are transmitted.
With ZFS, since it does not use the OS buffer cache, data is copied from the ZFS ARC to the buffer cache, where it is then shared with the mbuf etc. Solaris has a solution to solve this issue, and it might make sense to adapt our sendfile mechanism to work in a similar way. [Solaris PSARC: Copy Reduction Interface](https://www.mail-archive.com/opensolaris-arc@mail.opensolaris.org/msg17203.html) This creates two new VFS operations, reqzcbuf which asks the filesystem to prepare a UIO for zero copy operations. In the case of ZFS, this involves allocating a new array to track the ARC headers of each buffer that is loaned out from the ZFS ARC.”
But it’s really bad that using sendfile() causes Samba to hang.
Thanks,
-Youzhong
From: Timur I. Bakeyev [mailto:timur at com.bat.ru]
Sent: Sunday, November 05, 2017 7:23 PM
To: Youzhong Yang <Youzhong.Yang at mathworks.com>
Cc: samba-technical at lists.samba.org
Subject: Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Hi, Youzhong!
You didn't specify what file system do you use. Please, keep in mind that ZFS doesn't support 'sendfile' properly.
With regards,
Timur Bakeyev
On Fri, Nov 3, 2017 at 9:28 PM, Youzhong Yang via samba-technical <samba-technical at lists.samba.org<mailto:samba-technical at lists.samba.org>> wrote:
Sorry, I meant:
if 'use sendfile' is set to no, everything works well. But if it is set to yes, here is what happened.
From: Youzhong Yang
Sent: Friday, November 03, 2017 4:27 PM
To: samba-technical at lists.samba.org<mailto:samba-technical at lists.samba.org>
Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Hi All,
I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if 'use sendfile' is set to yes, everything works well. But if it is set to no, here is what happened:
* sendfile() first returns 35 (EAGAIN):
# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU ID FUNCTION:NAME
23 64274 sendfile:return
libc.so.7`__sys_sendfile+0xa
libsmbd-base-samba4.so`vfswrap_sendfile+0x127
libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
libtalloc.so.2.1.9`_tc_free_internal+0x152
libtalloc.so.2.1.9`_tc_free_children_internal+0xac
libtalloc.so.2.1.9`_tc_free_internal+0x331
libtalloc.so.2.1.9`_talloc_free_internal+0xb2
libtalloc.so.2.1.9`_talloc_free+0x114
libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35
* then smbd keeps calling writev(), which always returns -1 with errno being EAGAIN:
# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
16 63748 writev:return
libc.so.7`_writev+0xa
libsys-rw-samba4.so`sys_writev+0x21
libsys-rw-samba4.so`write_data_iov+0x88
libsys-rw-samba4.so`write_data+0x39
libsmbd-base-samba4.so`fake_sendfile+0x16e
libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
libtalloc.so.2.1.9`_tc_free_internal+0x152
libtalloc.so.2.1.9`_tc_free_children_internal+0xac
libtalloc.so.2.1.9`_tc_free_internal+0x331
libtalloc.so.2.1.9`_talloc_free_internal+0xb2
libtalloc.so.2.1.9`_talloc_free+0x114
libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35
* and this is the stack trace of smbd:
# echo bt | lldb -p 1810
(lldb) process attach --pid 1810
Process 1810 stopped
Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
Architecture set to: x86_64--freebsd11.1.
(lldb) bt
* thread #1
* frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
frame #1: libthr.so.3`__thr_writev(fd=<unavailable>, iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
frame #2: libsys-rw-samba4.so`sys_writev(fd=40, iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
frame #3: libsys-rw-samba4.so`write_data_iov(fd=40, orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
frame #4: libsys-rw-samba4.so`write_data(fd=40, buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60, fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660) at smb2_read.c:283
frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080, ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1593
frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60) at smb2_server.c:3697
frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0) at smb2_server.c:3011
frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0, status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680, location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000) at smb2_read.c:164
frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0, location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0, subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0) at smb2_read.c:109
frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0) at smb2_server.c:2641
frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60, fde_flags=1) at smb2_server.c:3950
frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at smb2_server.c:3988
frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe068) at tevent_poll.c:605
frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:662
frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:721
frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent_poll.c:678
frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/process.c:4140") at tevent.c:863
frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0, msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0, fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at server.c:1024
frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0, tvalp=0x00007fffffffe548) at tevent_poll.c:605
frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:662
frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:721
frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent_poll.c:678
frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0, location="../source3/smbd/server.c:1391") at tevent.c:863
frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0, parent=0x0000000812c52680) at server.c:1391
frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
frame #36: 0x0000000001027cf0 smbd`_start + 384
In my smb.conf, I have socket options set as follows:
socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576 SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5
By the way, I am running Samba 4.6.8. Has anyone experienced the same issue? Is it a FreeBSD kernel bug?
Thanks,
--Youzhong
More information about the samba-technical
mailing list