smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in smb.conf
Timur I. Bakeyev
timur at freebsd.org
Mon Nov 6 00:54:29 UTC 2017
Yes, I'm aware of the inefficiency of sendfile() on ZFS, but hanging of
Samba is something new. I'd suggest you to file a bug on bugzilla.samba.org
regarding that.
With best regards,
Timur Bakeyev.
On Mon, Nov 6, 2017 at 1:51 AM, Youzhong Yang <Youzhong.Yang at mathworks.com>
wrote:
> Hi Timur,
>
>
>
> That’s too bad. Yes, I am using zfs on freebsd.
>
>
>
> This one explains why:
>
>
>
> https://wiki.freebsd.org/DevSummit/20170907/ZFS
>
>
>
> “ZFS currently does not play nicely with sendfile(2). The idea with
> sendfile(2) is to share pages between the buffer cache and the mbuf, to
> prevent copying. The call results in a range of bytes from an open file
> descriptor being faulted into the buffer cache, and then those pages being
> shared all the way through to the network stack until the bytes are
> transmitted.
>
> With ZFS, since it does not use the OS buffer cache, data is copied from
> the ZFS ARC to the buffer cache, where it is then shared with the mbuf
> etc. Solaris has a solution to solve this issue, and it might make sense to
> adapt our sendfile mechanism to work in a similar way. [Solaris PSARC: Copy
> Reduction Interface](https://www.mail-archive.com/opensolaris-arc@
> mail.opensolaris.org/msg17203.html) This creates two new VFS operations,
> reqzcbuf which asks the filesystem to prepare a UIO for zero copy
> operations. In the case of ZFS, this involves allocating a new array to
> track the ARC headers of each buffer that is loaned out from the ZFS ARC.”
>
> But it’s really bad that using sendfile() causes Samba to hang.
>
> Thanks,
>
> -Youzhong
>
> *From:* Timur I. Bakeyev [mailto:timur at com.bat.ru]
> *Sent:* Sunday, November 05, 2017 7:23 PM
> *To:* Youzhong Yang <Youzhong.Yang at mathworks.com>
> *Cc:* samba-technical at lists.samba.org
> *Subject:* Re: smbd stuck at writev() on FreeBSD if 'use sendfile = yes'
> in smb.conf
>
>
>
> Hi, Youzhong!
>
> You didn't specify what file system do you use. Please, keep in mind that
> ZFS doesn't support 'sendfile' properly.
>
>
>
> With regards,
>
> Timur Bakeyev
>
>
>
> On Fri, Nov 3, 2017 at 9:28 PM, Youzhong Yang via samba-technical <
> samba-technical at lists.samba.org> wrote:
>
> Sorry, I meant:
>
> if 'use sendfile' is set to no, everything works well. But if it is set to
> yes, here is what happened.
>
> From: Youzhong Yang
> Sent: Friday, November 03, 2017 4:27 PM
> To: samba-technical at lists.samba.org
> Subject: smbd stuck at writev() on FreeBSD if 'use sendfile = yes' in
> smb.conf
>
>
> Hi All,
>
> I am struggling with a smbd issue on FreeBSD (11.1). In my smb.conf, if
> 'use sendfile' is set to yes, everything works well. But if it is set to
> no, here is what happened:
>
>
> * sendfile() first returns 35 (EAGAIN):
>
> # dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack();
> printf("errno = %d", errno);}'
> dtrace: description 'syscall::sendfile:return ' matched 1 probe
> CPU ID FUNCTION:NAME
> 23 64274 sendfile:return
> libc.so.7`__sys_sendfile+0xa
> libsmbd-base-samba4.so`vfswrap_sendfile+0x127
> libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
> libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
> libtalloc.so.2.1.9`_tc_free_internal+0x152
> libtalloc.so.2.1.9`_tc_free_children_internal+0xac
> libtalloc.so.2.1.9`_tc_free_internal+0x331
> libtalloc.so.2.1.9`_talloc_free_internal+0xb2
> libtalloc.so.2.1.9`_talloc_free+0x114
> libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
> libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
> libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
> libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
> libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
> libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
> libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
> libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
> libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
> libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
> libtevent.so.0.9.31`poll_event_loop_poll+0x75c
> errno = 35
>
>
> * then smbd keeps calling writev(), which always returns -1 with errno
> being EAGAIN:
>
> # dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno
> = %d", errno);}'
> 16 63748 writev:return
> libc.so.7`_writev+0xa
> libsys-rw-samba4.so`sys_writev+0x21
> libsys-rw-samba4.so`write_data_iov+0x88
> libsys-rw-samba4.so`write_data+0x39
> libsmbd-base-samba4.so`fake_sendfile+0x16e
> libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
> libtalloc.so.2.1.9`_tc_free_internal+0x152
> libtalloc.so.2.1.9`_tc_free_children_internal+0xac
> libtalloc.so.2.1.9`_tc_free_internal+0x331
> libtalloc.so.2.1.9`_talloc_free_internal+0xb2
> libtalloc.so.2.1.9`_talloc_free+0x114
> libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
> libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
> libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
> libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
> libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
> libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
> libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
> libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
> libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
> errno = 35
>
>
> * and this is the stack trace of smbd:
>
> # echo bt | lldb -p 1810
> (lldb) process attach --pid 1810
> Process 1810 stopped
>
> Executable module set to "/tmw-nas-3p/samba/sbin/smbd".
> Architecture set to: x86_64--freebsd11.1.
> (lldb) bt
> * thread #1
> * frame #0: 0x0000000805b4ba2a libc.so.7`_writev + 10
> frame #1: libthr.so.3`__thr_writev(fd=<unavailable>,
> iov=<unavailable>, iovcnt=<unavailable>) at thr_syscalls.c:634
> frame #2: libsys-rw-samba4.so`sys_writev(fd=40,
> iov=0x00007fffffffcd58, iovcnt=1) at sys_rw.c:101
> frame #3: libsys-rw-samba4.so`write_data_iov(fd=40,
> orig_iov=0x00007fffffffcd58, iovcnt=1) at sys_rw_data.c:49
> frame #4: libsys-rw-samba4.so`write_data(fd=40,
> buffer=0x000000081e7ac340, n=65536) at sys_rw_data.c:94
> frame #5: libsmbd-base-samba4.so`fake_sendfile(xconn=0x0000000812c74d60,
> fsp=0x000000081e438ce0, startpos=63438848, nread=839680) at reply.c:3329
> frame #6: libsmbd-base-samba4.so`smb2_sendfile_send_data(state=0x000000081e432660)
> at smb2_read.c:283
> frame #7: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432600,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1078
> frame #8: libtalloc.so.2`_tc_free_children_internal(tc=0x000000081e432080,
> ptr=0x000000081e4320e0, location="../source3/smbd/smb2_server.c:3697") at
> talloc.c:1593
> frame #9: libtalloc.so.2`_tc_free_internal(tc=0x000000081e432080,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1104
> frame #10: libtalloc.so.2`_talloc_free_internal(ptr=0x000000081e4320e0,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1174
> frame #11: libtalloc.so.2`_talloc_free(ptr=0x000000081e4320e0,
> location="../source3/smbd/smb2_server.c:3697") at talloc.c:1716
> frame #12: libsmbd-base-samba4.so`smbd_smb2_flush_send_queue(xconn=0x0000000812c74d60)
> at smb2_server.c:3697
> frame #13: libsmbd-base-samba4.so`smbd_smb2_request_reply(req=0x000000081e4320e0)
> at smb2_server.c:3011
> frame #14: libsmbd-base-samba4.so`smbd_smb2_request_done_ex(req=0x000000081e4320e0,
> status=(v = 0), body=(data = "\x11", length = 16), dyn=0x00007fffffffd680,
> location="../source3/smbd/smb2_read.c:164") at smb2_server.c:3159
> frame #15: libsmbd-base-samba4.so`smbd_smb2_request_read_done(subreq=0x0000000000000000)
> at smb2_read.c:164
> frame #16: libtevent.so.0`_tevent_req_notify_callback(req=0x000000081e4324d0,
> location="../source3/smbd/smb2_server.c:1387") at tevent_req.c:120
> frame #17: libsmbd-base-samba4.so`smbd_smb2_request_pending_queue(req=0x000000081e4320e0,
> subreq=0x000000081e4324d0, defer_time=500) at smb2_server.c:1387
> frame #18: libsmbd-base-samba4.so`smbd_smb2_request_process_read(req=0x000000081e4320e0)
> at smb2_read.c:109
> frame #19: libsmbd-base-samba4.so`smbd_smb2_request_dispatch(req=0x000000081e4320e0)
> at smb2_server.c:2641
> frame #20: libsmbd-base-samba4.so`smbd_smb2_io_handler(xconn=0x0000000812c74d60,
> fde_flags=1) at smb2_server.c:3950
> frame #21: libsmbd-base-samba4.so`smbd_smb2_connection_handler(ev=0x0000000812c5a1a0,
> fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c74d60) at
> smb2_server.c:3988
> frame #22: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0,
> tvalp=0x00007fffffffe068) at tevent_poll.c:605
> frame #23: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent_poll.c:662
> frame #24: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent.c:721
> frame #25: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent_poll.c:678
> frame #26: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/process.c:4140") at tevent.c:863
> frame #27: libsmbd-base-samba4.so`smbd_process(ev_ctx=0x0000000812c5a1a0,
> msg_ctx=0x0000000812c4f120, sock_fd=40, interactive=false) at process.c:4140
> frame #28: smbd`smbd_accept_connection(ev=0x0000000812c5a1a0,
> fde=0x0000000812c4fba0, flags=1, private_data=0x0000000812c428c0) at
> server.c:1024
> frame #29: libtevent.so.0`poll_event_loop_poll(ev=0x0000000812c5a1a0,
> tvalp=0x00007fffffffe548) at tevent_poll.c:605
> frame #30: libtevent.so.0`poll_event_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent_poll.c:662
> frame #31: libtevent.so.0`_tevent_loop_once(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent.c:721
> frame #32: libtevent.so.0`poll_event_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent_poll.c:678
> frame #33: libtevent.so.0`_tevent_loop_wait(ev=0x0000000812c5a1a0,
> location="../source3/smbd/server.c:1391") at tevent.c:863
> frame #34: smbd`smbd_parent_loop(ev_ctx=0x0000000812c5a1a0,
> parent=0x0000000812c52680) at server.c:1391
> frame #35: smbd`main(argc=5, argv=0x00007fffffffed08) at server.c:2050
> frame #36: 0x0000000001027cf0 smbd`_start + 384
>
> In my smb.conf, I have socket options set as follows:
>
> socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576
> SO_KEEPALIVE TCP_KEEPIDLE=15000 TCP_KEEPINTVL=15000 TCP_KEEPCNT=5
>
> By the way, I am running Samba 4.6.8. Has anyone experienced the same
> issue? Is it a FreeBSD kernel bug?
>
> Thanks,
>
> --Youzhong
>
>
>
More information about the samba-technical
mailing list