smbd: High CPU usage

[Jeremy Allison]

On Wed, Jul 18, 2018 at 11:38:25AM -0700, Jeremy Allison via samba-technical wrote:
> On Wed, Jul 18, 2018 at 09:20:10PM +0300, Vadim Lazovskiy via samba-technical wrote:
> > I would not agree.
> > sendfile is great. It makes you able to serve content directly from disk to
> > network without unnessesery copying to userspace. It saves you lots
> > syscalls and context switching.
> > As for our workload profile (1-2Gbit per server) it really saves CPU time.
> > 
> > If someone of samba developers treat this as bug and eager to fix it I
> > could provide extra information and do some debugging.
> 
> Need more details. Is this on Linux ? With SMB2 ?
> 
> It's probably this (from the Linux part of source3/lib/sendfile.c):
> 
>         while (total) {
>                 ssize_t nwritten;
>                 do {
>                         nwritten = sendfile(tofd, fromfd, &offset, total);
>                 } while (nwritten == -1 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK));
> 
> Try bracketing the sendfile with :
> 
> set_blocking(tofd, true);
> 
> ... sendfile call
> 
> set_blocking(tofd, false);
> 
> We set the send socket to non-blocking by default, but might need
> to explicitly change to blocking if sendfile is enabled (then set
> back to non-blocking after the sendfile).

Looking closer at this, it's interesting.

With SMB1 we explicitly set the incoming socket
to blocking - in smbd_add_connection() we have:

        xconn->transport.sock = sock_fd;
        smbd_echo_init(xconn);
        xconn->protocol = PROTOCOL_NONE;

        /* Ensure child is set to blocking mode */
        set_blocking(sock_fd,True);

with SMB2, we explicitly set it to non-blocking -
in smbd_initialize_smb2() we have:

        /* Ensure child is set to non-blocking mode */
        set_blocking(xconn->transport.sock, false);

which probably explains the sendfile problem if the client
is SMB2 (which most clients now are).

Now all our socket calls are tevent based, I
think we should probably change the SMB1 code
to match the SMB2 code and make the primary
socket non-blocking, then fix the sendfile
internal implementation to change to make the
socket temporarily blocking, and back to non-blocking
after the sendfile completes.

My guess is almost no-one turns on sendfile()
with SMB2 as Volker says unless you're on a
low power CPU it's better to use the async
pthread code by default. Having said that
we shouldn't spin on the CPU with SMB2 if
it gets turned on.

Inside recvfile() the code is structured to
allow it to work with either a non-blocking or
blocking socket. I could do the same for
the sendfile code path (which really needs
to work only with a blocking socket I think).

Jeremy.