Apply patch for socken listen backlog (FreeBSD)

[Volker Lendecke]

On Sat, Jan 12, 2019 at 10:21:40PM +0100, Osipov, Michael via samba-technical wrote:
> > This change looks of course ok. I'm not sure I like the #ifdef
> > FREEBSD, but if this is really something FreeBSD specific, then it
> > might be the right way to go.
> > 
> > I'd like to understand the patch and its necessity a bit better. The
> > listen backlog copes with the process doing the accept() being too
> > slow, right? Where exactly do you see the problems, where is smbd or
> > winbind too slow? The smbd process doing the accept should not do much
> > work before doing the fork, returning to polling the acceptor socket.
> > What is the scenario that you're fixing with this? If the parent smbd
> > does significant work between the accept calls, I think we also need
> > to fix that. Likewise for winbind.
> > 
> > What is it that makes this a FreeBSD specific problem?
> 
> This is netstat in idle:
> > # netstat -Lan
> > Current listen queue sizes (qlen/incqlen/maxqlen)
> > Proto Listen                           Local Address
> > tcp4  0/0/128                          127.0.0.1.6010
> > tcp4  0/0/50                           147.54.64.14.139
> > tcp4  0/0/50                           147.54.64.14.445
> > tcp4  0/0/128                          *.5555
> > tcp4  0/0/128                          *.21
> > tcp4  0/0/10                           127.0.0.1.25
> > tcp4  0/0/128                          *.22
> > tcp4  0/0/128                          *.10050
> > tcp4  0/0/128                          *.603
> > tcp4  0/0/128                          *.748
> > tcp4  0/0/128                          *.2049
> > tcp4  0/0/128                          *.1009
> > tcp4  0/0/128                          *.600
> > tcp4  0/0/128                          *.111
> > unix  0/0/5                            /var/db/samba4/winbindd_privileged/pipe
> > unix  0/0/5                            /var/run/samba4/winbindd/pipe
> > unix  0/0/5                            /var/run/samba4/nmbd/unexpected
> > unix  0/0/30                           /var/run/hald/dbus-EVU64eQXQL
> > unix  0/0/30                           /var/run/hald/dbus-1zay8aU2yN
> > unix  0/0/128                          /var/run/rpcbind.sock
> > unix  0/0/4                            /var/run/devd.pipe
> > unix  0/0/30                           /var/run/dbus/system_bus_socket
> > unix  0/0/4                            /var/run/devd.seqpacket.pipe
> 
> We had lockups with an overfull queue which is hard to monitor exactly at
> the time of the lockup because the entire system is unresponsive. I am not
> even 100% sure if that is the cause, but it could be.
> As you can see, everyone else is using -1, but Samba is not.

As I said, I'm fine with making that a config option. However, I'm not
certain this actually solves things. I would rather guess (and this is
just a guess) that it makes things worse.

The lockups and unusability I've seen so far were due to winbind
misbehaving, and due to fcntl locks being too slow. The winbind
misbehaviour has mostly been fixed, although we are still struggling
with DC failover behaviour. This is something that is actively being
worked on, although progress is slow there. The fcntl scalability
problems are not solved, but they don't happend anymore because Samba
for a while uses process shared robust mutexes for a while now. Those
are available in FreeBSD 11. There is one major fcntl user left that
can kill performance, and that's gencache.tdb. With Samba 4.10 that
one is gone too.

Why do I think it makes things worse to bump up the listen queue? In
the complete overload scenario we *have* to decline new connection
requests, and not queue them. With a larger listen queue more clients
will pile up for longer. This will cause clients to time out and
retry, making the problem worse. And while all those clients are
waiting in line, the system has no chance to recover from those
overloads.

Of course, all of this needs confirmation in a realistic test, or best
while watching a live system in that state.

> So I'd replace SMBD_LISTEN_BACKLOG and DEFAULT_LISTEN_BACKLOG with two
> options in the smb.conf and everyone will be happy.

Sure, go for it. But I would recommend finding the root cause for the
overloads and fix that.

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: 0551-370000-0, mailto:kontakt at sernet.de
Gesch.F.: Dr. Johannes Loxen und Reinhild Jung
AG Göttingen: HR-B 2816 - http://www.sernet.de