[Samba] systemctl stop winbind (with CTDB) hanging

[Martin Schwenke]

I'm running a clustered Samba stack in a container, with systemd inside
the container.  systemctl reliably gets stuck when shutting down
winbind, which means my CTDB startup (which attempts to force winbind
shutdown before startup) times out.

If I try from the command-line, I see this:

# systemctl start winbind
# time systemctl stop winbind </dev/null

real  1m30.168s
user  0m0.004s
sys   0m0.000s

So, this is hitting DefaultTimeoutStopSec.  The absolutely remarkable
thing is that systemd *knows* that winbind has already exited:

# systemctl status winbind
● winbind.service - Samba Winbind Daemon
   Loaded: loaded (/usr/lib/systemd/system/winbind.service; disabled; vendor preset: disa>
   Active: deactivating (final-sigterm) since Tue 2023-10-10 01:03:10 UTC; 53s ago
     Docs: man:winbindd(8)
           man:samba(7)
           man:smb.conf(5)
  Process: 10770 ExecStart=/usr/sbin/winbindd --foreground --no-process-group $WINBINDOPT>
 Main PID: 10770 (code=exited, status=0/SUCCESS)
   Status: "winbindd: ready to serve connections..."
    Tasks: 0 (limit: 1225141)
   Memory: 248.0K
   CGroup: /docker/2021fe3169835c9fc632f84035d600e3a635e5da79628a487135500b65efd879/syste>

How can it be that systemd knows that the process is gone but is still
"deactivating"?  Is there something we need to do to our winbind.system
file to avoid this nonsense?  I know I can probably change
TimeoutStopSec to < 90s, but that seems like something I shouldn't have
to do.  Is there something we need to do in winbindd?

I frequently see a similar thing when stopping CTDB (which still has
ctdb.system using forking mode).  Is this a known systemd problem?  I'm
reluctant to open a systemd issue because my capacity to cope with
being told I'm stupid is limited at the moment.

OS in the host and container is Rocky Linux 8.8.

Thanks for any suggestions...

peace & happiness,
martin