[Samba] ctdb tcp kill: remaining connections

Martin Schwenke martin at meltin.net
Wed Mar 1 22:53:41 UTC 2023


On Wed, 1 Mar 2023 16:18:58 +0000, Ulrich Sibiller
<ulrich.sibiller at atos.net> wrote:

> Martin Schwenke schrieb am 28.02.2023 23:03:

> Yes, we have that version running for some days now. It works
> significantly better than the previous version of ctdb 4.15. Of all
> the 139 takeovers in the last 2 days (unfortunately there were so
> many due to some other problem) I have seen only 4 that did not kill
> all expected connections and the number of missing connections was
> significantly smaller than before:

> Feb 27 14:49:18 serverpnn1 ctdb-eventd[85619]: 10.interface.debug: Killed 0/4 TCP connections to released IP x.x.253.252
> Feb 27 15:14:14 serverpnn1 ctdb-eventd[85619]: 10.interface.debug: Killed 450/463 TCP connections to released IP x.x.253.252
> Feb 27 15:27:09 serverpnn1 ctdb-eventd[85619]: 10.interface.debug: Killed 62/63 TCP connections to released IP y.y.139.15
> 2023-02-28T18:42:15.065129+01:00 serverpnn2 ctdb-eventd[29607]: 10.interface.debug: Killed 157/158 TCP connections to released IP x.x.253.95

Not perfect, but better...

> > If this doesn't work then it might be possible to increase the number
> > of attempts from 50 to 100 here:
> > 
> >         state->max_attempts = 50;
> > This means each ctdb_killtcp could take up to 10s.  This increases the
> > chances of the releaseip event timing out.  

> What's the timeout for releaseip? I have not seen that value yet. max_attempts and release/takeover timeouts should be configurables anyway, I'd say.

The timeout for releaseip is EventScriptTimeout, which defaults to 30
(seconds).  This is the total time for all scripts running an event.

For ctdb_killtcp, when it was basically rewritten, we considered adding
options for max_attempts, but decided to see if it was foolproof.  We
could now add those options.  Patches welcome too...

> BTW: I have seen this just today, which seems to be hardcoded, too:
> 2023-03-01T02:50:14.938438+01:00 serverpnn0 ctdbd[24778]: Maximum monitor timeout count 20 reached. Making node unhealthy
> 2023-03-01T02:50:14.938593+01:00 serverpnn0 ctdbd[24778]: monitor event failed - disabling node
> 2023-03-01T02:50:14.938812+01:00 serverpnn0 ctdbd[24778]: Node became UNHEALTHY. Ask recovery master to reallocate IPs

MonitorTimeoutCount defaults to 20 but can also be changed.

For this, you should check the output of:

  ctdb event status legacy monitor

All the scripts should be taking less than about 1s.

If 50.samba is taking many seconds, then this may indicate a DNS issue.
I have some work-in-progress changes to address this, and also to
monitor DNS (and other hosts).

> > Another improvement might be to change this message:
> > 
> >         DBG_DEBUG("Sending tickle ACK for connection '%s'\n",
> > to INFO level and then (in the first instance) debug this at INFO level
> > instead.  That might be a good change.  Queued.  
> 
> Hmm, for debugging it could come handy to record those messages (that
> tend to flood logs) separately for reference.

True, but for a basic understanding of what happened, it could be good
to just use INFO level.  Right now, I think you're doing developer
level debugging.  ;-)

> > Thanks.  There's now a patch in my queue for this.  OK to add this tag
> > to the commit?
> > 
> > Reported-by: Ulrich Sibiller <ulrich.sibiller at atos.net>  
> 
> Yes ;-)

OK.

> >>> past?  Please feel free to open a bug to remind me to look at this.  
> 
> Done: https://bugzilla.samba.org/show_bug.cgi?id=15320

Thanks.

> I am not really sure if that is necessary. The code in 10.interface
> that kills the connections uses this from /etc/ctdb/functions:

> get_tcp_connections_for_ip ()
> {
>     _ip="$1"
> 
>     ss -tn state established "src [$_ip]" | awk 'NR > 1 {print $3, $4}'
> }
> 
> which ignores the port and thus matches all connections for the ip anyway. On the other hand there's
> 
>        update_tickles 2049

> in /etc/ctdb/events/legacy/60.nfs without a corresponding tickle
> handling for lockd connections. I am thinking about adding an
> update_tickles 599 for lockd connections (what's the best way to
> determine that port?). Any objections?

We are in violent agreement!  :-)

The main problem is the code in 10.interface releaseip is only run when
the "releasing" node does not crash hard.  So, we need another method.

The port number is tricky because we need to support both kernel NFS
and NFS Ganesha.  We could add it to the call-out... but I'm not sure
it would be easy.  Probably better via a configuration variable.

I just rebased this old 2015 branch for the first time in ~6 years:

  https://git.samba.org/?p=martins/samba.git;a=shortlog;h=refs/heads/ctdb-connections

I think it would help.

I can't remember exactly why I abandoned it.  I think it might have
been because I was going to work on the dedicated connection tracking
code, but that seems to have started a couple of years later.  Or
perhaps Amitay pointed out a valid flaw that I no longer remember.  I
will have to take a more careful look.

> One further observation I had: For tests I had temporarily switched
> from IPAllocAlgorithm=2 to 1. While it worked it also resulted in a
> much worse distribution of the IPs. Some nodes had two IPs in the
> same network while others had none in that network. In that setup I
> found identical tickles being sent up to three times on TAKEIP. Is
> this on purpose? See attached logfile for details.

Yep, that's why the algorithm known as LCP² (or LCP2 or #2) exists.
While it doesn't know about different networks, it contains a very
clever heuristic (thought of by Ronnie Sahlberg) to try to spread IP
addresses from different networks.  With the speed optimisations it has
received over the years, it doesn't seem sane to use anything else.
Perhaps some time it can be improved to encourage certain addresses to
be more likely on certain nodes - not sure.

The 3 identical tickle ACKs are standard and should not differ between
algorithms.

I think we're getting somewhere...  :-)

peace & happiness,
martin



More information about the samba mailing list