CTDB Assigned IP not on an interface

Thu Sep 26 16:59:26 UTC 2019

Hi!

CTDB has been working great for us the last few weeks.  Today, I did a yum update – and now things aren’t working.  The public IP assigned is not actually being added to the interface, but CTDB claims the nodes are healthy.

All 3 nodes are showing the same events, just with their respective public IP’s.  This repeats over and over when set to debug level.

2019/09/26 12:56:06.225362 ctdb-recoverd[11615]: Trigger takeoverrun
2019/09/26 12:56:06.225419 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 0 of type 5 length 64 from node 0 to 1
2019/09/26 12:56:06.409819 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182796 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.410526 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182803 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.410904 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182805 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.411087 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182808 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.411257 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182811 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.411420 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182814 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.411755 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 0 of type 5 length 72 from node 1 to 0
2019/09/26 12:56:06.411805 ctdb-recoverd[11615]: Disabling takeover runs for 60 seconds
2019/09/26 12:56:06.413895 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 20510 of type 7 length 64 from node 2 to 0
2019/09/26 12:56:06.416270 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182820 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.416484 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182823 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.416873 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182828 of type 7 length 96 from node 1 to 0
2019/09/26 12:56:06.416912 ctdbd[11606]: Redundant release of IP 10.85.136.109/24 on interface __none__ (ip not held)
2019/09/26 12:56:06.416925 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:291 public address '10.85.136.109' now unassigned (old iface '__none__' refs[0])
2019/09/26 12:56:06.416949 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182830 of type 7 length 96 from node 1 to 0
2019/09/26 12:56:06.416970 ctdbd[11606]: Redundant release of IP 10.85.136.108/24 on interface __none__ (ip not held)
2019/09/26 12:56:06.416981 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:291 public address '10.85.136.108' now unassigned (old iface '__none__' refs[0])
2019/09/26 12:56:06.417232 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182832 of type 7 length 96 from node 1 to 0
2019/09/26 12:56:06.417260 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:260 public address '10.85.136.110' still assigned to iface 'ens192'
2019/09/26 12:56:06.417272 ctdbd[11606]: Takeover of IP 10.85.136.110/24 on interface ens192
2019/09/26 12:56:06.417290 ctdbd[11606]: ../ctdb/server/eventscript.c:659 Running event takeip with arguments ens192 10.85.136.110 24
2019/09/26 12:56:06.418494 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:494 sending TAKE_IP for '10.85.136.110'
2019/09/26 12:56:06.418543 ctdbd[11606]: ctdb_sys_send_arp: Created SOCKET FD:18 for sending arp
2019/09/26 12:56:06.427442 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182835 of type 7 length 64 from node 1 to 0
2019/09/26 12:56:06.427487 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:2282 Running "ipreallocated" event
2019/09/26 12:56:06.427505 ctdbd[11606]: ../ctdb/server/eventscript.c:659 Running event ipreallocated with arguments
2019/09/26 12:56:06.431728 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 0 of type 5 length 72 from node 1 to 0
2019/09/26 12:56:06.431786 ctdb-recoverd[11615]: Reenabling takeover runs
2019/09/26 12:56:06.531816 ctdbd[11606]: ctdb_sys_send_arp: Created SOCKET FD:18 for sending arp
2019/09/26 12:56:06.634656 ctdbd[11606]: ctdb_sys_send_arp: Created SOCKET FD:18 for sending arp
2019/09/26 12:56:07.223743 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1330 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.223825 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1343 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.223851 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1343 of type 8 length 48 from node 0 to 0
2019/09/26 12:56:07.223917 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1331 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.223944 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1344 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224084 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1344 of type 8 length 48 from node 0 to 0
2019/09/26 12:56:07.224127 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1332 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.224145 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1345 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224159 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1345 of type 8 length 292 from node 0 to 0
2019/09/26 12:56:07.224199 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1333 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.224217 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1346 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224229 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1346 of type 8 length 48 from node 0 to 0
2019/09/26 12:56:07.224265 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1334 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224281 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1347 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224296 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1347 of type 8 length 156 from node 0 to 0
2019/09/26 12:56:07.224336 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1335 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.224352 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1348 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224364 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1348 of type 8 length 48 from node 0 to 0
2019/09/26 12:56:07.224404 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1336 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224422 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1349 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.224434 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1349 of type 8 length 48 from node 0 to 0
2019/09/26 12:56:07.224466 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1337 of type 7 length 64 from node 0 to 1
2019/09/26 12:56:07.224521 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1338 of type 7 length 64 from node 0 to 2
2019/09/26 12:56:07.224902 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1350 of type 8 length 48 from node 1 to 0
2019/09/26 12:56:07.224939 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1351 of type 8 length 48 from node 2 to 0
2019/09/26 12:56:07.224980 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1339 of type 7 length 64 from node 0 to 1
2019/09/26 12:56:07.225175 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1352 of type 8 length 160 from node 1 to 0
2019/09/26 12:56:07.225220 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1340 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.225238 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1353 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.225252 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1353 of type 8 length 72 from node 0 to 0
2019/09/26 12:56:07.225291 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1341 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.225308 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1354 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.225322 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1354 of type 8 length 144 from node 0 to 0
2019/09/26 12:56:07.225360 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1342 of type 7 length 64 from node 0 to 4026531841
2019/09/26 12:56:07.225377 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1355 of type 7 length 64 from node 0 to 0
2019/09/26 12:56:07.225389 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1355 of type 8 length 144 from node 0 to 0
2019/09/26 12:56:07.225442 ctdb-recoverd[11615]: Assigned IP 10.85.136.110 not on an interface
2019/09/26 12:56:07.225476 ctdb-recoverd[11615]: Trigger takeoverrun

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:33:88 brd ff:ff:ff:ff:ff:ff
    inet 10.85.136.101/24 brd 10.85.136.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe85:3388/64 scope link
       valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:c1:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.136.101/24 brd 192.168.136.255 scope global ens224
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe85:c114/64 scope link
       valid_lft forever preferred_lft forever

Currently running ctdb.x86_64    4.9.1-6.el7. on Centos 7.7.1908

Nothing else is being logged to messages or any other log I’ve looked at.  Selinux is disabled.   How do I fix this?  Thanks!

Max DiOrio
IT Architect
[signature_620863466]
201 Fuller Road, Suite 202
Albany, NY 12203-3621
Phone: 518-238-6516 | Mobile: 518-944-5289
max.diorio at ieeeglobalspec.com<mailto:max.diorio at ieeeglobalspec.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 44823 bytes
Desc: image001.png
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190926/a742e772/image001.png>