CTDB Assigned IP not on an interface

Thu Sep 26 19:04:31 UTC 2019

Additional info - when running an strace on the process:

epoll_wait(11, [{EPOLLIN, {u32=2179214400, u64=94542999333952}}], 1, 114) = 1
ioctl(24, FIONREAD, [96])               = 0
read(24, "`\0\0\0BDTC\1\0\0\0\352k\35w\7\0\0\0\0\0\0\0\2\0\0\0W\24\1\0"..., 96) = 96
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
write(5, "2019/09/26 14:59:36.055468 ctdbd"..., 130) = 130
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 25
bind(25, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("10.85.136.108")}, 16) = -1 EADDRNOTAVAIL (Cannot assign requested address)
close(25)                               = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
write(5, "2019/09/26 14:59:36.055731 ctdbd"..., 118) = 118
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
write(5, "2019/09/26 14:59:36.055848 ctdbd"..., 153) = 153
write(21, "0\0\0\0BDTC\1\0\0\0\352k\35w\10\0\0\0\2\0\0\0\0\0\0\0W\24\1\0"..., 48) = 48

On 9/26/19, 1:00 PM, "samba-technical on behalf of Max DiOrio via samba-technical" <samba-technical-bounces at lists.samba.org on behalf of samba-technical at lists.samba.org> wrote:

    NOTE: This email originated from outside of the organization.

    Hi!

    CTDB has been working great for us the last few weeks.  Today, I did a yum update – and now things aren’t working.  The public IP assigned is not actually being added to the interface, but CTDB claims the nodes are healthy.

    All 3 nodes are showing the same events, just with their respective public IP’s.  This repeats over and over when set to debug level.

    2019/09/26 12:56:06.225362 ctdb-recoverd[11615]: Trigger takeoverrun
    2019/09/26 12:56:06.225419 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 0 of type 5 length 64 from node 0 to 1
    2019/09/26 12:56:06.409819 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182796 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.410526 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182803 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.410904 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182805 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.411087 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182808 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.411257 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182811 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.411420 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182814 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.411755 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 0 of type 5 length 72 from node 1 to 0
    2019/09/26 12:56:06.411805 ctdb-recoverd[11615]: Disabling takeover runs for 60 seconds
    2019/09/26 12:56:06.413895 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 20510 of type 7 length 64 from node 2 to 0
    2019/09/26 12:56:06.416270 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182820 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.416484 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182823 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.416873 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182828 of type 7 length 96 from node 1 to 0
    2019/09/26 12:56:06.416912 ctdbd[11606]: Redundant release of IP 10.85.136.109/24 on interface __none__ (ip not held)
    2019/09/26 12:56:06.416925 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:291 public address '10.85.136.109' now unassigned (old iface '__none__' refs[0])
    2019/09/26 12:56:06.416949 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182830 of type 7 length 96 from node 1 to 0
    2019/09/26 12:56:06.416970 ctdbd[11606]: Redundant release of IP 10.85.136.108/24 on interface __none__ (ip not held)
    2019/09/26 12:56:06.416981 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:291 public address '10.85.136.108' now unassigned (old iface '__none__' refs[0])
    2019/09/26 12:56:06.417232 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182832 of type 7 length 96 from node 1 to 0
    2019/09/26 12:56:06.417260 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:260 public address '10.85.136.110' still assigned to iface 'ens192'
    2019/09/26 12:56:06.417272 ctdbd[11606]: Takeover of IP 10.85.136.110/24 on interface ens192
    2019/09/26 12:56:06.417290 ctdbd[11606]: ../ctdb/server/eventscript.c:659 Running event takeip with arguments ens192 10.85.136.110 24
    2019/09/26 12:56:06.418494 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:494 sending TAKE_IP for '10.85.136.110'
    2019/09/26 12:56:06.418543 ctdbd[11606]: ctdb_sys_send_arp: Created SOCKET FD:18 for sending arp
    2019/09/26 12:56:06.427442 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 182835 of type 7 length 64 from node 1 to 0
    2019/09/26 12:56:06.427487 ctdbd[11606]: ../ctdb/server/ctdb_takeover.c:2282 Running "ipreallocated" event
    2019/09/26 12:56:06.427505 ctdbd[11606]: ../ctdb/server/eventscript.c:659 Running event ipreallocated with arguments
    2019/09/26 12:56:06.431728 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 0 of type 5 length 72 from node 1 to 0
    2019/09/26 12:56:06.431786 ctdb-recoverd[11615]: Reenabling takeover runs
    2019/09/26 12:56:06.531816 ctdbd[11606]: ctdb_sys_send_arp: Created SOCKET FD:18 for sending arp
    2019/09/26 12:56:06.634656 ctdbd[11606]: ctdb_sys_send_arp: Created SOCKET FD:18 for sending arp
    2019/09/26 12:56:07.223743 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1330 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.223825 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1343 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.223851 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1343 of type 8 length 48 from node 0 to 0
    2019/09/26 12:56:07.223917 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1331 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.223944 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1344 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224084 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1344 of type 8 length 48 from node 0 to 0
    2019/09/26 12:56:07.224127 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1332 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.224145 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1345 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224159 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1345 of type 8 length 292 from node 0 to 0
    2019/09/26 12:56:07.224199 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1333 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.224217 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1346 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224229 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1346 of type 8 length 48 from node 0 to 0
    2019/09/26 12:56:07.224265 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1334 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224281 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1347 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224296 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1347 of type 8 length 156 from node 0 to 0
    2019/09/26 12:56:07.224336 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1335 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.224352 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1348 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224364 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1348 of type 8 length 48 from node 0 to 0
    2019/09/26 12:56:07.224404 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1336 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224422 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1349 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.224434 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1349 of type 8 length 48 from node 0 to 0
    2019/09/26 12:56:07.224466 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1337 of type 7 length 64 from node 0 to 1
    2019/09/26 12:56:07.224521 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1338 of type 7 length 64 from node 0 to 2
    2019/09/26 12:56:07.224902 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1350 of type 8 length 48 from node 1 to 0
    2019/09/26 12:56:07.224939 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1351 of type 8 length 48 from node 2 to 0
    2019/09/26 12:56:07.224980 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1339 of type 7 length 64 from node 0 to 1
    2019/09/26 12:56:07.225175 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1352 of type 8 length 160 from node 1 to 0
    2019/09/26 12:56:07.225220 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1340 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.225238 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1353 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.225252 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1353 of type 8 length 72 from node 0 to 0
    2019/09/26 12:56:07.225291 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1341 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.225308 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1354 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.225322 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1354 of type 8 length 144 from node 0 to 0
    2019/09/26 12:56:07.225360 ctdbd[11606]: ../ctdb/server/ctdb_daemon.c:911 client request 1342 of type 7 length 64 from node 0 to 4026531841
    2019/09/26 12:56:07.225377 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1355 of type 7 length 64 from node 0 to 0
    2019/09/26 12:56:07.225389 ctdbd[11606]: ../ctdb/server/ctdb_server.c:188 ctdb request 1355 of type 8 length 144 from node 0 to 0
    2019/09/26 12:56:07.225442 ctdb-recoverd[11615]: Assigned IP 10.85.136.110 not on an interface
    2019/09/26 12:56:07.225476 ctdb-recoverd[11615]: Trigger takeoverrun

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:85:33:88 brd ff:ff:ff:ff:ff:ff
        inet 10.85.136.101/24 brd 10.85.136.255 scope global ens192
           valid_lft forever preferred_lft forever
        inet6 fe80::250:56ff:fe85:3388/64 scope link
           valid_lft forever preferred_lft forever
    3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:85:c1:14 brd ff:ff:ff:ff:ff:ff
        inet 192.168.136.101/24 brd 192.168.136.255 scope global ens224
           valid_lft forever preferred_lft forever
        inet6 fe80::250:56ff:fe85:c114/64 scope link
           valid_lft forever preferred_lft forever

    Currently running ctdb.x86_64    4.9.1-6.el7. on Centos 7.7.1908

    Nothing else is being logged to messages or any other log I’ve looked at.  Selinux is disabled.   How do I fix this?  Thanks!

    Max DiOrio
    IT Architect
    [signature_620863466]
    201 Fuller Road, Suite 202
    Albany, NY 12203-3621
    Phone: 518-238-6516 | Mobile: 518-944-5289
    max.diorio at ieeeglobalspec.com<mailto:max.diorio at ieeeglobalspec.com>