ctdb, haproxy, and ip_nonlocal_bind

[Wyllys Ingersoll]

I have a cluster in which I want to use both haproxy AND ctdb on the same
nodes - haproxy to manage NFSv4 (ganesha) and CTDB to manage SMB.  The
reason for doing this is that I've read several warnings about NOT using
NFSv4 with CTDB.

haproxy + keepalived require that the net.ipv4.ip_nonlocal_bind flag be set
to 1 which breaks ctdb's ability to manage the virtual public IP addresses
(among other things).

If I do not configure any public_addresses and just let haproxy configure
the virtual public IP addresses, CTDB is still unable to startup on both of
the nodes in my test cluster.  It will start on one or the other, but they
never sync up and come to an "OK" state on both nodes.

I have the "node address" value set in the [cluster] section of ctdb.conf
on each node to be the private address of that node and both private
addresses are listed in the nodes configuration file and the nodes are
connecting to each other privately, but they don't stay connected and the
2nd ctdb node never fully initializes and starts up.  At some point it just
begins flooding the logs with messages like this "node 192.168.113.14:4379
is already marked disconnected: 0 connected" and pegging the CPU at almost
100% until the disk with the logging completely fills up (which sounds like
a bug, btw).

Does anyone know of any way to make this sort of configuration work ?

Currently running Samba 4.10.10, haproxy 1.6.3, and Linux Kernel 4.19.34 on
Ubuntu 16.04.4

Any help would be much appreciated.

-Wyllys Ingersoll
 Keeper Technology, LLC