CTDB with net.ipv4.ip_nonlocal_bind=1 (was Re: Setting up CTDB on OCFS2 and VMs ...)

[Martin Schwenke]

On Tue, 30 Dec 2014 12:17:25 +0800, Min Wai Chan <dcmwai at gmail.com>
wrote:

> I'm not sure if I should start a new thread on this...

Definitely!  ;-)

I have at least changed the subject line.

> But I found that when using CTDB...
> 
> You cannot have the following (
> 
> */etc/sysctl.conf*
> 
> net.ipv4.ip_nonlocal_bind=1
> 
> 
> If such configure are available.
> 
> Both the CTDB node will only bind node0. (Even if it don't have the IP)

Yes, I guess that's why the documentation for this option says "may
break some applications".  This will cause 2 different types of
failures:

* The one you have noticed where CTDB on both nodes will bind to the
  same private IP address.  ctdbd(7) has the following DEBUG option:

       --listen=IPADDR
           This specifies which IP address that ctdbd will bind to.

           By default ctdbd will bind to the first address it finds in
           the /etc/ctdb/nodes file that is also present on the local
           system.

           This option is only required when you want to run multiple
           ctdbd daemons/nodes on the same physical host in which case
           there would be multiple entries in /etc/ctdb/nodes that
           would match a local interface.

  We use this when running tests.  There is no corresponding
  configuration option (which would be documented in ctdbd.conf(5)).
  Perhaps you've just given us a reason to consider adding one?  ;-)

  However, CTDB's "ctdb xpnn" CLI command depends on only being able to
  bind to a one address in the nodes file so it can determine the node
  number.  This would be harder to "fix" to accommodate
  ip_nonlocal_bind.  The way to do that would be to finally have the
  ctdb CLI tool also load configuration so that any "listen" option
  could be use in the xpnn code.  That looks like a non-trivial change.

  Another possibility would be to avoid using "ctdb xpnn" altogether.
  It is a hack to avoid "ctdb pnn" under heavy load when there might
  be issues talking to the daemon.  We use "ctdb xpnn" in scripts 5
  times.  One of those is in the 00.ctdb "startup" event.  We could
  instead drop the PNN value into a known file during the "setup" event
  and have the scripts just read the PNN value from that file.  I'm not
  sure if anything external actually uses "ctdb xpnn".

* Incorrect results when checking to see if a public IP address is
  present on an interface.  In a number of contexts CTDB uses bind(2) as
  a cheap way of checking if a public IP address is present on an
  interface.

  However, if you're using haproxy then you're probably not using
  CTDB's public IP addresses.

Can you please let us know whether you intend to use CTDB's public IP
addresses (no, I assume) and why you need to set ip_nonlocal_bind
(haproxy needs it, I assume)?

Meanwhile, I'll think about the changes needed to solve the 1st problem,
since it would allow CTDB to be used without public IP addresses when
ip_nonlocal_bind is set.

peace & happiness,
martin