CTDB with net.ipv4.ip_nonlocal_bind=1 (was Re: Setting up CTDB on OCFS2 and VMs ...)

[Min Wai Chan]

Dear Martin,

Thank you for the reply

I'm using haproxy and keepalive for a http proxy and also mariadb cluster.
When I was doing that I don't know that it would cause problem to the ctdb
cluster...

I think I've to move haproxy and keepalive away...

Or we have any other suggestion?

Regards,
Min Wai



On Tue, Dec 30, 2014 at 12:58 PM, Martin Schwenke <martin at meltin.net> wrote:

> On Tue, 30 Dec 2014 12:17:25 +0800, Min Wai Chan <dcmwai at gmail.com>
> wrote:
>
> > I'm not sure if I should start a new thread on this...
>
> Definitely!  ;-)
>
> I have at least changed the subject line.
>
> > But I found that when using CTDB...
> >
> > You cannot have the following (
> >
> > */etc/sysctl.conf*
> >
> > net.ipv4.ip_nonlocal_bind=1
> >
> >
> > If such configure are available.
> >
> > Both the CTDB node will only bind node0. (Even if it don't have the IP)
>
> Yes, I guess that's why the documentation for this option says "may
> break some applications".  This will cause 2 different types of
> failures:
>
> * The one you have noticed where CTDB on both nodes will bind to the
>   same private IP address.  ctdbd(7) has the following DEBUG option:
>
>        --listen=IPADDR
>            This specifies which IP address that ctdbd will bind to.
>
>            By default ctdbd will bind to the first address it finds in
>            the /etc/ctdb/nodes file that is also present on the local
>            system.
>
>            This option is only required when you want to run multiple
>            ctdbd daemons/nodes on the same physical host in which case
>            there would be multiple entries in /etc/ctdb/nodes that
>            would match a local interface.
>
>   We use this when running tests.  There is no corresponding
>   configuration option (which would be documented in ctdbd.conf(5)).
>   Perhaps you've just given us a reason to consider adding one?  ;-)
>
>   However, CTDB's "ctdb xpnn" CLI command depends on only being able to
>   bind to a one address in the nodes file so it can determine the node
>   number.  This would be harder to "fix" to accommodate
>   ip_nonlocal_bind.  The way to do that would be to finally have the
>   ctdb CLI tool also load configuration so that any "listen" option
>   could be use in the xpnn code.  That looks like a non-trivial change.
>
>   Another possibility would be to avoid using "ctdb xpnn" altogether.
>   It is a hack to avoid "ctdb pnn" under heavy load when there might
>   be issues talking to the daemon.  We use "ctdb xpnn" in scripts 5
>   times.  One of those is in the 00.ctdb "startup" event.  We could
>   instead drop the PNN value into a known file during the "setup" event
>   and have the scripts just read the PNN value from that file.  I'm not
>   sure if anything external actually uses "ctdb xpnn".
>
> * Incorrect results when checking to see if a public IP address is
>   present on an interface.  In a number of contexts CTDB uses bind(2) as
>   a cheap way of checking if a public IP address is present on an
>   interface.
>
>   However, if you're using haproxy then you're probably not using
>   CTDB's public IP addresses.
>
> Can you please let us know whether you intend to use CTDB's public IP
> addresses (no, I assume) and why you need to set ip_nonlocal_bind
> (haproxy needs it, I assume)?
>
> Meanwhile, I'll think about the changes needed to solve the 1st problem,
> since it would allow CTDB to be used without public IP addresses when
> ip_nonlocal_bind is set.
>
> peace & happiness,
> martin
>