connect(/var/lib/ctdb/ctdb.socket) failed: Connection refused

[steve]

On Mon, 2014-08-11 at 08:55 +1000, Martin Schwenke wrote:
> On Sun, 10 Aug 2014 16:49:49 +0200, steve <steve at steve-ss.com> wrote:
> 
> > 2014/08/10 16:31:02.920250 [ 3913]: Node became HEALTHY. Ask recovery
> > master 0 to perform ip reallocation
> > 
> >  ps aux|grep ctdbd
> > root      3913  1.4  0.8   4824  4224 ?        SLs  16:30   0:06 ctdbd
> > --dbdir=/usr/local/var/lib/ctdb
> > --dbdir-persistent=/usr/local/var/lib/ctdb/persistent
> > --event-script-dir=/usr/local/etc/ctdb/events.d
> > --public-addresses=/usr/local/etc/ctdb/public_addresses
> > --socket=/usr/local/var/lib/ctdb/ctdbd.socket --logfile=/var/log/syslog
> > root      4050  0.7  0.2   4428  1064 ?        S    16:30   0:03 ctdbd
> > --dbdir=/usr/local/var/lib/ctdb
> > --dbdir-persistent=/usr/local/var/lib/ctdb/persistent
> > --event-script-dir=/usr/local/etc/ctdb/events.d
> > --public-addresses=/usr/local/etc/ctdb/public_addresses
> > --socket=/usr/local/var/lib/ctdb/ctdbd.socket --logfile=/var/log/syslog
> > root      6950  0.0  0.1   7124   836 pts/3    S+   16:37   0:00 grep
> > --color=auto ctdbd
> > 
> > which ctdb
> > /usr/local/bin/ctdb
> > 
> > ctdb errors out:
> >  ctdb status
> > 2014/08/10 16:38:59.619340 [ 7383]: client/ctdb_client.c:267 Failed to
> > connect client socket to daemon. Errno:Connection refused(111)
> > common/cmdline.c:156 Failed to connect to daemon
> > 2014/08/10 16:38:59.626991 [ 7383]: Failed to init ctdb
> > 
> > Is the socket there?
> >  ls -l /usr/local/var/lib/ctdb
> > total 8
> > srwx------ 1 root root    0 ago 10 16:41 ctdbd.socket
> > drwx------ 2 root root 4096 ago 10 14:14 persistent
> > drwx------ 2 root root 4096 ago 10 16:41 state
> 
> > [...]
> 
> > Question: why is ctdb status failing?
> 
> What does the following tell you?
> 
>   strings /usr/local/bin/ctdb | grep 'ctdbd\.socket'
/usr/local/var/run/ctdb/ctdbd.socket
> 
> The location of the socket is set consistently between ctdbd and ctdb at
> compile time. You can't tell ctdbd a different location on the
> command-line and expect ctdb to be able to connect.
So why have the option to change it. Anyway, still no luck:
ctdb now works:
sudo ctdb status
Number of nodes:2
pnn:0 192.168.0.10     OK (THIS NODE)
pnn:1 192.168.0.11     DISCONNECTED|UNHEALTHY|INACTIVE
Generation:1673610329
Size:1
hash:0 lmaster:0
Recovery mode:NORMAL (0)
Recovery master:0

The ips have taken over correctly (the machine IP is 192.168.1.82 and
the cluster ips, 192.168.1.80 and 192.168.1.81)
 ping smbcluster
PING smbcluster.altea.site (192.168.1.80) 56(84) bytes of data.
64 bytes from smbcluster (192.168.1.80): icmp_seq=1 ttl=64 time=0.391 ms

We leave the domain, kill ctdbd, smbd and winbind then restart tyem all
and rejoin the domain:
sudo net ads join -UAdministrator
Enter Administrator's password:
Using short domain name -- ALTEA
Joined 'SMBCLUSTER' to dns domain 'altea.site'
Not doing automatic DNS update in a clustered setup.

However, the ctdb assigned address is not used:
Kerberos: AS-REQ Administrator at ALTEA.SITE from ipv4:192.168.1.82:47211
for krbtgt/ALTEA.SITE at ALTEA.SITE

The only db produced is:
 ls -l /usr/local/var/lib/ctdb/persistent
total 1280
-rw------- 1 root root 1310720 ago 11 10:25 ctdb.tdb.0

winbind works, but of course only on this one node because the IP is not
transferred.

Why isn't the domain join creating the persistent databases so that we
can swap to the other node? Why are the ctdb assigned ips not being
used?

Cheers,
Steve