ctdb-2.3-3.1 on SLES11 SP2: Timed out waiting for initialisation

[Martin Schwenke]

Hi Rainer,

[+David Disseldorp]

On Thu, 25 Jul 2013 15:10:52 +0200, Rainer Krienke
<krienke at uni-koblenz.de> wrote:

> in between my test-cluste ris up and running. Works fine again with ctdb
> 1.2 but ctdb 2.3 does not seem to be able to start up.
> 
> The problem is that in the startup loop in ctdb_wrapper::start() none of
> the ctdb runstate commands is able to connect to ctdb allthough ctdbd is
> running. First of all I set CTDB_STARTUP_TIMEOUT=30 to have more time to
> look for problems.
> 
> The error message when I run ctdb_wrapper manually is:
> 
> 2013/07/25 14:29:45.361089 [25872]: client/ctdb_client.c:275 Failed to
> connect client socket to daemon. Errno:No such file or directory(2)

> However ctdbd is running at this time:
> 
> Finally after running an strace on ctdb status I saw, that ctdb is
> accessing /tmp/ctdb.socket allthough in /etc/sysconfig/ctdb CTDB_SOCKET
> is set to "/var/lib/ctdb/ctdb.socket".  Looks like ctdb does not
> evalueate this option in /etc/sysconfig/ctdb	

OK, the main problem is that ctdbd_wrapper does not export
CTDB_SOCKET.  Sorry about that! This will be fixed in CTDB 2.4.

In the interim, you can just add:

  export CTDB_SOCKET

to /etc/sysconfig/ctdb.

Then ctdbd will start and ctdbd_wrapper will do the right thing and
keep it running.

To run ctdb tool commands you will need to have CTDB_SOCKET set in your
environment.  You could set it ~/.profile, /etc/profile.d/ctdb.sh or
similar.  That's a problem we'll try to solve in the longer term in
CTDB.

> After setting /tmp/ctdb.socket to be used as socket file on all nodes I
> was able to start ctdb on all nodes, but the would stay in state
> UNHEALTHY, because winbindd (samba 3.6.16) crashes, again because of
> /var/lib/ctdb/ctdb.socket setting:
> 
> 2013/07/25 14:50:26,  1] lib/ctdbd_conn.c:195(ctdbd_connect)
>   connect(/var/lib/ctdb/ctdb.socket) failed: Connection refused

Right.  You can set:

  ctdbd socket = <whatever>

in /etc/samba/smb.conf to point smbd/winbind/... to the correct socket
location.

> Looks for me as if packaging of ctdb and smb are not yet compatible as
> far as paths are concerned. I downloaded the precompiled packages from
> 
> http://download.opensuse.org/repositories/network:/samba:/STABLE/SLE_11

That's probably true.  SUSE should probably patch ctdb.spec.in to add
the configure option --with-socketpath=FILE if they're using a
non-default location.

David, is this something you're able to look at?

We are planning to change the default socket location in CTDB but we
(obviously!) need to think about it carefully.  :-)

Rainer, sorry that the upgrade has been a bit rocky.  Exporting
CTDB_SOCKET as described above should get things working...

peace & happiness,
martin