ctdb-2.3-3.1 on SLES11 SP2: Timed out waiting for initialisation

[Martin Schwenke]

Hi Rainer,

On Tue, 16 Jul 2013 08:49:30 +0200, Rainer Krienke
<krienke at uni-koblenz.de> wrote:

> I have a running ctdb cluster with 6 nodes based on a OCFS2 Cluster FS.
> Until now I am still running ctdb 1.2.52 which works without problems.
> 
> Yesterday I tried to upgrade ctdb to version 2.3 on one node. However
> when I start ctdb on this node using the same settings as for version
> 1.2  ctdb startes and everything looks fine buth then after exactly 10
> sec it says
> 
> Timed out waiting for initialisation - check logs - killing CTDB

I assume you're running Samba 3.  Can you please try setting the
Samba3AvoidDeadlocks tunable?

  CTDB_SET_Samba3AvoidDeadlocks=1

Sorry, we should have documented this better... but we released CTDB
2.0 without a NEWS file.  :-(

We've improved since then...

> However I cannot find anything really helpful in there. So I looked what
> timeout value is set to 10 sec on one of the other nodes:
> 
> sambanode1:~ # ctdb listvars|grep -i timeout
> [...]

The timeout comes from the new ctdbd_wrapper (although the timeout isn't
new - in CTDB 2.2 it was in the initscript).

However, right now I'm a little bit confused because I
think ctdbd_wrapper should succeed instead of timing out after ctdbd
moves to the FIRST_RECOVERY runstate:

> log.ctdb
> ---------------
> 2013/07/16 08:29:00.206512 [19648]: CTDB starting on node
> 2013/07/16 08:29:00.211208 [19649]: Starting CTDBD (Version 2.3) as PID:
[...]
> 2013/07/16 08:29:02.035237 [19649]: Set runstate to FIRST_RECOVERY (3)
[...]

I'll look at this some more tomorrow...  but for now please try setting
Samba3AvoidDeadlocks if you are running Samba 3.

peace & happiness,
martin