ctdb intermittent startup failure "Timed out waiting for lttng-sessiond"

[Martin Schwenke]

Hi Steve,

On Mon, 1 Aug 2016 12:05:41 -0500, Steve French
<smfrench at gmail.com> wrote:

> I noticed that ctdb (Samba 4.4-stable) frequently fails to start
> automatically on some of our systems.    RHEL 7.2 lttng problem?  "Timed
> out waiting for lttng-sessiond"   Any ideas what this means or how to
> workaround it? Anyone recognize this?  I see similar complaints from others
> (https://lists.lttng.org/pipermail/lttng-dev/2014-April/022763.html)
> 
> 2016/08/01 09:55:35.916683 [15896]: CTDB starting on node
> 
> 2016/08/01 09:55:35.916730 [15896]: Recovery lock file set to "". Disabling
> recovery lock checking
> 
> 2016/08/01 09:55:35.921198 [16507]: Starting CTDBD (Version 4.4.5-UNKNOWN)
> as PID: 16507
> 
> 2016/08/01 09:55:35.921266 [16507]: Created PID file /run/ctdb/ctdbd.pid
> 
> 2016/08/01 09:55:35.921284 [16507]: Set real-time scheduler priority
> 
> 2016/08/01 09:55:35.921398 [16507]: Set runstate to INIT (1)
> 
> 2016/08/01 09:55:35.921735 [16507]: Set event helper to
> "/usr/libexec/ctdb/ctdb_event_helper"
> 
> 2016/08/01 09:55:38.933533 [16507]: libust[16518/16518]: Error: Timed out
> waiting for lttng-sessiond (in lttng_ust_init() at lttng-ust-comm.c:1532)
> 
> 2016/08/01 16:55:02.103492 [16507]: No event for 25162 seconds!
> 
> 2016/08/01 16:55:02.103538 [16507]: libust[16768/16768]: Error: Timed out
> waiting for lttng-sessiond (in lttng_ust_init() at lttng-ust-comm.c:1532)
> 
> 2016/08/01 16:55:02.103586 [16507]: Event script '01.reclock init ' timed
> out after 25162.8s, pid: 16768
> 
> 2016/08/01 16:55:02.103807 [16507]: ../ctdb/server/eventscript.c:912
> eventscript for 'init' timedout. Immediately banning ourself for 300 seconds
> 
> 2016/08/01 16:55:02.103826 [16507]: ctdb exiting with error: Failed to run
> init event
> 
> 2016/08/01 16:55:02.103834 [16507]:
> 
> 2016/08/01 16:55:02.103842 [16507]: CTDB daemon shutting down
> 
> 2016/08/01 16:55:02.111101 [16507]: Removed PID file /run/ctdb/ctdbd.pid

I'm not sure about the lttng part.  I don't think we have a way of
using lttng with CTDB.

However, to me it looks like time jumped forward by a few hours and
CTDB reacted badly.  CTDB needs time to be stable...

peace & happiness,
martin