CTDB 2.2 problem

[Martin Schwenke]

On Tue, 04 Jun 2013 13:12:25 +0200, Stijn De Weirdt
<stijn.deweirdt at ugent.be> wrote:

> i just found the issue. i add some additional checks to verify other 
> dependencies that caused extra delay. apparently 2.2 is more 
> strict/sensitive.

Instead of waiting until ctdbd responds to the first ping, we now wait
until it has got through the initialisation phases that are most likely
to fail (i.e. "init" and "setup").  That was in response to the
observation that the initscript would succeed but ctdbd would
sometimes abort within a second or 2 due to a failure in the "setup"
event.  It seems better to do this extra wait so the initscript can
notice these very early failures and report them.

> if you want to reproduce the issue, add 'sleep 1' in the sysconfig/ctdb. 
> (the produced logs are very very short). i'm not considering this an issue.

Well, while implementing the above, we went from waiting up to 10
seconds before ctdbd would respond to a ping (i.e. listening on the
socket, event loop started) to waiting that same 10 seconds for CTDB to
do a bit more (i.e. process 2 events). The problems you've seen suggest
that we should seriously consider revisiting the length of the timeout.

That said, if you add a non-trivial delay to sysconfig/ctdb then, since
it is loaded/repeated for each eventscript, that delay is multiplied by
the number of active eventscripts.

For example, there are 14 eventscripts enabled by default.  If you add
a 0.25 second delay (due to extra processing) in sysconfig/ctdb then
that turns into a 3.5 second delay for each event.  When you consider
both the "init" and "setup" events, you're up to 7 seconds without
really trying.

If there's something you want to check then it might be worth adding an
extra eventscript and doing the check in the "init" event.  Then you'll
only do it once.

> apologies for bothering you with this

No problem.  It is always worth trying to make things better...  :-)

peace & happiness,
martin