[PATCH] ctdb-recoverd: Gently abort recovery when election is underway

[Martin Schwenke]

Sometimes the recovery daemon fails to get the recovery lock on one
node so that node is banned.  This seems to always happen during an
election.  The recovery is triggered because other nodes are found to
have recovery mode enabled.  They have recovery mode enabled because
an election has been forced.

The recovery daemon's main_loop() only does an initial check for an
election.  After that, a node can force an election and, in the
process, set itself to be the current winner.  In this situation,
verify_recmode() will always return MONITOR_RECOVERY_NEEDED so
do_recovery() is called.  If the previous recovery master hasn't
admitted defeat and released the recovery lock, then do_recovery()
will rightly fail.  However, it would be better if it failed a little
more gracefully, since this case is not that unusual.

Instead of trying to take the recovery lock, return early with an
error if there is an election in progress.  Note that the race is
still there but it is now much narrower.

There are probably more subtle ways of avoiding this issue, including
something like this in main_loop():

-   if (pnn != rec->recmaster) {
+   if (pnn != rec->recmaster || rec->election_timeout) {
            return;
    }
    
However, this check is done earlier so it leaves the race window open
a little wider.


Please review and push if OK.

This patch is in my ctdb branch:

  http://git.samba.org/?p=martins/samba.git;a=shortlog;h=refs/heads/ctdb

peace & happiness,
martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-ctdb-recoverd-Gently-abort-recovery-when-election-is.patch
Type: text/x-patch
Size: 2291 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20140710/7e0057eb/attachment.bin>