Split brain with Gluster (was: [PATCH] CTDB event script error (and unusual) path fixes (bug 12180))

[Martin Schwenke]

On Thu, 1 Sep 2016 05:53:19 +0000, 趙 良基 <hijk123452001 at hotmail.com>
wrote:

> I use gluster replica 3 volume as CTDB recovery lockfile path,but when split brain happend (split to 2 nodes and 1 node) the 1 node also can open the lockfile so that 1 node can takeover all IPs.
> 
> 2016/08/31 23:49:22.296820 [set_recmode: 5604]: ctdb_recovery_lock:fd=5 - (No such file or directory)
> 2016/08/31 23:52:19.963449 [recoverd:26006]: ctdb_recovery_lock:fd=37 - (Cannot assign requested address)
> 
> Volume Name: IFTSwiftShareVol
> Type: Replicate
> Volume ID: 2501be2f-90be-477f-947a-2832198c982c
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: CS174119:/export/IFT_lvol_LujgjV35tD/fs
> Brick2: CS7dc2b:/export/IFT_lvol_RhVVs85UEk/fs
> Brick3: CS81876:/export/IFT_lvol_RgEJJyCnZs/fs
> Options Reconfigured:
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> nfs.disable: on
> performance.readdir-ahead: on
> cluster.server-quorum-ratio: 51%

It sounds like lock coherence is not working on your cluster filesystem,
so it can not support the CTDB recovery lock. This is probably a
Gluster-specific issue.  Please see:

  https://wiki.samba.org/index.php/Ping_pong

for information about how to test your filesystem's lock coherence.

You also have not mentioned which version of CTDB you are trying to use.

By the way, you should not reply to existing thread to try to get help
with a new issue.  You should start a new thread...  :-)

peace & happiness,
martin