Setting up CTDB on OCFS2 and VMs ...

Richard Sharpe realrichardsharpe at gmail.com
Tue Dec 30 16:59:08 MST 2014


On Sun, Dec 28, 2014 at 8:10 PM, Günter Kukkukk <linux at kukkukk.com> wrote:
> Am 07.12.2014 um 14:27 schrieb Richard Sharpe:
>> On Sat, Dec 6, 2014 at 4:21 PM, Michael Adam <obnox at samba.org> wrote:
>>> On 2014-12-07 at 00:48 +0100, Michael Adam wrote:
>>>>
>>>> So the important bit is that in your case ctdb
>>>> is running unprotected from split brain.
>>>> The only reference to split brain is a notification
>>>> of user steve in case drbd detects a split brain.
>>>> If I get it right (there are no details about this
>>>> in the blog post), this means that until user steve
>>>> reacts to that notification the ctdb/samba cluster
>>>> runs happily in the split brain situation and
>>>> corrupts the users' data.
>>>
>>> Ok, maybe it is not quite as bad. The config snippet
>>>
>>> net {
>>>   allow-two-primaries;
>>>   after-sb-0pri discard-zero-changes;
>>>   after-sb-1pri discard-secondary;
>>>   after-sb-2pri disconnect;
>>> }
>>>
>>> Which is explained to some extent in
>>>
>>> http://www.drbd.org/users-guide/s-configure-split-brain-behavior.html
>>>
>>> seems to indicate that in case of split brain
>>> certain measures are potentially taken.
>>>
>>> Also read the explanations about DRBD split brain here:
>>>
>>> http://www.drbd.org/users-guide/s-split-brain-notification-and-recovery.html
>>>
>>> This states that DRDB split brain is different from
>>> cluster split brain (also called cluster partition).
>>>
>>> So I'd really like to know what happens in your
>>> setup in a split brain situation.
>>
>> Well, it turns out that drbd has this thing called dual-master mode,
>> which turns it into shared storage for two nodes only.
>>
>> So, as long as the OCFS2 DLM is also running, there should not be any
>> split-brain events.
>
> Hi Richard,
>
> i also spent some time with OCFS2 - but was *not* able to get the
> CTDB_RECOVERY_LOCK working properly! Which is a No-go for production.

I didn't have that problem and it all worked well. I still have those
nodes/VMs but will only have them for the next two weeks.

I will try to repro this elsewhere or at home, but I have other stuff
to do as well.

The problem I had was that CentOS 6.6 has moved to a slightly
different version of the Clustering tools, and I had to manually build
ocfs2_tools.

> When starting the 2nd node the log always showed:
>   ERROR: recovery lock file /mnt/ctdb_lock/ctdb_lockfile not locked when recovering!
> and both nodes started wild logging... :-(
>
> I then had a very close look at the ping_pong source and i think it can
> reliably be used to test the fcntl() locking features.
>
> With OCFS2 i was also *not* able to get sane results with the ping_pong test.
>
> For a 2 node cluster, even simple
>    ping_pong shared_file 3
> when running on both nodes should result in a significant drop in locks/second.
> This was not the case here.
>
> When using
>    ping_pong -rw shared_file 3
> some stuff *seems* to be working right - but not reliably.
> When starting the 2nd node, it *could* happen that
>   data increment = 2
> is shown right. But when you stop ping_pong on that node and start it again,
> it shows some random nature. Btw - the locks/second always dropped a lot, but
> that was the only reliable result.
>
> I'm not sure whether corosync, pacemaker and friends can really help here.
> That would be some heavy weight overkill...
>
> We'll see. I install GFS2 now ...
>
> Cheers, Günter
>
>>
>> Making sure that the DLM was running was why I put so much effort into
>> getting the ocfs2-tools code running.
>>
>> The disadvantage of using DRBD is that you cannot run more than a
>> 2-node cluster.
>>
>
>
> --
>



-- 
Regards,
Richard Sharpe
(何以解憂?唯有杜康。--曹操)


More information about the samba-technical mailing list