SV: ctdb relock file issues with glusterfs

Morten Bøhmer Morten.Bohmer at
Wed Oct 17 15:40:05 MDT 2012


I had my gluster-volume area on /mnt/gluster/lock and mounted the glustervolume on /mnt/lock

In other words I made on gluster volume exclusively for ctdb.

From: patrick medina [pgmedinajr at]
Sent: Wednesday, October 17, 2012 11:01 PM
To: Morten Bøhmer
Cc: Christopher R. Hertel; samba-technical at
Subject: Re: SV: ctdb relock file issues with glusterfs

Yes, a very big thank you to Christopher and everyone else, that mount command did the trick for me as well.

At first this didn't work, but I moved the location of the lock file to the root of the gluster share.  (/mnt/gluster/ctdb/lock  to /mnt/gluster/lock)  Now we're all healthy and happy!

Morten, was your lock file in the same location or did you have to move it as well?


On Mon, Oct 15, 2012 at 4:07 PM, Morten Bøhmer <Morten.Bohmer at<mailto:Morten.Bohmer at>> wrote:

This did it for me :)

Ping_pong is now showing correct results


-----Opprinnelig melding-----
Fra: samba-technical-bounces at<mailto:samba-technical-bounces at> [mailto:samba-technical-bounces at<mailto:samba-technical-bounces at>] På vegne av Christopher R. Hertel
Sendt: 15. oktober 2012 20:50
Til: samba-technical at<mailto:samba-technical at>
Emne: Re: SV: ctdb relock file issues with glusterfs

Morten, Patrick:

Please try your tests with the following option on mount:

Let us know whether that changes your test results.


Chris -)-----

On 10/15/2012 10:59 AM, Morten Bøhmer wrote:
> Thank you.
> For the heck of it I installed a couple of Centos virtual servers and configure ctdb+glusterfs+xfs+samba, got it working, but without relock.
> Not sure how important it is, but I guess time will show :)
> Morten
> Fra: patrick medina [mailto:pgmedinajr at<mailto:pgmedinajr at>]
> Sendt: 15. oktober 2012 17:57
> Til: Morten Bøhmer
> Kopi: Michael Adam; samba-technical at<mailto:samba-technical at>
> Emne: Re: ctdb relock file issues with glusterfs
> Morning Morten,
> I have been out of the office since Thursday, but am back today and ready to knock this out.  I'll keep you posted on what i find later this afternoon.
> Cheers
> On Fri, Oct 12, 2012 at 7:34 AM, Morten Bøhmer <Morten.Bohmer at<mailto:Morten.Bohmer at><mailto:Morten.Bohmer at<mailto:Morten.Bohmer at>>> wrote:
> Hi Patrick
> Any luck with your setup yet ?
> I am now seriously looking into trying some other clusterfs to make ctdb work.
> Morten
> Fra: patrick medina
> [mailto:pgmedinajr at<mailto:pgmedinajr at><mailto:pgmedinajr at<mailto:pgmedinajr at>>]
> Sendt: 10. oktober 2012 17:56
> Til: Michael Adam
> Kopi: Morten Bøhmer;
> samba-technical at<mailto:samba-technical at><mailto:samba-technical at<mailto:samba-technical at>
> >
> Emne: Re: ctdb relock file issues with glusterfs
> Thanks Michael,
> The way you explained ping_pong (going from "1"
> to "2") isn't explain as well on the wiki so i'll test and most likely verify it will not increment.
> Cheers - Gil
> On Wed, Oct 10, 2012 at 4:03 AM, Michael Adam <obnox at<mailto:obnox at><mailto:obnox at<mailto:obnox at>>> wrote:
> Hi folks,
> as indicated elsewhere already, before even trying to start and debug
> ctdb, you should make sure that your cluster setup provides correct
> posix fcntl byte range locks, by using the ping_pong tool shipped with
> the ctdb package:
> It is important to verify that the locks really reach "the other
> node", i.e. there is real lock contention.
> This can in particular be tested with the -rw option to
> ping_pong: If you run "ping_pong -rw /path/to/file 3" on one node and
> then "ping_pong -rw /path/to/file 3" on a second node, you should see
> the "data increment" notice (going from "1"
> to "2"), indicating that you now have two processes operating on the
> same file. If this stays constant (at 1) then your gluster setup does
> not provide sufficient fcntl byte range lock support.
> Another way to verify this without "-rw" is using file that is one too
> small:  run "ping_pong /path/to/file 2" on one node and then the same
> command on a second node. These should block and not print positive
> lock rates. If instead both happily print positive lock rates then
> your locks don't reach the other node and you need to fix your
> setup...
> Cheers - Michael
> On 2012-10-09 at 22:21 +0000, Morten Bøhmer wrote:
>> Can confirm that I am experiencing the exact same issue.
>> Would love to be able to solve this .....
>> Morten
>> ________________________________________
>> From:
>> samba-technical-bounces at<mailto:samba-technical-bounces at><mailto:samba-technical-bounce<mailto:samba-technical-bounce>
>> s at<mailto:s at>>
>> [samba-technical-bounces at<mailto:samba-technical-bounces at><mailto:samba-technical-bounc<mailto:samba-technical-bounc>
>> es at<mailto:es at>>] on behalf of patrick medina
>> [pgmedinajr at<mailto:pgmedinajr at><mailto:pgmedinajr at<mailto:pgmedinajr at>>]
>> Sent: Wednesday, October 10, 2012 12:10 AM
>> To:
>> samba-technical at<mailto:samba-technical at><mailto:samba-technical at lists.samba.or<mailto:samba-technical at lists.samba.or>
>> g>
>> Subject: Re: ctdb relock file issues with glusterfs
>> Afternoon/Morning Samba folks,
>> I finally made some progress this afternoon, let me explain what I found.
>> 1.  When I created the lock file, I had set it to chmod 777
>> (rwxrwxrwx) Thinking about permissions, I recreated the lock file with rw-r--r--.
>>   After doing this I am now able to bring one node to healthy at a
>> time, but the other node will stay unhealthy.  I am able to juggle
>> healthy nodes by shutting the ctdb service down and the 2nd node will become healthy.
>> Log file on the unhealthy nodes complain about the recovery lock file
>> not
>> locked:
>> 2012/10/09 14:55:40.335328 [set_recmode:16493]: ctdb_recovery_lock:
>> Got recovery lock on '/mnt/gluster/ctdb/lock'
>> 2012/10/09 14:55:40.335448 [set_recmode:16493]: ERROR: recovery lock
>> file /mnt/gluster/ctdb/lock not locked when recovering!
>> 2.  I created new mount point on one of the nodes, so each node has a
>> unique mount to gluster.  Depending on which node starts first, the
>> unhealthy node complaints about the others recovery lock location.
>> How can this be if each node has it's on config file to go off of?
>> Node1:  CTDB_RECOVERY_LOCK="/mnt/fuse/ctdb/lock"
>> ctdb_recovery_lock: Unable to open /mnt/gluster/ctdb/lock - (No such
>> file or directory)
>> Node2:  CTDB_RECOVERY_LOCK="/mnt/gluster/ctdb/lock"
>> ctdb_recovery_lock: Unable to open /mnt/fuse/ctdb/lock - (No such
>> file or
>> directory)
>> Thanks again, I'm not sure where to troubleshoot next.
>> Regards,
>> Gilbert
>> On Tue, Oct 9, 2012 at 5:20 AM, Martin Schwenke <martin at<mailto:martin at><mailto:martin at<mailto:martin at>>> wrote:
>>> On Tue, 9 Oct 2012 16:32:12 +1100, Amitay Isaacs
>>> <amitay at<mailto:amitay at><mailto:amitay at<mailto:amitay at>>>
>>> wrote:
>>>> On Tue, Oct 9, 2012 at 1:55 PM, patrick medina
>>>> <pgmedinajr at<mailto:pgmedinajr at><mailto:pgmedinajr at<mailto:pgmedinajr at>>>
>>> wrote:
>>>>> Howdy samba folks,
>>>>> I've been running into a lot of issues lately with ctdb's re-lock
>>>>> file
>>> and
>>>>> glusterfs as the shared storage.  When I started, I could get one
>>>>> or
>>> the
>>>>> other node to become healthy, but at least one would complain it
>>>>> could
>>> not
>>>>> lock the re-lock file.  Nowi'm at the point where neither node
>>>>> will
>>> become
>>>>> healthy and stay in a recovery loop.  Just to be sure it was the
>>> re-lock
>>>>> file, I commented it out in the config and both nodes became healthy.
>>>> What version of CTDB are you using? Can you attach the log file
>>>> where you notice CTDB is continuously going in recovery? It would
>>>> be useful to get log files from all the nodes.
>>> Michael Adam and I took a look at this on the weekend.  Gilbert sent
>>> me some logs and this was happening:
>>>    ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
>>> That seems to indicate that locking isn't working as expected...
>>> peace & happiness,
>>> martin

"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team --     -)-----   Christopher R. Hertel
jCIFS Team --   -)-----   ubiqx development, uninq.
ubiqx Team --     -)-----   crh at<mailto:crh at>
OnLineBook --    -)-----   crh at<mailto:crh at>

More information about the samba-technical mailing list