ctdb reclock + fcntl posix locks with ocfs2
nicolas at ecarnot.net
Thu Nov 28 04:19:49 MST 2013
What to tweak for ctdb reclock dir to handle correct fcntl locks on
[Less short version]
We are using in production a 2 nodes cluster that is working fine :
Ubuntu server 12.10, cman, samba, ctdb.
The ctdb reclock are done on a dedicated GFS2 partition, and the data
storage (samba user shares) are done on a dedicated OCFS2 partition.
At the time being, I am setting up a similar cluster but with updated
versions, distribs and with the will to simplify some parts :
- 2 nodes, CentOS 6.4 64 bits, as two oVirt VM
- cman 3.0.12
- corosync 1.4.1
- samba 3.6.9
- ctdb 18.104.22.168-3
- ocfs2 1.8.0-10.el6
- ctdb lock LUN and ocfs2 user data both stored in a remote equalogic
- UEK kernel : 2.6.39-400.211.1.el6uek.x86_64
I would like to get rid of two different clustered filesystem types
(gfs2, ocfs2), and keep only ocfs2, as my 3 To of data are already
stored this way.
Once everything is setup, I run ctdb on the first node, then on the
second, and I'm facing a problem I remember I already had to cope with 2
years ago on the production cluster :
ctdb_recovery_lock: Got recovery lock on '/ctdb/.ctdb.lock'
ERROR: recovery lock file /ctdb/.ctdb.lock not locked when recovering!
This was two years ago, but I remember I had to switch back to store
this lock mechanism on a GFS2 partition.
I took the time to search and test many things.
Ping_pong tests showed this :
* 1 *
ping_pong /ctdb/test.dat 3
showing ~ 2.1 M locks/s, and /NOT/ dropping to a different value when
ran on the second node. Symmetrical, though.
* 2 *
ping_pong -rw /ctdb/test.dat 3
showing ~ 330 k locks/s, and dropping to 190 locks/s when ran on node 2.
* 3 *
ping_pong -rw -m /ctdb/test.dat 3
showing ~ 2.1 M locks/s, and dropping to 88 k locks/s when ran on node
2, and showing several data increment oscillating between 1 and
In the test 1, according to the ctdb webpage advices, the value _NOT_
dropping seems to indicate an issue, so I tried to validate the correct
POSIX locking, following the advices found there :
and the tests were successful.
I found that this samba-technical mailing list helped someone to solve a
similar problem when storing the ctdb lock on a gluster FS
but on ocfs2, I have no mean to use the --direct-io-mode=enable mount
option). In the OCFS2 man page, I read this :
This mount option has been deprecated in OCFS2 1.6. It has been used in
the past (OCFS2 1.2 and OCFS2 1.4), to force the Oracle RDBMS to
issue direct IOs to the hosted data files, control files, redo logs,
archive logs, voting disk, cluster registry, etc. It has been deprecated
because it is no longer required. Oracle RDBMS users should instead
use the init.ora parameter, filesystemio_options, to enable direct IOs."
I also found this relevant bug :
but the version of filesystem and kernel I'm using should be more than
capable of handling all this. This is confirmed on Oracle's ocfs2 1.6
My cluster.conf contains the following line :
<dlm plock_ownership="1" plock_rate_limit="0"/>
because this seemed correct to me according to what I understood so far.
I'd be very glad to hear your advices on what I could change next, or
what to check.
On my side, I will now test the setup I wanted to avoid : store the ctdb
lock on a GFS2 partition, and see if it works as on our old production
More information about the samba-technical