Setting up CTDB on OCFS2 and VMs ...

Thu Dec 4 11:08:36 MST 2014

Hi folks,

Here are the steps I used, as far as I can remember them. Please
excuse any mistakes and be prepared to think for yourself when
following them.

1. Create two VirtualBox VMs with enough memory and disk for your
Linux Distro. I used CentOS 6.6 with 4GB and 20GB. (I actually
installed CentOS 6.3 and upgraded because I had the ISO handy.) You
will also need an extra interface on each VM for the clustering
private network. I set them to an internal type.

2. Because you will need a shared disk, create one:

vboxmanage createhd --filename ~/VirtualBox\ VMs/SharedHD1 --size
10240 --variant Fixed --format VDI # Creates a 10GB fixed sized disk
vboxmanage modifyhd 22ae1fcc-fda7-4e42-be9f-3b8bd7fc0c0e --type
shareable # Make it shareable.

Note, for that second command use the UUID for your disk which you can
find with:

vboxmanage list hdds --brief

Also, use the GUI to add the shared disk to both VMs.

3. Install the OS on each of the VMs.

4. I installed a bunch of clustering RPMs next:

yum install openais corosync pacemaker-libs pacemaker-libs-devel gcc
corosync-devel openais-devel rpm-build e2fsprogs-devel libuuid-devel
git pygtk2 python-devel readline-devel clusterlib-devel redhat-lsb
sqlite-devel gnutls-devel byacc flex nss-devel

It is not clear to me that openais was needed, for example

5. Next I installed Oracles UEK and ocfs2-tools

wget http://public-yum.oracle.com/public-yum-ol6.repo /etc/yum.repos.d
wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 -O
/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
yum install kernel-uek kernel-uek-devel
yum install ocfs2-tools
yum install openaislib-devel corosync-devel # It is not clear that I
needed to install the first

echo 'KERNEL=="ocfs2_control", NAME="misc/ocfs2_control", MODE="0660"'
> /etc/udev/rules.d/99-ocfs2_control.rules

reboot # on each

6. Configure cman and pacemaker

# configure corosync first
cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

# Make sure that bindnetaddr is defined and points to your private
interface. I set it to 192.168.122.0

# Make sure that the mcastaddr is defined. I used 239.255.1.1
# make sure that the mcastport is defined. I used 5405

# Copy that file to the other node.
scp /etc/corosync/corosync.conf root at 172.16.170.6:/etc/corosync

/etc/init.d/pacemaker stop  # Stop these in case they were running
/etc/init.d/corosync stop # Same here

yum install ccs pcs

# Create a cluster

ccs -f /etc/cluster/cluster.conf --createcluster ctdbdemo
ccs -f /etc/cluster/cluster.conf --addnode ocfs2-1
ccs -f /etc/cluster/cluster.conf --addnode ocfs2-2
ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-1
ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-2
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-1
pcmk-redirect port=ocfs2-1
ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-2
pcmk-redirect port=ocfs2-2

# Copy the cluster config file to the other node:
scp /etc/cluster/cluster.conf root at 172.16.170.6:/etc/cluster

#Now turn off NetworkManager:
chkconfig NetworkManager off
service NetworkManager stop

# now start the cluster
service cman start
pcs property set stonith-enabled=false
service pacemaker start

# Also start it on the other node(s).

# Now check the status:
    [root at ocfs2-1 ~]# crm_mon -1
    Last updated: Thu Dec  4 09:40:16 2014
    Last change: Tue Dec  2 10:12:50 2014
    Stack: cman
    Current DC: ocfs2-2 - partition with quorum
    Version: 1.1.11-97629de
    2 Nodes configured
    0 Resources configured

    Online: [ ocfs2-1 ocfs2-2 ]

If you do not see all the other nodes online, then you have to debug
the problem.

These are essentially the steps from here:
http://clusterlabs.org/quickstart-redhat.html

7. Configure the Oracle cluster

o2cb add-cluster ctdbdemo
o2cb add-node --ip 192.168.122.10 --port --number 1 ctdbdemo ocfs2-1
o2cb add-node --ip 192.168.122.10 --port 7777 --number 1 ctdbdemo ocfs2-1
o2cb add-node --ip 192.168.122.11 --port 7777 --number 2 ctdbdemo ocfs2-2

service o2cb configure # This step will fail claiming that it can't
find /sbin/ocfs2_controld.cman
#
# However, it does the important stuff.
#
# NOTE, during the configuration steps you MUST SELECT cman AS THE
CLUSTER STACK!
#

8. Find and install the ocfs2-tools git repos

git clone git://oss.oracle.com/git/ocfs2-tools.git ocfs2-tools

# install stuff needed
yum install libaio libaio-devel
yum install pacemaker-libs-devel

# Now build
cd ocfs2-tools
./configure
make

# This will likely fail. If it first fails complaining about
xml/tree.h then you can do the following:
CPPFLAGS='-I/usr/include/libxml2' ./configure
make

# It might complain again complaining about some AIS include files that are no
# longer in the packages installed. That is OK. It should have built
ocfs2_controld.cman,
# so copy it where it needs to be:

cp ocfs2_controld.cman /usr/sbin/
scp ocfs2_controld.cman root at 172.16.170.6:/usr/sbin/

# Now stop those two you started and start everything:

service pacemaker stop
service cman stop

service cman start
service o2cb start
service pacemaker start

8. Create the shared shared file system on one node:

mkfs.ocfs2 -L CTDBdemocommon --cluster-name ctdbdemo --cluster-stack
ocfs2 -N 4 /dev/sdb

9. Mount it on both and ensure that you can create files/dirs on one
node and see them on the other node.

10. Install ctdb and samba

11. Configure samba for the domain you want to join

# Make sure you have clustering = yes and the other things you need.

12. Configure ctdb (/etc/sysconfig/ctdb) and make sure that you disable winbindd

13. Start ctdb on all nodes

# You must have ctdb started so that the secrets file will get distributed

14. join the domain

15. Enable winbindd in the ctdb config

16. Restart ctdb on all nodes

At this point you should be done. The steps you need might vary.

I have limited time to help you with this.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)