Setting up CTDB on OCFS2 and VMs ...

Thu Dec 11 09:47:24 MST 2014

On 06/12/14 15:33, Rowland Penny wrote:
> On 06/12/14 15:24, Richard Sharpe wrote:
>> On Sat, Dec 6, 2014 at 2:58 AM, Rowland Penny 
>> <repenny241155 at gmail.com> wrote:
>>> On 04/12/14 18:08, Richard Sharpe wrote:
>>>> Hi folks,
>>>>
>>>> Here are the steps I used, as far as I can remember them. Please
>>>> excuse any mistakes and be prepared to think for yourself when
>>>> following them.
>>>>
>>>> 1. Create two VirtualBox VMs with enough memory and disk for your
>>>> Linux Distro. I used CentOS 6.6 with 4GB and 20GB. (I actually
>>>> installed CentOS 6.3 and upgraded because I had the ISO handy.) You
>>>> will also need an extra interface on each VM for the clustering
>>>> private network. I set them to an internal type.
>>>>
>>>> 2. Because you will need a shared disk, create one:
>>>>
>>>> vboxmanage createhd --filename ~/VirtualBox\ VMs/SharedHD1 --size
>>>> 10240 --variant Fixed --format VDI # Creates a 10GB fixed sized disk
>>>> vboxmanage modifyhd 22ae1fcc-fda7-4e42-be9f-3b8bd7fc0c0e --type
>>>> shareable # Make it shareable.
>>>>
>>>> Note, for that second command use the UUID for your disk which you can
>>>> find with:
>>>>
>>>> vboxmanage list hdds --brief
>>>>
>>>> Also, use the GUI to add the shared disk to both VMs.
>>>>
>>>> 3. Install the OS on each of the VMs.
>>>>
>>>> 4. I installed a bunch of clustering RPMs next:
>>>>
>>>> yum install openais corosync pacemaker-libs pacemaker-libs-devel gcc
>>>> corosync-devel openais-devel rpm-build e2fsprogs-devel libuuid-devel
>>>> git pygtk2 python-devel readline-devel clusterlib-devel redhat-lsb
>>>> sqlite-devel gnutls-devel byacc flex nss-devel
>>>>
>>>> It is not clear to me that openais was needed, for example
>>>>
>>>> 5. Next I installed Oracles UEK and ocfs2-tools
>>>>
>>>> wget http://public-yum.oracle.com/public-yum-ol6.repo /etc/yum.repos.d
>>>> wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 -O
>>>> /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
>>>> yum install kernel-uek kernel-uek-devel
>>>> yum install ocfs2-tools
>>>> yum install openaislib-devel corosync-devel # It is not clear that I
>>>> needed to install the first
>>>>
>>>> echo 'KERNEL=="ocfs2_control", NAME="misc/ocfs2_control", MODE="0660"'
>>>>> /etc/udev/rules.d/99-ocfs2_control.rules
>>>> reboot # on each
>>>>
>>>> 6. Configure cman and pacemaker
>>>>
>>>> # configure corosync first
>>>> cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
>>>>
>>>> # Make sure that bindnetaddr is defined and points to your private
>>>> interface. I set it to 192.168.122.0
>>>>
>>>> # Make sure that the mcastaddr is defined. I used 239.255.1.1
>>>> # make sure that the mcastport is defined. I used 5405
>>>>
>>>> # Copy that file to the other node.
>>>> scp /etc/corosync/corosync.conf root at 172.16.170.6:/etc/corosync
>>>>
>>>> /etc/init.d/pacemaker stop  # Stop these in case they were running
>>>> /etc/init.d/corosync stop # Same here
>>>>
>>>> yum install ccs pcs
>>>>
>>>> # Create a cluster
>>>>
>>>> ccs -f /etc/cluster/cluster.conf --createcluster ctdbdemo
>>>> ccs -f /etc/cluster/cluster.conf --addnode ocfs2-1
>>>> ccs -f /etc/cluster/cluster.conf --addnode ocfs2-2
>>>> ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk
>>>> ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-1
>>>> ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-2
>>>> ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-1
>>>> pcmk-redirect port=ocfs2-1
>>>> ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-2
>>>> pcmk-redirect port=ocfs2-2
>>>>
>>>> # Copy the cluster config file to the other node:
>>>> scp /etc/cluster/cluster.conf root at 172.16.170.6:/etc/cluster
>>>>
>>>> #Now turn off NetworkManager:
>>>> chkconfig NetworkManager off
>>>> service NetworkManager stop
>>>>
>>>> # now start the cluster
>>>> service cman start
>>>> pcs property set stonith-enabled=false
>>>> service pacemaker start
>>>>
>>>> # Also start it on the other node(s).
>>>>
>>>> # Now check the status:
>>>>       [root at ocfs2-1 ~]# crm_mon -1
>>>>       Last updated: Thu Dec  4 09:40:16 2014
>>>>       Last change: Tue Dec  2 10:12:50 2014
>>>>       Stack: cman
>>>>       Current DC: ocfs2-2 - partition with quorum
>>>>       Version: 1.1.11-97629de
>>>>       2 Nodes configured
>>>>       0 Resources configured
>>>>
>>>>
>>>>       Online: [ ocfs2-1 ocfs2-2 ]
>>>>
>>>> If you do not see all the other nodes online, then you have to debug
>>>> the problem.
>>>>
>>>> These are essentially the steps from here:
>>>> http://clusterlabs.org/quickstart-redhat.html
>>>>
>>>> 7. Configure the Oracle cluster
>>>>
>>>> o2cb add-cluster ctdbdemo
>>>> o2cb add-node --ip 192.168.122.10 --port --number 1 ctdbdemo ocfs2-1
>>>> o2cb add-node --ip 192.168.122.10 --port 7777 --number 1 ctdbdemo 
>>>> ocfs2-1
>>>> o2cb add-node --ip 192.168.122.11 --port 7777 --number 2 ctdbdemo 
>>>> ocfs2-2
>>>>
>>>> service o2cb configure # This step will fail claiming that it can't
>>>> find /sbin/ocfs2_controld.cman
>>>> #
>>>> # However, it does the important stuff.
>>>> #
>>>> # NOTE, during the configuration steps you MUST SELECT cman AS THE
>>>> CLUSTER STACK!
>>>> #
>>>>
>>>> 8. Find and install the ocfs2-tools git repos
>>>>
>>>> git clone git://oss.oracle.com/git/ocfs2-tools.git ocfs2-tools
>>>>
>>>> # install stuff needed
>>>> yum install libaio libaio-devel
>>>> yum install pacemaker-libs-devel
>>>>
>>>> # Now build
>>>> cd ocfs2-tools
>>>> ./configure
>>>> make
>>>>
>>>> # This will likely fail. If it first fails complaining about
>>>> xml/tree.h then you can do the following:
>>>> CPPFLAGS='-I/usr/include/libxml2' ./configure
>>>> make
>>>>
>>>> # It might complain again complaining about some AIS include files 
>>>> that
>>>> are no
>>>> # longer in the packages installed. That is OK. It should have built
>>>> ocfs2_controld.cman,
>>>> # so copy it where it needs to be:
>>>>
>>>> cp ocfs2_controld.cman /usr/sbin/
>>>> scp ocfs2_controld.cman root at 172.16.170.6:/usr/sbin/
>>>>
>>>> # Now stop those two you started and start everything:
>>>>
>>>> service pacemaker stop
>>>> service cman stop
>>>>
>>>> service cman start
>>>> service o2cb start
>>>> service pacemaker start
>>>>
>>>> 8. Create the shared shared file system on one node:
>>>>
>>>> mkfs.ocfs2 -L CTDBdemocommon --cluster-name ctdbdemo --cluster-stack
>>>> ocfs2 -N 4 /dev/sdb
>>>>
>>>> 9. Mount it on both and ensure that you can create files/dirs on one
>>>> node and see them on the other node.
>>>>
>>>> 10. Install ctdb and samba
>>>>
>>>> 11. Configure samba for the domain you want to join
>>>>
>>>> # Make sure you have clustering = yes and the other things you need.
>>>>
>>>> 12. Configure ctdb (/etc/sysconfig/ctdb) and make sure that you 
>>>> disable
>>>> winbindd
>>>>
>>>> 13. Start ctdb on all nodes
>>>>
>>>> # You must have ctdb started so that the secrets file will get 
>>>> distributed
>>>>
>>>> 14. join the domain
>>>>
>>>> 15. Enable winbindd in the ctdb config
>>>>
>>>> 16. Restart ctdb on all nodes
>>>>
>>>> At this point you should be done. The steps you need might vary.
>>>>
>>>> I have limited time to help you with this.
>>>>
>>> OK, I have followed Richards 'howto' but using Debian 7.7 instead of 
>>> Centos,
>>> I also used the standard Debian kernel. I have got up to step 9, 
>>> after a bit
>>> of a battle and it all started so well. :-)
>>>
>>> Most of the required packages are available from the repos:
>>>
>>> apt-get install openais corosync pacemaker cman ocfs2-tools-cman
>>> ocfs2-tools-pacemaker
>>>
>>> Unfortunately, it turned out that pacemaker is not built to use the 
>>> cman
>>> stack, so I had to rebuild it
>>>
>>> Next problem, ccs and pcs are not available, so I had to download & 
>>> build
>>> them, though even this was not without problems, ccs put
>>> 'empty_cluster.conf' in the wrong place and pcs is hardwired to use
>>> /usr/libexec
>>>
>>> Next problem 'o2cb' appears to be called 'o2cb_ctl' on Debian.
>>>
>>> Started cman, o2cb and pacemaker (first time round, this is where I 
>>> found
>>> that pacemaker wouldn't work with cman)
>>>
>>> I then created the shared shared file system and mounted it on both 
>>> nodes
>> OK, looks like you got the real stuff done.
>>
>>> At this point I have a shared cluster, but in a way that I cannot 
>>> see any
>>> sane sysadmin using. Most of the software is heavily modified or not
>>> available from the distros repos. I am going to have to stop and 
>>> think about
>>> this and see if there is a Debian way of doing this, without modifying
>>> anything or using anything that is not available from a repo.
>
> OK, I am getting closer :-)
>
> I have got it working with just packages available from Debian repos, 
> apart from 'ccs', once I find  a replacement for this, I will move 
> onto ctdb & samba.
>
> Rowland
>
>> Indeed, I cannot imagine anyone using the approach I used for
>> production as well. Having stuff that needs to be rebuilt is not a
>> good idea, and it would be useful to get a minimal complete set of
>> RPMs together and fix ocfs2-tools so that the correct things are there
>> and the build works out what is needed.
>>
>> However, I only currently need the OCFS2 setup for testing purposes
>> while we get our file system supporting FCNTL locks and so I have a
>> reference to work with.
>>
>> Not sure if I have the time to fix anything.
>>
>

OK, I officially give up! :-D

I can get a two node cluster working on Debian 7.7 very easily, no 
compiling of software, but after this, it goes downhill fast. Whatever I 
do I cannot get ctdb to work, I feel a lot of this is down to the very 
small amount of documentation and what there is, seems to be biased 
towards redhat.

Searching the internet brings up very little and what I find is 
contradictory, also quite a lot of what I found would say something 
like, 'setup ctdb' but wouldn't explain how or why. Don't get me started 
on 'nodes', just what ipaddress should I use for each node? do I really 
need two network interfaces and why?

So, I have come to the conclusion, I personally cannot setup Debian, 
ocf2s, corosync, ctdb and samba, because I cannot find any documentation 
that is complete and easily understandable. It is no good the people 
writing ctdb knowing how to set it up, they need to look at the 
documentation before they go much further.

Sorry if this upsets anybody, it just how I feel at the moment.

Rowland