Setting up CTDB on OCFS2 and VMs ...

[Richard Sharpe]

On Thu, Dec 11, 2014 at 9:46 AM, Rowland Penny <repenny241155 at gmail.com> wrote:
> On 11/12/14 17:28, steve wrote:
>>
>> On 11/12/14 17:47, Rowland Penny wrote:
>>>
>>> On 06/12/14 15:33, Rowland Penny wrote:
>>>>
>>>> On 06/12/14 15:24, Richard Sharpe wrote:
>>>>>
>>>>> On Sat, Dec 6, 2014 at 2:58 AM, Rowland Penny
>>>>> <repenny241155 at gmail.com> wrote:
>>>>>>
>>>>>> On 04/12/14 18:08, Richard Sharpe wrote:
>>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Here are the steps I used, as far as I can remember them. Please
>>>>>>> excuse any mistakes and be prepared to think for yourself when
>>>>>>> following them.
>>>>>>>
>>>>>>> 1. Create two VirtualBox VMs with enough memory and disk for your
>>>>>>> Linux Distro. I used CentOS 6.6 with 4GB and 20GB. (I actually
>>>>>>> installed CentOS 6.3 and upgraded because I had the ISO handy.) You
>>>>>>> will also need an extra interface on each VM for the clustering
>>>>>>> private network. I set them to an internal type.
>>>>>>>
>>>>>>> 2. Because you will need a shared disk, create one:
>>>>>>>
>>>>>>> vboxmanage createhd --filename ~/VirtualBox\ VMs/SharedHD1 --size
>>>>>>> 10240 --variant Fixed --format VDI # Creates a 10GB fixed sized disk
>>>>>>> vboxmanage modifyhd 22ae1fcc-fda7-4e42-be9f-3b8bd7fc0c0e --type
>>>>>>> shareable # Make it shareable.
>>>>>>>
>>>>>>> Note, for that second command use the UUID for your disk which you
>>>>>>> can
>>>>>>> find with:
>>>>>>>
>>>>>>> vboxmanage list hdds --brief
>>>>>>>
>>>>>>> Also, use the GUI to add the shared disk to both VMs.
>>>>>>>
>>>>>>> 3. Install the OS on each of the VMs.
>>>>>>>
>>>>>>> 4. I installed a bunch of clustering RPMs next:
>>>>>>>
>>>>>>> yum install openais corosync pacemaker-libs pacemaker-libs-devel gcc
>>>>>>> corosync-devel openais-devel rpm-build e2fsprogs-devel libuuid-devel
>>>>>>> git pygtk2 python-devel readline-devel clusterlib-devel redhat-lsb
>>>>>>> sqlite-devel gnutls-devel byacc flex nss-devel
>>>>>>>
>>>>>>> It is not clear to me that openais was needed, for example
>>>>>>>
>>>>>>> 5. Next I installed Oracles UEK and ocfs2-tools
>>>>>>>
>>>>>>> wget http://public-yum.oracle.com/public-yum-ol6.repo
>>>>>>> /etc/yum.repos.d
>>>>>>> wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 -O
>>>>>>> /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
>>>>>>> yum install kernel-uek kernel-uek-devel
>>>>>>> yum install ocfs2-tools
>>>>>>> yum install openaislib-devel corosync-devel # It is not clear that I
>>>>>>> needed to install the first
>>>>>>>
>>>>>>> echo 'KERNEL=="ocfs2_control", NAME="misc/ocfs2_control",
>>>>>>> MODE="0660"'
>>>>>>>>
>>>>>>>> /etc/udev/rules.d/99-ocfs2_control.rules
>>>>>>>
>>>>>>> reboot # on each
>>>>>>>
>>>>>>> 6. Configure cman and pacemaker
>>>>>>>
>>>>>>> # configure corosync first
>>>>>>> cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
>>>>>>>
>>>>>>> # Make sure that bindnetaddr is defined and points to your private
>>>>>>> interface. I set it to 192.168.122.0
>>>>>>>
>>>>>>> # Make sure that the mcastaddr is defined. I used 239.255.1.1
>>>>>>> # make sure that the mcastport is defined. I used 5405
>>>>>>>
>>>>>>> # Copy that file to the other node.
>>>>>>> scp /etc/corosync/corosync.conf root at 172.16.170.6:/etc/corosync
>>>>>>>
>>>>>>> /etc/init.d/pacemaker stop  # Stop these in case they were running
>>>>>>> /etc/init.d/corosync stop # Same here
>>>>>>>
>>>>>>> yum install ccs pcs
>>>>>>>
>>>>>>> # Create a cluster
>>>>>>>
>>>>>>> ccs -f /etc/cluster/cluster.conf --createcluster ctdbdemo
>>>>>>> ccs -f /etc/cluster/cluster.conf --addnode ocfs2-1
>>>>>>> ccs -f /etc/cluster/cluster.conf --addnode ocfs2-2
>>>>>>> ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk
>>>>>>> ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-1
>>>>>>> ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-2
>>>>>>> ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-1
>>>>>>> pcmk-redirect port=ocfs2-1
>>>>>>> ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-2
>>>>>>> pcmk-redirect port=ocfs2-2
>>>>>>>
>>>>>>> # Copy the cluster config file to the other node:
>>>>>>> scp /etc/cluster/cluster.conf root at 172.16.170.6:/etc/cluster
>>>>>>>
>>>>>>> #Now turn off NetworkManager:
>>>>>>> chkconfig NetworkManager off
>>>>>>> service NetworkManager stop
>>>>>>>
>>>>>>> # now start the cluster
>>>>>>> service cman start
>>>>>>> pcs property set stonith-enabled=false
>>>>>>> service pacemaker start
>>>>>>>
>>>>>>> # Also start it on the other node(s).
>>>>>>>
>>>>>>> # Now check the status:
>>>>>>>       [root at ocfs2-1 ~]# crm_mon -1
>>>>>>>       Last updated: Thu Dec  4 09:40:16 2014
>>>>>>>       Last change: Tue Dec  2 10:12:50 2014
>>>>>>>       Stack: cman
>>>>>>>       Current DC: ocfs2-2 - partition with quorum
>>>>>>>       Version: 1.1.11-97629de
>>>>>>>       2 Nodes configured
>>>>>>>       0 Resources configured
>>>>>>>
>>>>>>>
>>>>>>>       Online: [ ocfs2-1 ocfs2-2 ]
>>>>>>>
>>>>>>> If you do not see all the other nodes online, then you have to debug
>>>>>>> the problem.
>>>>>>>
>>>>>>> These are essentially the steps from here:
>>>>>>> http://clusterlabs.org/quickstart-redhat.html
>>>>>>>
>>>>>>> 7. Configure the Oracle cluster
>>>>>>>
>>>>>>> o2cb add-cluster ctdbdemo
>>>>>>> o2cb add-node --ip 192.168.122.10 --port --number 1 ctdbdemo ocfs2-1
>>>>>>> o2cb add-node --ip 192.168.122.10 --port 7777 --number 1 ctdbdemo
>>>>>>> ocfs2-1
>>>>>>> o2cb add-node --ip 192.168.122.11 --port 7777 --number 2 ctdbdemo
>>>>>>> ocfs2-2
>>>>>>>
>>>>>>> service o2cb configure # This step will fail claiming that it can't
>>>>>>> find /sbin/ocfs2_controld.cman
>>>>>>> #
>>>>>>> # However, it does the important stuff.
>>>>>>> #
>>>>>>> # NOTE, during the configuration steps you MUST SELECT cman AS THE
>>>>>>> CLUSTER STACK!
>>>>>>> #
>>>>>>>
>>>>>>> 8. Find and install the ocfs2-tools git repos
>>>>>>>
>>>>>>> git clone git://oss.oracle.com/git/ocfs2-tools.git ocfs2-tools
>>>>>>>
>>>>>>> # install stuff needed
>>>>>>> yum install libaio libaio-devel
>>>>>>> yum install pacemaker-libs-devel
>>>>>>>
>>>>>>> # Now build
>>>>>>> cd ocfs2-tools
>>>>>>> ./configure
>>>>>>> make
>>>>>>>
>>>>>>> # This will likely fail. If it first fails complaining about
>>>>>>> xml/tree.h then you can do the following:
>>>>>>> CPPFLAGS='-I/usr/include/libxml2' ./configure
>>>>>>> make
>>>>>>>
>>>>>>> # It might complain again complaining about some AIS include files
>>>>>>> that
>>>>>>> are no
>>>>>>> # longer in the packages installed. That is OK. It should have built
>>>>>>> ocfs2_controld.cman,
>>>>>>> # so copy it where it needs to be:
>>>>>>>
>>>>>>> cp ocfs2_controld.cman /usr/sbin/
>>>>>>> scp ocfs2_controld.cman root at 172.16.170.6:/usr/sbin/
>>>>>>>
>>>>>>> # Now stop those two you started and start everything:
>>>>>>>
>>>>>>> service pacemaker stop
>>>>>>> service cman stop
>>>>>>>
>>>>>>> service cman start
>>>>>>> service o2cb start
>>>>>>> service pacemaker start
>>>>>>>
>>>>>>> 8. Create the shared shared file system on one node:
>>>>>>>
>>>>>>> mkfs.ocfs2 -L CTDBdemocommon --cluster-name ctdbdemo --cluster-stack
>>>>>>> ocfs2 -N 4 /dev/sdb
>>>>>>>
>>>>>>> 9. Mount it on both and ensure that you can create files/dirs on one
>>>>>>> node and see them on the other node.
>>>>>>>
>>>>>>> 10. Install ctdb and samba
>>>>>>>
>>>>>>> 11. Configure samba for the domain you want to join
>>>>>>>
>>>>>>> # Make sure you have clustering = yes and the other things you need.
>>>>>>>
>>>>>>> 12. Configure ctdb (/etc/sysconfig/ctdb) and make sure that you
>>>>>>> disable
>>>>>>> winbindd
>>>>>>>
>>>>>>> 13. Start ctdb on all nodes
>>>>>>>
>>>>>>> # You must have ctdb started so that the secrets file will get
>>>>>>> distributed
>>>>>>>
>>>>>>> 14. join the domain
>>>>>>>
>>>>>>> 15. Enable winbindd in the ctdb config
>>>>>>>
>>>>>>> 16. Restart ctdb on all nodes
>>>>>>>
>>>>>>> At this point you should be done. The steps you need might vary.
>>>>>>>
>>>>>>> I have limited time to help you with this.
>>>>>>>
>>>>>> OK, I have followed Richards 'howto' but using Debian 7.7 instead of
>>>>>> Centos,
>>>>>> I also used the standard Debian kernel. I have got up to step 9,
>>>>>> after a bit
>>>>>> of a battle and it all started so well. :-)
>>>>>>
>>>>>> Most of the required packages are available from the repos:
>>>>>>
>>>>>> apt-get install openais corosync pacemaker cman ocfs2-tools-cman
>>>>>> ocfs2-tools-pacemaker
>>>>>>
>>>>>> Unfortunately, it turned out that pacemaker is not built to use the
>>>>>> cman
>>>>>> stack, so I had to rebuild it
>>>>>>
>>>>>> Next problem, ccs and pcs are not available, so I had to download &
>>>>>> build
>>>>>> them, though even this was not without problems, ccs put
>>>>>> 'empty_cluster.conf' in the wrong place and pcs is hardwired to use
>>>>>> /usr/libexec
>>>>>>
>>>>>> Next problem 'o2cb' appears to be called 'o2cb_ctl' on Debian.
>>>>>>
>>>>>> Started cman, o2cb and pacemaker (first time round, this is where I
>>>>>> found
>>>>>> that pacemaker wouldn't work with cman)
>>>>>>
>>>>>> I then created the shared shared file system and mounted it on both
>>>>>> nodes
>>>>>
>>>>> OK, looks like you got the real stuff done.
>>>>>
>>>>>> At this point I have a shared cluster, but in a way that I cannot
>>>>>> see any
>>>>>> sane sysadmin using. Most of the software is heavily modified or not
>>>>>> available from the distros repos. I am going to have to stop and
>>>>>> think about
>>>>>> this and see if there is a Debian way of doing this, without modifying
>>>>>> anything or using anything that is not available from a repo.
>>>>
>>>>
>>>> OK, I am getting closer :-)
>>>>
>>>> I have got it working with just packages available from Debian repos,
>>>> apart from 'ccs', once I find  a replacement for this, I will move
>>>> onto ctdb & samba.
>>>>
>>>> Rowland
>>>>
>>>>> Indeed, I cannot imagine anyone using the approach I used for
>>>>> production as well. Having stuff that needs to be rebuilt is not a
>>>>> good idea, and it would be useful to get a minimal complete set of
>>>>> RPMs together and fix ocfs2-tools so that the correct things are there
>>>>> and the build works out what is needed.
>>>>>
>>>>> However, I only currently need the OCFS2 setup for testing purposes
>>>>> while we get our file system supporting FCNTL locks and so I have a
>>>>> reference to work with.
>>>>>
>>>>> Not sure if I have the time to fix anything.
>>>>>
>>>>
>>>
>>> OK, I officially give up! :-D
>>
>> Aw, c'mon Rowland. You're almost there! Just take corosync out of the mix
>> and use something else which _is_ documented instead;)
>>>
>>>
>>> I can get a two node cluster working on Debian 7.7 very easily, no
>>> compiling of software, but after this, it goes downhill fast. Whatever I
>>> do I cannot get ctdb to work, I feel a lot of this is down to the very
>>> small amount of documentation and what there is, seems to be biased
>>> towards redhat.
>>
>> Is Debian smbd built wth cluster? Ubuntu isn't.
>>>
>>>
>>> Searching the internet brings up very little and what I find is
>>> contradictory, also quite a lot of what I found would say something
>>> like, 'setup ctdb' but wouldn't explain how or why. Don't get me started
>>> on 'nodes', just what ipaddress should I use for each node?
>>
>> One addr. needs to be your domain; whatever subnet your current filer
>> uses. Connect that to the switch. The other, for the internal traffic is
>> another network. We use 192.168.1 for the domain and 192.168.0 for the
>> crossover.
>>  do I really
>>>
>>> need two network interfaces and why?
>>
>> No, but it's faster if you do and it saves having to route. Keep the
>> internal stuff away from the domain if you can would be our advice. With
>> only 1 cable, you can't
>>>
>>>
>>> So, I have come to the conclusion, I personally cannot setup Debian,
>>> ocf2s, corosync, ctdb and samba, because I cannot find any documentation
>>> that is complete and easily understandable. It is no good the people
>>> writing ctdb knowing how to set it up, they need to look at the
>>> documentation before they go much further.
>>
>> +1
>> I see that there has been some activity of late, prompted by some
>> searching questioning techniques. This is great to see.
>>>
>>>
>>> Sorry if this upsets anybody, it just how I feel at the moment.
>>
>> Meanwhile, if you want a no nonsense, documented 2 node cluster that just
>> works, just see our blog.
>>>
>>>
>> HTH
>>>
>>> Rowland
>>>
>>
> Hi Steve, I know that I can set it as you have done, I was trying to follow
> Richards setup, but using Debian instead of redhat, I just cannot get it to
> continue working once I bring CTDB into the picture and yes, Samba on Debian
> does have cluster support.
>
> There just doesn't seem to be full and easily readable documentation
> available, the type of documentation that not only tells you what to do, but
> also why. The documentation that is available seems to be biased towards
> redhat and expects a deeper knowledge of the subject than a normal user
> would have.
>
> So, as I said, I give up.

If I get some time over the Christmas break I might try to set that
up. At the moment I do not have the VMs available to try this because
my main machine already has 4 VMs running on it and I am not sure I
can afford the extra memory hit. Hmmm, maybe I can. I have 64GB ...
can you document for me the steps you took? I am not familiar with
Debian.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)