Setting up CTDB on OCFS2 and VMs ...

Thu Dec 11 11:32:31 MST 2014

I just tried building a single-node "cluster" on debian with ctdb.
I can check building a 4 node cluster next week when I am home from my travels.

To get ctdb running on ubuntu 14.10, as root:

1, Install the ctdb package:
apt-get install ctdb

2, create a missing directory
mkdir -p /var/lib/run/ctdb

3, remove the reclock file
vi /etc/default/ctdb
and comment out CTDB_RECOVERY_LOCK

4, create a nodes file
vi /etc/ctdb/nodes
and add the line   127.0.0.1

5, create a public addresses file
vi /etc/ctdb/public_addresses
and add the two lines
127.0.0.2/8 lo
127.0.0.3/8 lo

6, start ctdb
service ctdb start

then check everything looks fine with 'ctbb status' and 'tail
/var/log/ctdb/log.ctdb'

That will not really create a very interesting cluster, just one node,
two public addresses and all on loopback.
But this should at least verify that ctdbd will start and run.
Then you can just shut it down and edit
/etc/ctdb/nodes|public_addresses and make them more interesting.

I personally suggest never running anything smaller than 4 node
clusters for real data.

Please see
man ctdb
man ctdbd
less /etc/default/ctdb
http://ctdb.samba.org/configuring.html

it should contain most to get started with ctdb.

regards
ronnie sahlberg.

On Thu, Dec 11, 2014 at 1:03 PM, ronnie sahlberg
<ronniesahlberg at gmail.com> wrote:
> So you can not even get ctdbd to start ?
>
> I can help you setting it up.
> Have you looked at http://ctdb.samba.org/configuring.html    ?
>
> You really only need to set three(two) arguments for a basic ctdb setup :
>
> CTDB_NODES
> CTDB_RECOVERY_LOCK
> CTDB_PUBLIC_ADDRESSES
>
> You can skip/remove CTDB_RECOVERY_LOCK if you do not have a cluster
> filesystem that supports fcntl() locking.
> So maybe leave that out for now until Richards text on how to build
> the lock manager is finished.
> This just means that you do not have a split brain protection, but you
> can skip that as long until you plan to start running in production.
>
> CTDB_NODES is the file where you list the static ip address for every
> node in the cluster.
> Each node must have an identical version of this file.
> On big clusters, the recommendation is usually to run this on a
> separate private network that is dedicated for ctdb traffic.
> This is just to make sure that latency and recovery bandwidth is as
> good as possible for the ctdb traffic, by not having to compete with
> the i/o from tens of thousands of windows clients.
> But you don't NEED this to be a private network.
> If you have a small cluster and only light traffic you can just use a
> single network everything and thus have these addresses and the public
> addresses below all share the same interface/subnet.
>
>
> CTDB_PUBLIC_ADDRESSES
> Is a file that contains all the public addresses for the cluster.
> Each line contains one ip address/mask and associated interface.
> For small clusters, this file will usually be identical on all nodes
> in the cluster.
>
>
> And that should really be all there is to it to at least get ctdb working.
>
> If you configure this and start ctdb on all nodes,   what happens and
> what does 'ctdb status'  print ?
>
>
> Also see
> man ctdb
> and
> man ctdbd
>
> The two manpages contain a lot of good information.
>
> regards
> ronnie sahlberg
>
>
>
>
> On Thu, Dec 11, 2014 at 12:46 PM, Rowland Penny <repenny241155 at gmail.com> wrote:
>> On 11/12/14 17:28, steve wrote:
>>>
>>> On 11/12/14 17:47, Rowland Penny wrote:
>>>>
>>>> On 06/12/14 15:33, Rowland Penny wrote:
>>>>>
>>>>> On 06/12/14 15:24, Richard Sharpe wrote:
>>>>>>
>>>>>> On Sat, Dec 6, 2014 at 2:58 AM, Rowland Penny
>>>>>> <repenny241155 at gmail.com> wrote:
>>>>>>>
>>>>>>> On 04/12/14 18:08, Richard Sharpe wrote:
>>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Here are the steps I used, as far as I can remember them. Please
>>>>>>>> excuse any mistakes and be prepared to think for yourself when
>>>>>>>> following them.
>>>>>>>>
>>>>>>>> 1. Create two VirtualBox VMs with enough memory and disk for your
>>>>>>>> Linux Distro. I used CentOS 6.6 with 4GB and 20GB. (I actually
>>>>>>>> installed CentOS 6.3 and upgraded because I had the ISO handy.) You
>>>>>>>> will also need an extra interface on each VM for the clustering
>>>>>>>> private network. I set them to an internal type.
>>>>>>>>
>>>>>>>> 2. Because you will need a shared disk, create one:
>>>>>>>>
>>>>>>>> vboxmanage createhd --filename ~/VirtualBox\ VMs/SharedHD1 --size
>>>>>>>> 10240 --variant Fixed --format VDI # Creates a 10GB fixed sized disk
>>>>>>>> vboxmanage modifyhd 22ae1fcc-fda7-4e42-be9f-3b8bd7fc0c0e --type
>>>>>>>> shareable # Make it shareable.
>>>>>>>>
>>>>>>>> Note, for that second command use the UUID for your disk which you
>>>>>>>> can
>>>>>>>> find with:
>>>>>>>>
>>>>>>>> vboxmanage list hdds --brief
>>>>>>>>
>>>>>>>> Also, use the GUI to add the shared disk to both VMs.
>>>>>>>>
>>>>>>>> 3. Install the OS on each of the VMs.
>>>>>>>>
>>>>>>>> 4. I installed a bunch of clustering RPMs next:
>>>>>>>>
>>>>>>>> yum install openais corosync pacemaker-libs pacemaker-libs-devel gcc
>>>>>>>> corosync-devel openais-devel rpm-build e2fsprogs-devel libuuid-devel
>>>>>>>> git pygtk2 python-devel readline-devel clusterlib-devel redhat-lsb
>>>>>>>> sqlite-devel gnutls-devel byacc flex nss-devel
>>>>>>>>
>>>>>>>> It is not clear to me that openais was needed, for example
>>>>>>>>
>>>>>>>> 5. Next I installed Oracles UEK and ocfs2-tools
>>>>>>>>
>>>>>>>> wget http://public-yum.oracle.com/public-yum-ol6.repo
>>>>>>>> /etc/yum.repos.d
>>>>>>>> wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 -O
>>>>>>>> /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
>>>>>>>> yum install kernel-uek kernel-uek-devel
>>>>>>>> yum install ocfs2-tools
>>>>>>>> yum install openaislib-devel corosync-devel # It is not clear that I
>>>>>>>> needed to install the first
>>>>>>>>
>>>>>>>> echo 'KERNEL=="ocfs2_control", NAME="misc/ocfs2_control",
>>>>>>>> MODE="0660"'
>>>>>>>>>
>>>>>>>>> /etc/udev/rules.d/99-ocfs2_control.rules
>>>>>>>>
>>>>>>>> reboot # on each
>>>>>>>>
>>>>>>>> 6. Configure cman and pacemaker
>>>>>>>>
>>>>>>>> # configure corosync first
>>>>>>>> cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
>>>>>>>>
>>>>>>>> # Make sure that bindnetaddr is defined and points to your private
>>>>>>>> interface. I set it to 192.168.122.0
>>>>>>>>
>>>>>>>> # Make sure that the mcastaddr is defined. I used 239.255.1.1
>>>>>>>> # make sure that the mcastport is defined. I used 5405
>>>>>>>>
>>>>>>>> # Copy that file to the other node.
>>>>>>>> scp /etc/corosync/corosync.conf root at 172.16.170.6:/etc/corosync
>>>>>>>>
>>>>>>>> /etc/init.d/pacemaker stop  # Stop these in case they were running
>>>>>>>> /etc/init.d/corosync stop # Same here
>>>>>>>>
>>>>>>>> yum install ccs pcs
>>>>>>>>
>>>>>>>> # Create a cluster
>>>>>>>>
>>>>>>>> ccs -f /etc/cluster/cluster.conf --createcluster ctdbdemo
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addnode ocfs2-1
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addnode ocfs2-2
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addfencedev pcmk agent=fence_pcmk
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-1
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addmethod pcmk-redirect ocfs2-2
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-1
>>>>>>>> pcmk-redirect port=ocfs2-1
>>>>>>>> ccs -f /etc/cluster/cluster.conf --addfenceinst pcmk ocfs2-2
>>>>>>>> pcmk-redirect port=ocfs2-2
>>>>>>>>
>>>>>>>> # Copy the cluster config file to the other node:
>>>>>>>> scp /etc/cluster/cluster.conf root at 172.16.170.6:/etc/cluster
>>>>>>>>
>>>>>>>> #Now turn off NetworkManager:
>>>>>>>> chkconfig NetworkManager off
>>>>>>>> service NetworkManager stop
>>>>>>>>
>>>>>>>> # now start the cluster
>>>>>>>> service cman start
>>>>>>>> pcs property set stonith-enabled=false
>>>>>>>> service pacemaker start
>>>>>>>>
>>>>>>>> # Also start it on the other node(s).
>>>>>>>>
>>>>>>>> # Now check the status:
>>>>>>>>       [root at ocfs2-1 ~]# crm_mon -1
>>>>>>>>       Last updated: Thu Dec  4 09:40:16 2014
>>>>>>>>       Last change: Tue Dec  2 10:12:50 2014
>>>>>>>>       Stack: cman
>>>>>>>>       Current DC: ocfs2-2 - partition with quorum
>>>>>>>>       Version: 1.1.11-97629de
>>>>>>>>       2 Nodes configured
>>>>>>>>       0 Resources configured
>>>>>>>>
>>>>>>>>
>>>>>>>>       Online: [ ocfs2-1 ocfs2-2 ]
>>>>>>>>
>>>>>>>> If you do not see all the other nodes online, then you have to debug
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>> These are essentially the steps from here:
>>>>>>>> http://clusterlabs.org/quickstart-redhat.html
>>>>>>>>
>>>>>>>> 7. Configure the Oracle cluster
>>>>>>>>
>>>>>>>> o2cb add-cluster ctdbdemo
>>>>>>>> o2cb add-node --ip 192.168.122.10 --port --number 1 ctdbdemo ocfs2-1
>>>>>>>> o2cb add-node --ip 192.168.122.10 --port 7777 --number 1 ctdbdemo
>>>>>>>> ocfs2-1
>>>>>>>> o2cb add-node --ip 192.168.122.11 --port 7777 --number 2 ctdbdemo
>>>>>>>> ocfs2-2
>>>>>>>>
>>>>>>>> service o2cb configure # This step will fail claiming that it can't
>>>>>>>> find /sbin/ocfs2_controld.cman
>>>>>>>> #
>>>>>>>> # However, it does the important stuff.
>>>>>>>> #
>>>>>>>> # NOTE, during the configuration steps you MUST SELECT cman AS THE
>>>>>>>> CLUSTER STACK!
>>>>>>>> #
>>>>>>>>
>>>>>>>> 8. Find and install the ocfs2-tools git repos
>>>>>>>>
>>>>>>>> git clone git://oss.oracle.com/git/ocfs2-tools.git ocfs2-tools
>>>>>>>>
>>>>>>>> # install stuff needed
>>>>>>>> yum install libaio libaio-devel
>>>>>>>> yum install pacemaker-libs-devel
>>>>>>>>
>>>>>>>> # Now build
>>>>>>>> cd ocfs2-tools
>>>>>>>> ./configure
>>>>>>>> make
>>>>>>>>
>>>>>>>> # This will likely fail. If it first fails complaining about
>>>>>>>> xml/tree.h then you can do the following:
>>>>>>>> CPPFLAGS='-I/usr/include/libxml2' ./configure
>>>>>>>> make
>>>>>>>>
>>>>>>>> # It might complain again complaining about some AIS include files
>>>>>>>> that
>>>>>>>> are no
>>>>>>>> # longer in the packages installed. That is OK. It should have built
>>>>>>>> ocfs2_controld.cman,
>>>>>>>> # so copy it where it needs to be:
>>>>>>>>
>>>>>>>> cp ocfs2_controld.cman /usr/sbin/
>>>>>>>> scp ocfs2_controld.cman root at 172.16.170.6:/usr/sbin/
>>>>>>>>
>>>>>>>> # Now stop those two you started and start everything:
>>>>>>>>
>>>>>>>> service pacemaker stop
>>>>>>>> service cman stop
>>>>>>>>
>>>>>>>> service cman start
>>>>>>>> service o2cb start
>>>>>>>> service pacemaker start
>>>>>>>>
>>>>>>>> 8. Create the shared shared file system on one node:
>>>>>>>>
>>>>>>>> mkfs.ocfs2 -L CTDBdemocommon --cluster-name ctdbdemo --cluster-stack
>>>>>>>> ocfs2 -N 4 /dev/sdb
>>>>>>>>
>>>>>>>> 9. Mount it on both and ensure that you can create files/dirs on one
>>>>>>>> node and see them on the other node.
>>>>>>>>
>>>>>>>> 10. Install ctdb and samba
>>>>>>>>
>>>>>>>> 11. Configure samba for the domain you want to join
>>>>>>>>
>>>>>>>> # Make sure you have clustering = yes and the other things you need.
>>>>>>>>
>>>>>>>> 12. Configure ctdb (/etc/sysconfig/ctdb) and make sure that you
>>>>>>>> disable
>>>>>>>> winbindd
>>>>>>>>
>>>>>>>> 13. Start ctdb on all nodes
>>>>>>>>
>>>>>>>> # You must have ctdb started so that the secrets file will get
>>>>>>>> distributed
>>>>>>>>
>>>>>>>> 14. join the domain
>>>>>>>>
>>>>>>>> 15. Enable winbindd in the ctdb config
>>>>>>>>
>>>>>>>> 16. Restart ctdb on all nodes
>>>>>>>>
>>>>>>>> At this point you should be done. The steps you need might vary.
>>>>>>>>
>>>>>>>> I have limited time to help you with this.
>>>>>>>>
>>>>>>> OK, I have followed Richards 'howto' but using Debian 7.7 instead of
>>>>>>> Centos,
>>>>>>> I also used the standard Debian kernel. I have got up to step 9,
>>>>>>> after a bit
>>>>>>> of a battle and it all started so well. :-)
>>>>>>>
>>>>>>> Most of the required packages are available from the repos:
>>>>>>>
>>>>>>> apt-get install openais corosync pacemaker cman ocfs2-tools-cman
>>>>>>> ocfs2-tools-pacemaker
>>>>>>>
>>>>>>> Unfortunately, it turned out that pacemaker is not built to use the
>>>>>>> cman
>>>>>>> stack, so I had to rebuild it
>>>>>>>
>>>>>>> Next problem, ccs and pcs are not available, so I had to download &
>>>>>>> build
>>>>>>> them, though even this was not without problems, ccs put
>>>>>>> 'empty_cluster.conf' in the wrong place and pcs is hardwired to use
>>>>>>> /usr/libexec
>>>>>>>
>>>>>>> Next problem 'o2cb' appears to be called 'o2cb_ctl' on Debian.
>>>>>>>
>>>>>>> Started cman, o2cb and pacemaker (first time round, this is where I
>>>>>>> found
>>>>>>> that pacemaker wouldn't work with cman)
>>>>>>>
>>>>>>> I then created the shared shared file system and mounted it on both
>>>>>>> nodes
>>>>>>
>>>>>> OK, looks like you got the real stuff done.
>>>>>>
>>>>>>> At this point I have a shared cluster, but in a way that I cannot
>>>>>>> see any
>>>>>>> sane sysadmin using. Most of the software is heavily modified or not
>>>>>>> available from the distros repos. I am going to have to stop and
>>>>>>> think about
>>>>>>> this and see if there is a Debian way of doing this, without modifying
>>>>>>> anything or using anything that is not available from a repo.
>>>>>
>>>>>
>>>>> OK, I am getting closer :-)
>>>>>
>>>>> I have got it working with just packages available from Debian repos,
>>>>> apart from 'ccs', once I find  a replacement for this, I will move
>>>>> onto ctdb & samba.
>>>>>
>>>>> Rowland
>>>>>
>>>>>> Indeed, I cannot imagine anyone using the approach I used for
>>>>>> production as well. Having stuff that needs to be rebuilt is not a
>>>>>> good idea, and it would be useful to get a minimal complete set of
>>>>>> RPMs together and fix ocfs2-tools so that the correct things are there
>>>>>> and the build works out what is needed.
>>>>>>
>>>>>> However, I only currently need the OCFS2 setup for testing purposes
>>>>>> while we get our file system supporting FCNTL locks and so I have a
>>>>>> reference to work with.
>>>>>>
>>>>>> Not sure if I have the time to fix anything.
>>>>>>
>>>>>
>>>>
>>>> OK, I officially give up! :-D
>>>
>>> Aw, c'mon Rowland. You're almost there! Just take corosync out of the mix
>>> and use something else which _is_ documented instead;)
>>>>
>>>>
>>>> I can get a two node cluster working on Debian 7.7 very easily, no
>>>> compiling of software, but after this, it goes downhill fast. Whatever I
>>>> do I cannot get ctdb to work, I feel a lot of this is down to the very
>>>> small amount of documentation and what there is, seems to be biased
>>>> towards redhat.
>>>
>>> Is Debian smbd built wth cluster? Ubuntu isn't.
>>>>
>>>>
>>>> Searching the internet brings up very little and what I find is
>>>> contradictory, also quite a lot of what I found would say something
>>>> like, 'setup ctdb' but wouldn't explain how or why. Don't get me started
>>>> on 'nodes', just what ipaddress should I use for each node?
>>>
>>> One addr. needs to be your domain; whatever subnet your current filer
>>> uses. Connect that to the switch. The other, for the internal traffic is
>>> another network. We use 192.168.1 for the domain and 192.168.0 for the
>>> crossover.
>>>  do I really
>>>>
>>>> need two network interfaces and why?
>>>
>>> No, but it's faster if you do and it saves having to route. Keep the
>>> internal stuff away from the domain if you can would be our advice. With
>>> only 1 cable, you can't
>>>>
>>>>
>>>> So, I have come to the conclusion, I personally cannot setup Debian,
>>>> ocf2s, corosync, ctdb and samba, because I cannot find any documentation
>>>> that is complete and easily understandable. It is no good the people
>>>> writing ctdb knowing how to set it up, they need to look at the
>>>> documentation before they go much further.
>>>
>>> +1
>>> I see that there has been some activity of late, prompted by some
>>> searching questioning techniques. This is great to see.
>>>>
>>>>
>>>> Sorry if this upsets anybody, it just how I feel at the moment.
>>>
>>> Meanwhile, if you want a no nonsense, documented 2 node cluster that just
>>> works, just see our blog.
>>>>
>>>>
>>> HTH
>>>>
>>>> Rowland
>>>>
>>>
>> Hi Steve, I know that I can set it as you have done, I was trying to follow
>> Richards setup, but using Debian instead of redhat, I just cannot get it to
>> continue working once I bring CTDB into the picture and yes, Samba on Debian
>> does have cluster support.
>>
>> There just doesn't seem to be full and easily readable documentation
>> available, the type of documentation that not only tells you what to do, but
>> also why. The documentation that is available seems to be biased towards
>> redhat and expects a deeper knowledge of the subject than a normal user
>> would have.
>>
>> So, as I said, I give up.
>>
>> Rowland