[Samba] CTDB+GFS2+CMAN. clean_start="0" or clean_start="1"?

[Michael Adam]

Yauheni Labko wrote:
> Thank you for the answer, Michael.
> 
> As far as I understood clean_star="1" is absolutely ok for GFS/GFS2?

Sorry, I am not an expert in GFS settings. (But read on...)

> CTDB is not going to work without Red Hat Cluster manage. CMAN starts 
> dlm_controld and gfs_controld. ccsd handles node-to-node communication.

Well GFS needs the cman processes, so CTDB needs them, too.
But CTDB only uses one lock file in the cluster file system.
Apart from that, the CTDB daemons communicate with each other
via tcp all on their own.

> I think GPFS has the similar manager like CMAN. The clean_start="1" is the only 
> setting which can provide the necessary access to the GFS/GFS2 partitions as 
> CTDB required. Correct me if I'm wrong.

Sorry again. CTDB is completely ignorant with respect to GFS or
CMAN configuration options. It only needs a cluster file system
that supports POSIX fcntl() byte range locks. CTDB basicall treats
the file system as a black box.
So CTDB does not care about the value "clean_start" as such. Just make
sure you don't sure that you don't start ctdbd before the cman
stuff is up and running and the  GFS file system is mounted.

> Btw, i thought OCFS2 is not ready to use with CTDB due to the lacks of some 
> features. This was primary reason why I started  with GFS.

OCFS2 was lacking support of POSIX fcntl byte range locks (which
are required to run ctdb) until recently. But this has changed!
I have not tried it myself, but I think Jim McDonough
(jmcd at samba.org, I have added him to Cc) might be able to give
you some details (versions and such).

> I left manual fencing for testing only. I was going to use iLO in production.

OK.

Hope this somewhat helps... :-)

Cheers - Michael

> Yauheni Labko (Eugene Lobko)
> Junior System Administrator
> Chapdelaine & Co.
> (212)208-9150
> 
> > CTDB is pretty ignorant of CMAN as such.
> > It just relies on a cluster file system, like GFS2.
> 
> 
> > So you should only start ctdbd when the cluster is up
> > and the gfs2 file system is mounted. I think you should
> > not start ctdbd as a cluster service managed by cman,
> > since ctdbd can be considered a cluster manager for
> > certain services (like samba...) itself. Apart from
> > that, ctdb should be considered pretty much independent
> > of the red hat cluster manager.
> >
> > CTDB needs a file in the cluster file system, the
> > recovery lock file. The location of this file (or a
> > directory, in which such a file can be created) should
> > be specified in the CTDB_RECOVERY_LOCK=... setting
> > in /etc/sysconfig/ctdb.
> >
> > At a glance, your cluster.conf looks sane, but
> > I think manual fencing can be a real problem with
> > cman.
> >
> > GPFS is very well tested with ctdb.
> > I think there are many people testing ctdb with gfs2.
> > I have heard positive feedback of people using ctdb
> > with GlusterFS and lustre (and recently with ocfs2).
> >
> > You might want to join the #ctdb irc channel on freenode.
> > There are ususally some people around with more expertise
> > in gfs2 than me.
> >
> > Cheers - Michael
> >
> > Yauheni Labko wrote:
> > > Hi everybody,
> > >
> > > I have tested CTDB+GFS2+CMAN under Debian. It works good but I do not
> > > understand some points.
> > > It is possible to run the CTDB defining it under services section in
> > > cluster.conf but running it on the second node shuts down the process at
> > > the first one. My CTDB configuration implies 2 active-active nodes.
> > >
> > > Does CTDB care if the node starts with clean_start="0" or
> > > clean_start="1"? man fenced says this is a safe way especially during
> > > startup because it prevents a data corruption if a node was dead for some
> > > reason. From my understanding CTDB uses CMAN only as "module" to get
> > > access to gfs/gfs2 partitions. Or maybe it is better to look at GPFS and
> > > LustreFS?
> > >
> > > Could anybody show the working configuration of cluster.conf for
> > > CTDB+GFS2+CMAN?
> > >
> > > I used the following cluster.conf and ctd conf:
> > >
> > > <?xml version="1.0"?>
> > > <cluster name="smb-cluster" config_version="8">
> > >   <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
> > >   <cman expected_votes="1" two_node="1"/>
> > >   <cman cluster_id="101"/>
> > >   <clusternodes>
> > >     <clusternode name="smb01" votes="1" nodeid="1">
> > >       <fence>
> > >         <!-- Handle fencing manually -->
> > >         <method name="human">
> > >           <device name="human" nodename="smb01"/>
> > >         </method>
> > >       </fence>
> > >     </clusternode>
> > >     <clusternode name="smb02" votes="1" nodeid="2">
> > >       <fence>
> > >         <!-- Handle fencing manually -->
> > >         <method name="human">
> > >           <device name="human" nodename="smb02"/>
> > >         </method>
> > >       </fence>
> > >     </clusternode>
> > >   </clusternodes>
> > >   <fencedevices>
> > >     <!-- Define manual fencing -->
> > >     <fencedevice name="human" agent="fence_manual"/>
> > >     <!-- Define ilo fencing -->
> > >     <fencedevice name="ilo" agent="fence_ilo" login="admin"
> > > password="foo"/> </fencedevices>
> > > </cluster>
> > >
> > > # Options to ctdbd. This is read by /etc/init.d/ctdb
> > > CTDB_RECOVERY_LOCK="/smb-ctdb/.ctdb_locking"
> > > CTDB_PUBLIC_INTERFACE=eth2
> > > CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
> > > CTDB_MANAGES_SAMBA=yes
> > > CTDB_INIT_STYLE=ubuntu
> > > CTDB_NODES=/etc/ctdb/nodes
> > > CTDB_NOTIFY_SCRIPT=/etc/ctdb/notify.sh
> > > CTDB_DBDIR=/var/ctdb
> > > CTDB_DBDIR_PERSISTENT=/var/ctdb/persistent
> > > CTDB_SOCKET=/tmp/ctdb.socket
> > > CTDB_LOGFILE=/var/log/ctdb.log
> > > CTDB_DEBUGLEVEL=2
> > >
> > > Yauheni Labko (Eugene Lobko)
> > > Junior System Administrator
> > > Chapdelaine & Co.
> > > (212)208-9150
> 
> 
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 206 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba/attachments/20090811/46a8888d/attachment.pgp>