Setting up CTDB on OCFS2 and VMs ...

Tue Dec 16 00:53:42 MST 2014

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Rowland,

did you see that you have som Problems with IPs on node 1?
2014/12/15 16:32:28.300370 [recoverd: 2497]: Failed to find node to
cover ip 192.168.0.9
2014/12/15 16:32:28.300412 [recoverd: 2497]: Failed to find node to
cover ip 192.168.0.8
I also had some problems with IP and nameresolutions at the beginning.
After I solved that problem everything was fine.

Stefan

Am 15.12.14 um 17:49 schrieb Rowland Penny:
> On 15/12/14 14:14, Stefan Kania wrote: Hi Rowland,
> 
> Am 15.12.14 um 14:25 schrieb Rowland Penny:
>>>> On 13/12/14 13:55, Michael Adam wrote:
>>>>> On 2014-12-13 at 12:31 +0000, Rowland Penny wrote:
>>>>>> OK, I now have a single node up and running as per the 
>>>>>> instructions provided by Ronnie.
>>>>> Yay!
>>>>> 
>>>>>> I just have a few questions:
>>>>>> 
>>>>>> there is this in the ctdb log
>>>>>> 
>>>>>> 2014/12/13 11:52:43.522708 [ 5740]: Set runstate to INIT
>>>>>> (1) 2014/12/13 11:52:43.540992 [ 5740]: 00.ctdb: awk:
>>>>>> line 2: function gensub never defined 2014/12/13
>>>>>> 11:52:43.543178 [ 5740]: 00.ctdb: awk: line 2: function
>>>>>> gensub never defined 2014/12/13 11:52:43.545354 [ 5740]:
>>>>>> 00.ctdb: awk: line 2: function gensub never defined
>>>>> In Debian, there are several possible awk versions that
>>>>> provide awk: at least: mawk, gawk, original-awk. Which is
>>>>> chosen if multiple are installed, depends on the
>>>>> alternatives-mechanism: update-alternatives --display awk
>>>>> update-alternatives --edit awk
>>>>> 
>>>>> A quick web search has reveiled that only the gawk (gnu
>>>>> awk) variant might feature the needed gensub function.
>>>>> 
>>>>> Maybe we should change "awk" to "gawk" in our scripts and 
>>>>> packages would need to adapt their dependencies.
>>>>> 
>>>>> 
>>>>>> 2014/12/13 11:52:56.931393 [recoverd: 5887]: We are
>>>>>> still serving a public IP '127.0.0.3' that we should not
>>>>>> be serving. Removing it 2014/12/13 11:52:56.931536 [
>>>>>> 5740]: Could not find which interface the ip address is
>>>>>> hosted on. can not release it 2014/12/13 11:52:56.931648
>>>>>> [recoverd: 5887]: We are still serving a public IP
>>>>>> '127.0.0.2' that we should not be serving. Removing it
>>>>>> 
>>>>>> The above three lines are there 4 times
>>>>> I guess this will not be the case any more, when you move
>>>>> to a more realistic setup where you don't use loopback for
>>>>> hosting nodes internal and public addresses, but for a
>>>>> start that is ok.
>>>>> 
>>>>>> the final 4 lines are:
>>>>>> 
>>>>>> 2014/12/13 11:53:02.982441 [ 5740]: monitor event OK -
>>>>>> node re-enabled 2014/12/13 11:53:02.982480 [ 5740]: Node
>>>>>> became HEALTHY. Ask recovery master 0 to perform ip
>>>>>> reallocation 2014/12/13 11:53:02.982733 [recoverd: 5887]:
>>>>>> Node 0 has changed flags - now 0x0  was 0x2 2014/12/13
>>>>>> 11:53:02.983266 [recoverd: 5887]: Takeover run starting
>>>>>> 2014/12/13 11:53:03.046859 [recoverd: 5887]: Takeover run
>>>>>> completed successfully
>>>>>> 
>>>>>> ctdb status shows:
>>>>>> 
>>>>>> Number of nodes:1 pnn:0 127.0.0.1        OK (THIS NODE) 
>>>>>> Generation:740799152 Size:1 hash:0 lmaster:0 Recovery 
>>>>>> mode:NORMAL (0) Recovery master:0
>>>>> Great!
>>>>> 
>>>>>> Now I know it works, I just have to pull it all
>>>>>> together.
>>>>> Right. Next step: take a "real" ethernet interface and
>>>>> first use that for nodes address. You can even start here
>>>>> with a single node.
>>>>> 
>>>>> You can also go towards more realistic clusters in two
>>>>> steps: First no public addresse, only the nodes file. That
>>>>> is the core of a ctdb cluster. Then you can go towards
>>>>> cluster-resource management and add public addresse and
>>>>> also CTDB_MANAGES_SAMBA and friends.
>>>>> 
>>>>> One further note: Virtual machines or even containers (lxc
>>>>> or docker) are awesome for setting up such clusters for
>>>>> learing and testing. I use that for development myselves.
>>>>> 
>>>>> And here is one (imho) very neat trick: If you use lxc
>>>>> containers (or docker can probably also do that), you can
>>>>> completely take the complexity of having to set up a
>>>>> cluster file system out of the equation: You can just bind
>>>>> mount a directory of the host file system into the node
>>>>> containers' root file systems by the lxc fstab file.
>>>>> Thereby you have a posix-file system that is shared between
>>>>> the nodes and you can use that as cluster FS.
>>>>> 
>>>>> This way, you can concentrate on ctdb and samba immediately
>>>>> until you are comfortable with that.
>>>>> 
>>>>> I wanted at some point to provide a mechanism to set such a
>>>>> thing up automatically, by just providing some config
>>>>> files. Maybe I'll investigate the vagrant+puppet approach
>>>>> that Ralph Böhme has recently posted in this or a related
>>>>> thread...
>>>>> 
>>>>> Cheers - Michael
>>>>> 
>>>> Getting closer :-)
>>>> 
>>>> I now have two ctdb nodes up and running:
>>>> 
>>>> root at cluster1:~# ctdb status Number of nodes:3 (including 1
>>>> deleted nodes) pnn:1 192.168.1.10     OK (THIS NODE) pnn:2
>>>> 192.168.1.11 OK Generation:1073761636 Size:2 hash:0 lmaster:1
>>>> hash:1 lmaster:2 Recovery mode:NORMAL (0) Recovery master:1
>>>> 
>>>> This is with CTDB_RECOVERY_LOCK turned off, if I turn it on,
>>>> the nodes go unhealthy. I am putting the lockfile on the
>>>> shared cluster, should I be putting it somewhere else, it
>>>> says at the top of /etc/default/ctdb :
>>>> 
>>>> # Shared recovery lock file to avoid split brain.  No
>>>> default. # # Do NOT run CTDB without a recovery lock file
>>>> unless you know exactly # what you are doing. 
>>>> #CTDB_RECOVERY_LOCK=/some/place/on/shared/storage
>>>> 
>>>> As I don't know what I am doing, I need to run the recovery 
>>>> lockfile :-D
>>>> 
>>>> Rowland
> Are there any errors in /var/log/log.ctdb? Which version of ctdb
> you are using? And could you post your /etc/sysconfig/ctdb file?
> 
> 
> Regards
> 
> Stefan
> 
> 
> 
> Well I could answer: there is nothing in /var/log/log.ctdb,
> Version 2.5.3 and I do not have /etc/sysconfig/ctdb :-D
> 
> But instead, I will attach a tarball containing
> /var/log/ctdb/log.ctdb from both nodes, it also contains
> /etc/default/ctdb :-)
> 
> Rowland
> 

- -- 
Stefan Kania
Landweg 13
25693 St. Michaelisdonn

Signieren jeder E-Mail hilft Spam zu reduzieren. Signieren Sie ihre
E-Mail. Weiter Informationen unter http://www.gnupg.org

Mein Schlüssel liegt auf

hkp://subkeys.pgp.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.16 (Darwin)

iEYEARECAAYFAlSP5QYACgkQ2JOGcNAHDTauVgCeNrWaRZCWsCKNe5f0Rb7l3yqH
ypoAoLtO4O6dxuaPltRHFXB+aSoR29EB
=dNV4
-----END PGP SIGNATURE-----