ctdb relock file issues with glusterfs
patrick medina
pgmedinajr at gmail.com
Tue Oct 9 10:06:44 MDT 2012
Our test lab just went semi production so I built a replica on another
stack of servers we have.
Results are the same unfortunately, but here are the log files from both
clients running ctdb.
Client 1 is running ERR=4, client 2 is running ERR=2
Cheers,
Gilbert
On Tue, Oct 9, 2012 at 5:20 AM, Martin Schwenke <martin at meltin.net> wrote:
> On Tue, 9 Oct 2012 16:32:12 +1100, Amitay Isaacs <amitay at gmail.com>
> wrote:
>
> > On Tue, Oct 9, 2012 at 1:55 PM, patrick medina <pgmedinajr at gmail.com>
> wrote:
> > > Howdy samba folks,
> > >
> > > I've been running into a lot of issues lately with ctdb's re-lock file
> and
> > > glusterfs as the shared storage. When I started, I could get one or
> the
> > > other node to become healthy, but at least one would complain it could
> not
> > > lock the re-lock file. Nowi'm at the point where neither node will
> become
> > > healthy and stay in a recovery loop. Just to be sure it was the
> re-lock
> > > file, I commented it out in the config and both nodes became healthy.
>
> > What version of CTDB are you using? Can you attach the log file where
> > you notice CTDB is continuously going in recovery? It would be useful
> > to get log files from all the nodes.
>
> Michael Adam and I took a look at this on the weekend. Gilbert sent me
> some logs and this was happening:
>
> ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
>
> That seems to indicate that locking isn't working as expected...
>
> peace & happiness,
> martin
>
-------------- next part --------------
2012/10/05 15:35:32.482765 [27401]: CTDB starting on node
2012/10/05 15:35:32.498562 [27403]: Starting CTDBD as pid : 27403
2012/10/05 15:35:32.498631 [27403]: Set scheduler to SCHED_FIFO
2012/10/05 15:35:32.498809 [27403]: ctdb chose network address 10.100.55.205:4379 pnn 1
2012/10/05 15:35:32.505202 [27403]: server/eventscript.c:754 Starting eventscript init
2012/10/05 15:35:32.769780 [27403]: server/eventscript.c:485 Eventscript init finished with state 0
2012/10/05 15:35:32.770329 [27403]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface 'eth0' refs[0])
2012/10/05 15:35:32.770556 [27403]: Keepalive monitoring has been started
2012/10/05 15:35:32.770569 [27403]: Monitoring has been started
2012/10/05 15:35:32.770615 [27403]: server/eventscript.c:754 Starting eventscript setup
2012/10/05 15:35:32.866831 [recoverd:27474]: monitor_cluster starting
2012/10/05 15:35:32.873975 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=17870283321406128128
2012/10/05 15:35:32.874046 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=17365880163140632576
2012/10/05 15:35:32.874082 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=17582052945254416384
2012/10/05 15:35:32.874113 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=17942340915444056064
2012/10/05 15:35:32.874143 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=17798225727368200192
2012/10/05 15:35:32.874173 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=18014398509481984000
2012/10/05 15:35:32.874211 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=18086456103519911936
2012/10/05 15:35:32.874238 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=18158513697557839872
2012/10/05 15:35:32.874266 [27403]: server/ctdb_daemon.c:187 Registered message handler for srvid=17654110539292344320
2012/10/05 15:35:32.874570 [recoverd:27474]: server/ctdb_recoverd.c:3029 Initial recovery master set - forcing election
2012/10/05 15:35:32.874619 [recoverd:27474]: server/ctdb_recoverd.c:2160 Force an election
2012/10/05 15:35:32.874663 [27403]: Freeze priority 1
2012/10/05 15:35:32.874706 [27403]: Freeze priority 2
2012/10/05 15:35:32.874738 [27403]: Freeze priority 3
2012/10/05 15:35:32.874815 [recoverd:27474]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/05 15:35:33.078212 [27403]: server/eventscript.c:485 Eventscript setup finished with state 0
2012/10/05 15:35:33.771484 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:34.772547 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:35.773589 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:36.379695 [recoverd:27474]: server/ctdb_recoverd.c:913 Election timed out
2012/10/05 15:35:36.379997 [recoverd:27474]: Recmaster node 0 is disconnected. Force reelection
2012/10/05 15:35:36.380010 [recoverd:27474]: server/ctdb_recoverd.c:2160 Force an election
2012/10/05 15:35:36.380043 [27403]: Freeze priority 1
2012/10/05 15:35:36.380080 [27403]: Freeze priority 2
2012/10/05 15:35:36.380112 [27403]: Freeze priority 3
2012/10/05 15:35:36.380175 [recoverd:27474]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/05 15:35:36.774165 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:37.771275 [27403]: 10.100.55.205:4379: connected to 10.100.55.209:4379 - 1 connected
2012/10/05 15:35:37.774762 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:38.775734 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:39.776826 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:39.885423 [recoverd:27474]: server/ctdb_recoverd.c:913 Election timed out
2012/10/05 15:35:39.892796 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:39.892856 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:40.759223 [27403]: Freeze priority 1
2012/10/05 15:35:40.763758 [27403]: Freeze priority 2
2012/10/05 15:35:40.764397 [27403]: Freeze priority 3
2012/10/05 15:35:40.777910 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:40.782267 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:40.782280 [27403]: Monitoring has been disabled
2012/10/05 15:35:40.782289 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:40.861680 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:40.893872 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:40.893952 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:41.408871 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:41.409395 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:41.410150 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:41.411671 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:41.412154 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:41.413166 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:41.413188 [27403]: Thawing priority 1
2012/10/05 15:35:41.413195 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:41.413220 [27403]: Thawing priority 2
2012/10/05 15:35:41.413228 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:41.413243 [27403]: Thawing priority 3
2012/10/05 15:35:41.413250 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:41.459674 [set_recmode:27612]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:41.459792 [set_recmode:27612]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:41.682382 [27403]: Freeze priority 1
2012/10/05 15:35:41.683106 [27403]: Freeze priority 2
2012/10/05 15:35:41.684006 [27403]: Freeze priority 3
2012/10/05 15:35:41.685276 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:41.685325 [27403]: Monitoring has been disabled
2012/10/05 15:35:41.685335 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:41.750573 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:41.772722 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:41.773058 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:41.773668 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:41.774638 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:41.774925 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:41.775627 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:41.775645 [27403]: Thawing priority 1
2012/10/05 15:35:41.775651 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:41.775670 [27403]: Thawing priority 2
2012/10/05 15:35:41.775676 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:41.775687 [27403]: Thawing priority 3
2012/10/05 15:35:41.775692 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:41.778448 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:41.778527 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:41.787981 [set_recmode:27678]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:41.788054 [set_recmode:27678]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:41.890218 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:41.890295 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:42.686189 [27403]: Freeze priority 1
2012/10/05 15:35:42.687334 [27403]: Freeze priority 2
2012/10/05 15:35:42.688316 [27403]: Freeze priority 3
2012/10/05 15:35:42.689745 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:42.689808 [27403]: Monitoring has been disabled
2012/10/05 15:35:42.689818 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:42.752988 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:42.753534 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:42.753905 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:42.754502 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:42.755566 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:42.755866 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:42.756732 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:42.756751 [27403]: Thawing priority 1
2012/10/05 15:35:42.756758 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:42.756775 [27403]: Thawing priority 2
2012/10/05 15:35:42.756781 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:42.756791 [27403]: Thawing priority 3
2012/10/05 15:35:42.756797 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:42.779026 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:42.779104 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:42.794034 [set_recmode:27744]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:42.794110 [set_recmode:27744]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:42.892267 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:42.892345 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:43.707517 [27403]: Freeze priority 1
2012/10/05 15:35:43.708290 [27403]: Freeze priority 2
2012/10/05 15:35:43.708970 [27403]: Freeze priority 3
2012/10/05 15:35:43.710102 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:43.710138 [27403]: Monitoring has been disabled
2012/10/05 15:35:43.710147 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:43.780534 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:43.780570 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:43.792536 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:43.793117 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:43.793434 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:43.794004 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:43.795191 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:43.795501 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:43.796455 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:43.796474 [27403]: Thawing priority 1
2012/10/05 15:35:43.796481 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:43.796497 [27403]: Thawing priority 2
2012/10/05 15:35:43.796504 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:43.796528 [27403]: Thawing priority 3
2012/10/05 15:35:43.796536 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:43.839600 [set_recmode:27810]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:43.839670 [set_recmode:27810]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:43.894312 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:43.894393 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:44.688412 [27403]: Freeze priority 1
2012/10/05 15:35:44.689176 [27403]: Freeze priority 2
2012/10/05 15:35:44.689866 [27403]: Freeze priority 3
2012/10/05 15:35:44.690994 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:44.691023 [27403]: Monitoring has been disabled
2012/10/05 15:35:44.691039 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:44.750430 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:44.751889 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:44.752265 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:44.752999 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:44.754260 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:44.754636 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:44.755447 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:44.755465 [27403]: Thawing priority 1
2012/10/05 15:35:44.755472 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:44.755488 [27403]: Thawing priority 2
2012/10/05 15:35:44.755495 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:44.755505 [27403]: Thawing priority 3
2012/10/05 15:35:44.755512 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:44.769224 [set_recmode:27876]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:44.769283 [set_recmode:27876]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:44.781691 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:44.781714 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:44.896301 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:44.896376 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:45.690872 [27403]: Freeze priority 1
2012/10/05 15:35:45.691571 [27403]: Freeze priority 2
2012/10/05 15:35:45.692216 [27403]: Freeze priority 3
2012/10/05 15:35:45.693218 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:45.693252 [27403]: Monitoring has been disabled
2012/10/05 15:35:45.693261 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:45.742973 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:45.766153 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:45.766593 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:45.767362 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:45.768635 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:45.769005 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:45.769924 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:45.769953 [27403]: Thawing priority 1
2012/10/05 15:35:45.769960 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:45.769980 [27403]: Thawing priority 2
2012/10/05 15:35:45.769987 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:45.769998 [27403]: Thawing priority 3
2012/10/05 15:35:45.770004 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:45.782793 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:45.782825 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:45.811984 [set_recmode:27942]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:45.812039 [set_recmode:27942]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:45.898400 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:45.898479 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:46.697740 [27403]: Freeze priority 1
2012/10/05 15:35:46.698515 [27403]: Freeze priority 2
2012/10/05 15:35:46.699180 [27403]: Freeze priority 3
2012/10/05 15:35:46.700297 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:46.700328 [27403]: Monitoring has been disabled
2012/10/05 15:35:46.700343 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:46.773213 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:46.773806 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:46.774160 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:46.774755 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:46.775893 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:46.776206 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:46.777079 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:46.777098 [27403]: Thawing priority 1
2012/10/05 15:35:46.777105 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:46.777119 [27403]: Thawing priority 2
2012/10/05 15:35:46.777126 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:46.777135 [27403]: Thawing priority 3
2012/10/05 15:35:46.777141 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:46.783808 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:46.783860 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:46.900555 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:46.900648 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:47.077936 [set_recmode:28008]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:47.078010 [set_recmode:28008]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:47.707776 [27403]: Freeze priority 1
2012/10/05 15:35:47.708541 [27403]: Freeze priority 2
2012/10/05 15:35:47.709223 [27403]: Freeze priority 3
2012/10/05 15:35:47.710351 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:47.710379 [27403]: Monitoring has been disabled
2012/10/05 15:35:47.710389 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:47.768454 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:47.769036 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:47.769371 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:47.770030 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:47.771279 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:47.771655 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:47.772552 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:47.772577 [27403]: Thawing priority 1
2012/10/05 15:35:47.772584 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:47.772603 [27403]: Thawing priority 2
2012/10/05 15:35:47.772609 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:47.772619 [27403]: Thawing priority 3
2012/10/05 15:35:47.772625 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:47.784870 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:47.784913 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:47.790948 [set_recmode:28074]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:47.791023 [set_recmode:28074]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:47.902636 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:47.902929 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:48.695450 [27403]: Freeze priority 1
2012/10/05 15:35:48.696222 [27403]: Freeze priority 2
2012/10/05 15:35:48.696926 [27403]: Freeze priority 3
2012/10/05 15:35:48.698016 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:48.698044 [27403]: Monitoring has been disabled
2012/10/05 15:35:48.698053 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:48.748396 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:48.768752 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:48.769117 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:48.769679 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:48.770817 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:48.771191 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:48.771995 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:48.772017 [27403]: Thawing priority 1
2012/10/05 15:35:48.772025 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:48.772044 [27403]: Thawing priority 2
2012/10/05 15:35:48.772051 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:48.772061 [27403]: Thawing priority 3
2012/10/05 15:35:48.772068 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:48.785954 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:48.785988 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:48.801415 [set_recmode:28140]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:48.801456 [set_recmode:28140]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:48.904644 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:48.904721 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:49.703154 [27403]: Freeze priority 1
2012/10/05 15:35:49.703918 [27403]: Freeze priority 2
2012/10/05 15:35:49.704572 [27403]: Freeze priority 3
2012/10/05 15:35:49.705690 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:49.705719 [27403]: Monitoring has been disabled
2012/10/05 15:35:49.705728 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:49.752976 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:49.754372 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:49.754722 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:49.755365 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:49.756496 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:49.756809 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:49.757597 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:49.757621 [27403]: Thawing priority 1
2012/10/05 15:35:49.757628 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:49.757644 [27403]: Thawing priority 2
2012/10/05 15:35:49.757651 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:49.757661 [27403]: Thawing priority 3
2012/10/05 15:35:49.757668 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:49.775638 [set_recmode:28206]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:49.775704 [set_recmode:28206]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:49.787046 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:49.787104 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:49.906802 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:49.906879 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:50.713638 [27403]: Freeze priority 1
2012/10/05 15:35:50.714502 [27403]: Freeze priority 2
2012/10/05 15:35:50.715315 [27403]: Freeze priority 3
2012/10/05 15:35:50.716483 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:50.716527 [27403]: Monitoring has been disabled
2012/10/05 15:35:50.716537 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:50.763275 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:50.778942 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:50.779333 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:50.780003 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:50.781128 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:50.781448 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:50.782345 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:50.782366 [27403]: Thawing priority 1
2012/10/05 15:35:50.782372 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:50.782391 [27403]: Thawing priority 2
2012/10/05 15:35:50.782397 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:50.782406 [27403]: Thawing priority 3
2012/10/05 15:35:50.782412 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:50.788117 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:50.788145 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:50.825500 [set_recmode:28272]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:50.825568 [set_recmode:28272]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:50.908798 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:50.908875 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:51.700241 [27403]: Freeze priority 1
2012/10/05 15:35:51.700999 [27403]: Freeze priority 2
2012/10/05 15:35:51.701689 [27403]: Freeze priority 3
2012/10/05 15:35:51.702842 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:51.702873 [27403]: Monitoring has been disabled
2012/10/05 15:35:51.702882 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:51.776050 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:51.776572 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:51.776973 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:51.777507 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:51.778522 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:51.778829 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:51.779680 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:51.779698 [27403]: Thawing priority 1
2012/10/05 15:35:51.779705 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:51.779718 [27403]: Thawing priority 2
2012/10/05 15:35:51.779725 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:51.779734 [27403]: Thawing priority 3
2012/10/05 15:35:51.779740 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:51.789609 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:51.789662 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:51.792210 [set_recmode:28338]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:51.792248 [set_recmode:28338]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:51.910753 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:51.910827 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:52.707534 [27403]: Freeze priority 1
2012/10/05 15:35:52.708328 [27403]: Freeze priority 2
2012/10/05 15:35:52.709031 [27403]: Freeze priority 3
2012/10/05 15:35:52.710179 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:52.710209 [27403]: Monitoring has been disabled
2012/10/05 15:35:52.710226 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:52.757718 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:52.768248 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:52.768628 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:52.769269 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:52.770394 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:52.770709 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:52.771514 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:52.771532 [27403]: Thawing priority 1
2012/10/05 15:35:52.771539 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:52.771556 [27403]: Thawing priority 2
2012/10/05 15:35:52.771562 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:52.771572 [27403]: Thawing priority 3
2012/10/05 15:35:52.771577 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:52.790713 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:52.790748 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:52.814518 [set_recmode:28404]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:52.814570 [set_recmode:28404]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:52.912889 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:52.913163 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:53.718627 [27403]: Freeze priority 1
2012/10/05 15:35:53.719406 [27403]: Freeze priority 2
2012/10/05 15:35:53.720077 [27403]: Freeze priority 3
2012/10/05 15:35:53.721202 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:53.721238 [27403]: Monitoring has been disabled
2012/10/05 15:35:53.721248 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:53.791481 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:53.791512 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:53.798951 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:53.799524 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:53.799900 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:53.800546 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:53.801690 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:53.802004 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:53.802874 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:53.802893 [27403]: Thawing priority 1
2012/10/05 15:35:53.802900 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:53.802914 [27403]: Thawing priority 2
2012/10/05 15:35:53.802921 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:53.802930 [27403]: Thawing priority 3
2012/10/05 15:35:53.802935 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:53.845599 [set_recmode:28470]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:53.845676 [set_recmode:28470]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:53.914845 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:53.914907 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:54.705627 [27403]: Freeze priority 1
2012/10/05 15:35:54.706425 [27403]: Freeze priority 2
2012/10/05 15:35:54.707118 [27403]: Freeze priority 3
2012/10/05 15:35:54.708215 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:54.708257 [27403]: Monitoring has been disabled
2012/10/05 15:35:54.708273 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:54.755205 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:54.756818 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:54.757173 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:54.757794 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:54.759042 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:54.759399 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:54.760190 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:54.760213 [27403]: Thawing priority 1
2012/10/05 15:35:54.760220 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:54.760247 [27403]: Thawing priority 2
2012/10/05 15:35:54.760255 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:54.760265 [27403]: Thawing priority 3
2012/10/05 15:35:54.760270 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:54.772877 [set_recmode:28536]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:54.772925 [set_recmode:28536]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:54.792333 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:54.792358 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:54.916940 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:54.917018 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:55.713035 [27403]: Freeze priority 1
2012/10/05 15:35:55.713788 [27403]: Freeze priority 2
2012/10/05 15:35:55.714491 [27403]: Freeze priority 3
2012/10/05 15:35:55.715630 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:55.715666 [27403]: Monitoring has been disabled
2012/10/05 15:35:55.715676 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:55.792675 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:55.792706 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:55.798792 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:55.801141 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:55.801492 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:55.802131 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:55.803313 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:55.803632 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:55.804405 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:55.804423 [27403]: Thawing priority 1
2012/10/05 15:35:55.804430 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:55.804445 [27403]: Thawing priority 2
2012/10/05 15:35:55.804452 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:55.804462 [27403]: Thawing priority 3
2012/10/05 15:35:55.804478 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:55.844308 [set_recmode:28602]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:55.844382 [set_recmode:28602]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:55.919169 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:55.919248 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:56.708958 [27403]: Freeze priority 1
2012/10/05 15:35:56.709702 [27403]: Freeze priority 2
2012/10/05 15:35:56.710351 [27403]: Freeze priority 3
2012/10/05 15:35:56.711434 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:56.711462 [27403]: Monitoring has been disabled
2012/10/05 15:35:56.711472 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:56.765273 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:56.769741 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:56.770059 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:56.770675 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:56.771780 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:56.772135 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:56.772919 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:56.772939 [27403]: Thawing priority 1
2012/10/05 15:35:56.772946 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:56.772963 [27403]: Thawing priority 2
2012/10/05 15:35:56.772970 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:56.772980 [27403]: Thawing priority 3
2012/10/05 15:35:56.772987 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:56.785413 [set_recmode:28668]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:56.785491 [set_recmode:28668]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:56.793936 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:56.793978 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:56.921222 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:56.921299 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:57.726537 [27403]: Freeze priority 1
2012/10/05 15:35:57.727273 [27403]: Freeze priority 2
2012/10/05 15:35:57.727935 [27403]: Freeze priority 3
2012/10/05 15:35:57.728987 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:57.729024 [27403]: Monitoring has been disabled
2012/10/05 15:35:57.729035 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:57.776105 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:57.794023 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:57.794103 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:57.794116 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:57.794359 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:57.794947 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:57.796105 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:57.796402 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:57.797207 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:57.797227 [27403]: Thawing priority 1
2012/10/05 15:35:57.797234 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:57.797253 [27403]: Thawing priority 2
2012/10/05 15:35:57.797259 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:57.797281 [27403]: Thawing priority 3
2012/10/05 15:35:57.797288 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:57.841632 [set_recmode:28734]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:57.841693 [set_recmode:28734]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:57.923209 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:57.923478 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:58.713967 [27403]: Freeze priority 1
2012/10/05 15:35:58.714729 [27403]: Freeze priority 2
2012/10/05 15:35:58.715376 [27403]: Freeze priority 3
2012/10/05 15:35:58.716518 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:58.716548 [27403]: Monitoring has been disabled
2012/10/05 15:35:58.716564 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:58.768581 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:58.769089 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:58.769440 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:58.770042 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:58.771259 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:58.771658 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:58.772668 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:58.772697 [27403]: Thawing priority 1
2012/10/05 15:35:58.772704 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:58.772723 [27403]: Thawing priority 2
2012/10/05 15:35:58.772730 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:58.772740 [27403]: Thawing priority 3
2012/10/05 15:35:58.772747 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:58.784996 [set_recmode:28800]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:58.785057 [set_recmode:28800]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:58.794617 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:58.794675 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:58.925330 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:58.925414 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:59.720203 [27403]: Freeze priority 1
2012/10/05 15:35:59.720944 [27403]: Freeze priority 2
2012/10/05 15:35:59.721644 [27403]: Freeze priority 3
2012/10/05 15:35:59.722704 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:59.722736 [27403]: Monitoring has been disabled
2012/10/05 15:35:59.722745 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:59.784122 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:59.784681 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:59.785027 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:59.785620 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:59.786750 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:59.787074 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:59.787965 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:59.787984 [27403]: Thawing priority 1
2012/10/05 15:35:59.787991 [27403]: Release freeze handler for prio 1
2012/10/05 15:35:59.788006 [27403]: Thawing priority 2
2012/10/05 15:35:59.788013 [27403]: Release freeze handler for prio 2
2012/10/05 15:35:59.788024 [27403]: Thawing priority 3
2012/10/05 15:35:59.788040 [27403]: Release freeze handler for prio 3
2012/10/05 15:35:59.795621 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:59.795677 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:35:59.829698 [set_recmode:28866]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:59.829752 [set_recmode:28866]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:35:59.927417 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:35:59.927499 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:00.716435 [27403]: Freeze priority 1
2012/10/05 15:36:00.717144 [27403]: Freeze priority 2
2012/10/05 15:36:00.717776 [27403]: Freeze priority 3
2012/10/05 15:36:00.718824 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:00.718848 [27403]: Monitoring has been disabled
2012/10/05 15:36:00.718856 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:00.771474 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:00.772611 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:00.772918 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:00.773490 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:00.774534 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:00.774826 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:00.775566 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:00.775582 [27403]: Thawing priority 1
2012/10/05 15:36:00.775589 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:00.775602 [27403]: Thawing priority 2
2012/10/05 15:36:00.775608 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:00.775618 [27403]: Thawing priority 3
2012/10/05 15:36:00.775636 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:00.791604 [set_recmode:28933]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:00.791670 [set_recmode:28933]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:00.796243 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:00.796268 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:00.929407 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:00.929484 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:01.736384 [27403]: Freeze priority 1
2012/10/05 15:36:01.737117 [27403]: Freeze priority 2
2012/10/05 15:36:01.737746 [27403]: Freeze priority 3
2012/10/05 15:36:01.738740 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:01.738769 [27403]: Monitoring has been disabled
2012/10/05 15:36:01.738779 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:01.787768 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:01.788227 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:01.788529 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:01.789118 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:01.790248 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:01.790563 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:01.791435 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:01.791452 [27403]: Thawing priority 1
2012/10/05 15:36:01.791459 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:01.791472 [27403]: Thawing priority 2
2012/10/05 15:36:01.791486 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:01.791496 [27403]: Thawing priority 3
2012/10/05 15:36:01.791502 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:01.797252 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:01.797285 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:01.834889 [set_recmode:28999]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:01.834958 [set_recmode:28999]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:01.931445 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:01.931530 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:02.720372 [27403]: Freeze priority 1
2012/10/05 15:36:02.721114 [27403]: Freeze priority 2
2012/10/05 15:36:02.721798 [27403]: Freeze priority 3
2012/10/05 15:36:02.722787 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:02.722823 [27403]: Monitoring has been disabled
2012/10/05 15:36:02.722834 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:02.771375 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:02.771858 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:02.772187 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:02.772779 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:02.773914 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:02.774236 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:02.775152 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:02.775172 [27403]: Thawing priority 1
2012/10/05 15:36:02.775179 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:02.775195 [27403]: Thawing priority 2
2012/10/05 15:36:02.775202 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:02.775212 [27403]: Thawing priority 3
2012/10/05 15:36:02.775219 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:02.787744 [set_recmode:29067]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:02.787814 [set_recmode:29067]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:02.798367 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:02.798402 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:02.933570 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:02.933829 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:03.728275 [27403]: Freeze priority 1
2012/10/05 15:36:03.729056 [27403]: Freeze priority 2
2012/10/05 15:36:03.729707 [27403]: Freeze priority 3
2012/10/05 15:36:03.730774 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:03.730806 [27403]: Monitoring has been disabled
2012/10/05 15:36:03.730827 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:03.796473 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:03.797015 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:03.797326 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:03.797836 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:03.798567 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:03.798579 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:03.798804 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:03.799082 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:03.799880 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:03.799904 [27403]: Thawing priority 1
2012/10/05 15:36:03.799911 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:03.799923 [27403]: Thawing priority 2
2012/10/05 15:36:03.799930 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:03.799939 [27403]: Thawing priority 3
2012/10/05 15:36:03.799946 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:03.842182 [set_recmode:29136]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:03.842254 [set_recmode:29136]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:03.934181 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:03.934260 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:04.724268 [27403]: Freeze priority 1
2012/10/05 15:36:04.725134 [27403]: Freeze priority 2
2012/10/05 15:36:04.725909 [27403]: Freeze priority 3
2012/10/05 15:36:04.727049 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:04.727092 [27403]: Monitoring has been disabled
2012/10/05 15:36:04.727101 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:04.799440 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:04.799481 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:04.799528 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:04.800233 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:04.800578 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:04.801101 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:04.802104 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:04.802394 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:04.803132 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:04.803148 [27403]: Thawing priority 1
2012/10/05 15:36:04.803154 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:04.803167 [27403]: Thawing priority 2
2012/10/05 15:36:04.803173 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:04.803182 [27403]: Thawing priority 3
2012/10/05 15:36:04.803188 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:04.815686 [set_recmode:29206]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:04.815774 [set_recmode:29206]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:04.936126 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:04.936203 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:05.725272 [27403]: Freeze priority 1
2012/10/05 15:36:05.726030 [27403]: Freeze priority 2
2012/10/05 15:36:05.726713 [27403]: Freeze priority 3
2012/10/05 15:36:05.727837 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:05.727883 [27403]: Monitoring has been disabled
2012/10/05 15:36:05.727893 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:05.775246 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:05.776258 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:05.776649 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:05.777333 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:05.778645 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:05.779036 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:05.780023 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:05.780073 [27403]: Thawing priority 1
2012/10/05 15:36:05.780080 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:05.780100 [27403]: Thawing priority 2
2012/10/05 15:36:05.780107 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:05.780118 [27403]: Thawing priority 3
2012/10/05 15:36:05.780124 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:05.800127 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:05.800164 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:05.817826 [set_recmode:29276]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:05.817888 [set_recmode:29276]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:05.938327 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:05.938408 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:06.748170 [27403]: Freeze priority 1
2012/10/05 15:36:06.749007 [27403]: Freeze priority 2
2012/10/05 15:36:06.749873 [27403]: Freeze priority 3
2012/10/05 15:36:06.751095 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:06.751127 [27403]: Monitoring has been disabled
2012/10/05 15:36:06.751136 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:06.800487 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:06.800575 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:06.800623 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:06.811120 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:06.811552 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:06.812202 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:06.813444 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:06.813807 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:06.814587 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:06.814608 [27403]: Thawing priority 1
2012/10/05 15:36:06.814615 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:06.814634 [27403]: Thawing priority 2
2012/10/05 15:36:06.814641 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:06.814651 [27403]: Thawing priority 3
2012/10/05 15:36:06.814657 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:06.858264 [set_recmode:29347]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:06.858332 [set_recmode:29347]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:06.940399 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:06.940480 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:07.728693 [27403]: Freeze priority 1
2012/10/05 15:36:07.729480 [27403]: Freeze priority 2
2012/10/05 15:36:07.730179 [27403]: Freeze priority 3
2012/10/05 15:36:07.731305 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:07.731336 [27403]: Monitoring has been disabled
2012/10/05 15:36:07.731345 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:07.779503 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:07.795290 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:07.795668 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:07.796263 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:07.797409 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:07.797748 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:07.798549 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:07.798569 [27403]: Thawing priority 1
2012/10/05 15:36:07.798576 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:07.798610 [27403]: Thawing priority 2
2012/10/05 15:36:07.798617 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:07.798628 [27403]: Thawing priority 3
2012/10/05 15:36:07.798634 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:07.801208 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:07.801235 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:07.810953 [set_recmode:29414]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:07.811002 [set_recmode:29414]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:07.942346 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:07.942679 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:08.730852 [27403]: Freeze priority 1
2012/10/05 15:36:08.731566 [27403]: Freeze priority 2
2012/10/05 15:36:08.732226 [27403]: Freeze priority 3
2012/10/05 15:36:08.733283 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:08.733318 [27403]: Monitoring has been disabled
2012/10/05 15:36:08.733328 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:08.781497 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:08.781993 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:08.782315 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:08.782869 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:08.783937 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:08.784232 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:08.785095 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:08.785117 [27403]: Thawing priority 1
2012/10/05 15:36:08.785135 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:08.785154 [27403]: Thawing priority 2
2012/10/05 15:36:08.785161 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:08.785170 [27403]: Thawing priority 3
2012/10/05 15:36:08.785177 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:08.801801 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:08.801859 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:08.822577 [set_recmode:29480]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:08.822656 [set_recmode:29480]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:08.944475 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:08.944552 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:09.754458 [27403]: Freeze priority 1
2012/10/05 15:36:09.755176 [27403]: Freeze priority 2
2012/10/05 15:36:09.755815 [27403]: Freeze priority 3
2012/10/05 15:36:09.756858 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:09.756886 [27403]: Monitoring has been disabled
2012/10/05 15:36:09.756894 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:09.802560 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:09.802597 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:09.802635 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:09.815615 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:09.815975 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:09.816549 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:09.817693 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:09.818022 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:09.818774 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:09.818794 [27403]: Thawing priority 1
2012/10/05 15:36:09.818802 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:09.818820 [27403]: Thawing priority 2
2012/10/05 15:36:09.818827 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:09.818838 [27403]: Thawing priority 3
2012/10/05 15:36:09.818845 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:09.862611 [set_recmode:29546]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:09.862687 [set_recmode:29546]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:09.946524 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:09.946600 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:10.734375 [27403]: Freeze priority 1
2012/10/05 15:36:10.735092 [27403]: Freeze priority 2
2012/10/05 15:36:10.735745 [27403]: Freeze priority 3
2012/10/05 15:36:10.736769 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:10.736816 [27403]: Monitoring has been disabled
2012/10/05 15:36:10.736826 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:10.785947 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:10.787396 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:10.787753 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:10.788380 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:10.789590 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:10.789912 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:10.790665 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:10.790681 [27403]: Thawing priority 1
2012/10/05 15:36:10.790688 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:10.790703 [27403]: Thawing priority 2
2012/10/05 15:36:10.790709 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:10.790718 [27403]: Thawing priority 3
2012/10/05 15:36:10.790724 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:10.803114 [set_recmode:29612]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:10.803188 [set_recmode:29612]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:10.803300 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:10.803317 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:10.948646 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:10.948723 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:11.735203 [27403]: Freeze priority 1
2012/10/05 15:36:11.736008 [27403]: Freeze priority 2
2012/10/05 15:36:11.736659 [27403]: Freeze priority 3
2012/10/05 15:36:11.737770 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:11.737801 [27403]: Monitoring has been disabled
2012/10/05 15:36:11.737810 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:11.790608 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:11.794937 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:11.795269 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:11.795829 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:11.796985 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:11.797288 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:11.798042 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:11.798061 [27403]: Thawing priority 1
2012/10/05 15:36:11.798067 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:11.798085 [27403]: Thawing priority 2
2012/10/05 15:36:11.798091 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:11.798100 [27403]: Thawing priority 3
2012/10/05 15:36:11.798106 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:11.803943 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:11.803972 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:11.840922 [set_recmode:29678]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:11.840978 [set_recmode:29678]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:11.950655 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:11.950739 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:12.743039 [27403]: Freeze priority 1
2012/10/05 15:36:12.743785 [27403]: Freeze priority 2
2012/10/05 15:36:12.744461 [27403]: Freeze priority 3
2012/10/05 15:36:12.745584 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:12.745616 [27403]: Monitoring has been disabled
2012/10/05 15:36:12.745626 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:12.804000 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:12.804032 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:12.806986 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:12.807533 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:12.807862 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:12.808521 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:12.809655 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:12.809991 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:12.810859 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:12.810876 [27403]: Thawing priority 1
2012/10/05 15:36:12.810882 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:12.810897 [27403]: Thawing priority 2
2012/10/05 15:36:12.810904 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:12.810927 [27403]: Thawing priority 3
2012/10/05 15:36:12.810934 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:12.841393 [set_recmode:29744]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:12.841462 [set_recmode:29744]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:12.952643 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:12.952908 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:13.739205 [27403]: Freeze priority 1
2012/10/05 15:36:13.740005 [27403]: Freeze priority 2
2012/10/05 15:36:13.740687 [27403]: Freeze priority 3
2012/10/05 15:36:13.741782 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:13.741814 [27403]: Monitoring has been disabled
2012/10/05 15:36:13.741823 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:13.789199 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:13.804544 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:13.804590 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:13.818291 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:13.818638 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:13.819151 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:13.820151 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:13.820431 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:13.821193 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:13.821213 [27403]: Thawing priority 1
2012/10/05 15:36:13.821219 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:13.821237 [27403]: Thawing priority 2
2012/10/05 15:36:13.821243 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:13.821253 [27403]: Thawing priority 3
2012/10/05 15:36:13.821259 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:13.920146 [set_recmode:29810]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:13.920215 [set_recmode:29810]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:13.954698 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:13.954767 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:14.740718 [27403]: Freeze priority 1
2012/10/05 15:36:14.741422 [27403]: Freeze priority 2
2012/10/05 15:36:14.742081 [27403]: Freeze priority 3
2012/10/05 15:36:14.743093 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:14.743124 [27403]: Monitoring has been disabled
2012/10/05 15:36:14.743133 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:14.788609 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:14.789217 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:14.789627 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:14.790490 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:14.791598 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:14.791886 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:14.792697 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:14.792714 [27403]: Thawing priority 1
2012/10/05 15:36:14.792721 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:14.792736 [27403]: Thawing priority 2
2012/10/05 15:36:14.792742 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:14.792752 [27403]: Thawing priority 3
2012/10/05 15:36:14.792758 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:14.805241 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:14.805279 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:14.805330 [set_recmode:29876]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:14.805367 [set_recmode:29876]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:14.957396 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:14.957475 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:15.764013 [27403]: Freeze priority 1
2012/10/05 15:36:15.764831 [27403]: Freeze priority 2
2012/10/05 15:36:15.765514 [27403]: Freeze priority 3
2012/10/05 15:36:15.766640 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:15.766670 [27403]: Monitoring has been disabled
2012/10/05 15:36:15.766680 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:15.805502 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:15.805548 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:15.825645 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:15.826152 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:15.826497 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:15.827125 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:15.828424 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:15.828814 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:15.829839 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:15.829874 [27403]: Thawing priority 1
2012/10/05 15:36:15.829881 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:15.829901 [27403]: Thawing priority 2
2012/10/05 15:36:15.829909 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:15.829918 [27403]: Thawing priority 3
2012/10/05 15:36:15.829924 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:15.871826 [set_recmode:29942]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:15.871904 [set_recmode:29942]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:15.958442 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:15.958519 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:16.744210 [27403]: Freeze priority 1
2012/10/05 15:36:16.745016 [27403]: Freeze priority 2
2012/10/05 15:36:16.745740 [27403]: Freeze priority 3
2012/10/05 15:36:16.746874 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:16.746904 [27403]: Monitoring has been disabled
2012/10/05 15:36:16.746913 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:16.794087 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:16.795310 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:16.795645 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:16.796148 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:16.797141 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:16.797458 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:16.798232 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:16.798249 [27403]: Thawing priority 1
2012/10/05 15:36:16.798255 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:16.798270 [27403]: Thawing priority 2
2012/10/05 15:36:16.798277 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:16.798286 [27403]: Thawing priority 3
2012/10/05 15:36:16.798292 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:16.806282 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:16.806321 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:16.810157 [set_recmode:30008]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:16.810211 [set_recmode:30008]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:16.960464 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:16.960534 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:17.744749 [27403]: Freeze priority 1
2012/10/05 15:36:17.745566 [27403]: Freeze priority 2
2012/10/05 15:36:17.746265 [27403]: Freeze priority 3
2012/10/05 15:36:17.747369 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:17.747401 [27403]: Monitoring has been disabled
2012/10/05 15:36:17.747410 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:17.795269 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:17.796834 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:17.797227 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:17.797826 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:17.798897 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:17.799198 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:17.799968 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:17.799986 [27403]: Thawing priority 1
2012/10/05 15:36:17.799994 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:17.800010 [27403]: Thawing priority 2
2012/10/05 15:36:17.800017 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:17.800026 [27403]: Thawing priority 3
2012/10/05 15:36:17.800033 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:17.806772 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:17.806807 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:17.842160 [set_recmode:30074]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:17.842215 [set_recmode:30074]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:17.962688 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:17.962960 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:18.752837 [27403]: Freeze priority 1
2012/10/05 15:36:18.753619 [27403]: Freeze priority 2
2012/10/05 15:36:18.754319 [27403]: Freeze priority 3
2012/10/05 15:36:18.755445 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:18.755477 [27403]: Monitoring has been disabled
2012/10/05 15:36:18.755487 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:18.804996 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:18.805464 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:18.805757 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:18.806336 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:18.806857 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:18.806870 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:18.807378 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:18.807672 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:18.808472 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:18.808489 [27403]: Thawing priority 1
2012/10/05 15:36:18.808495 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:18.808510 [27403]: Thawing priority 2
2012/10/05 15:36:18.808516 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:18.808525 [27403]: Thawing priority 3
2012/10/05 15:36:18.808531 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:18.826480 [set_recmode:30140]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:18.826532 [set_recmode:30140]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:18.964630 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:18.964709 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:19.763784 [27403]: Freeze priority 1
2012/10/05 15:36:19.764537 [27403]: Freeze priority 2
2012/10/05 15:36:19.765224 [27403]: Freeze priority 3
2012/10/05 15:36:19.766321 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:19.766352 [27403]: Monitoring has been disabled
2012/10/05 15:36:19.766380 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:19.807902 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:19.807941 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:19.827335 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:19.827919 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:19.828259 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:19.828876 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:19.830099 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:19.830422 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:19.831299 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:19.831317 [27403]: Thawing priority 1
2012/10/05 15:36:19.831324 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:19.831338 [27403]: Thawing priority 2
2012/10/05 15:36:19.831345 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:19.831355 [27403]: Thawing priority 3
2012/10/05 15:36:19.831374 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:19.873800 [set_recmode:30206]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:19.873865 [set_recmode:30206]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:19.966648 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:19.966722 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:20.749923 [27403]: Freeze priority 1
2012/10/05 15:36:20.750719 [27403]: Freeze priority 2
2012/10/05 15:36:20.751389 [27403]: Freeze priority 3
2012/10/05 15:36:20.752551 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:20.752580 [27403]: Monitoring has been disabled
2012/10/05 15:36:20.752590 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:20.808598 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:20.808637 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:20.808680 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:20.810813 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:20.811200 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:20.811850 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:20.813076 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:20.813406 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:20.814190 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:20.814209 [27403]: Thawing priority 1
2012/10/05 15:36:20.814215 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:20.814231 [27403]: Thawing priority 2
2012/10/05 15:36:20.814238 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:20.814247 [27403]: Thawing priority 3
2012/10/05 15:36:20.814252 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:20.826539 [set_recmode:30272]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:20.826594 [set_recmode:30272]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:20.968789 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:20.968870 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:21.759161 [27403]: Freeze priority 1
2012/10/05 15:36:21.759908 [27403]: Freeze priority 2
2012/10/05 15:36:21.760578 [27403]: Freeze priority 3
2012/10/05 15:36:21.761654 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:21.761691 [27403]: Monitoring has been disabled
2012/10/05 15:36:21.761700 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:21.809505 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:21.809550 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:21.839721 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:21.840341 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:21.840691 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:21.841298 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:21.842412 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:21.842753 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:21.843823 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:21.843842 [27403]: Thawing priority 1
2012/10/05 15:36:21.843849 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:21.843866 [27403]: Thawing priority 2
2012/10/05 15:36:21.843872 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:21.843881 [27403]: Thawing priority 3
2012/10/05 15:36:21.843887 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:21.886370 [set_recmode:30338]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:21.886432 [set_recmode:30338]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:21.970867 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:21.970948 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:22.754329 [27403]: Freeze priority 1
2012/10/05 15:36:22.755098 [27403]: Freeze priority 2
2012/10/05 15:36:22.755775 [27403]: Freeze priority 3
2012/10/05 15:36:22.756928 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:22.756959 [27403]: Monitoring has been disabled
2012/10/05 15:36:22.756968 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:22.810277 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:22.810312 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:22.818705 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:22.819258 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:22.819598 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:22.820228 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:22.821388 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:22.821732 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:22.822584 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:22.822630 [27403]: Thawing priority 1
2012/10/05 15:36:22.822640 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:22.822653 [27403]: Thawing priority 2
2012/10/05 15:36:22.822660 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:22.822669 [27403]: Thawing priority 3
2012/10/05 15:36:22.822675 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:22.835097 [set_recmode:30404]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:22.835158 [set_recmode:30404]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:22.972989 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:22.973250 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:23.771329 [27403]: Freeze priority 1
2012/10/05 15:36:23.772080 [27403]: Freeze priority 2
2012/10/05 15:36:23.772732 [27403]: Freeze priority 3
2012/10/05 15:36:23.773785 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:23.773829 [27403]: Monitoring has been disabled
2012/10/05 15:36:23.773839 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:23.810664 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:23.810705 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:23.847117 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:23.847688 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:23.847986 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:23.848524 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:23.849557 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:23.849857 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:23.850699 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:23.850717 [27403]: Thawing priority 1
2012/10/05 15:36:23.850724 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:23.850738 [27403]: Thawing priority 2
2012/10/05 15:36:23.850745 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:23.850755 [27403]: Thawing priority 3
2012/10/05 15:36:23.850761 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:23.894435 [set_recmode:30470]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:23.894505 [set_recmode:30470]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:23.974924 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:23.974998 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:24.757721 [27403]: Freeze priority 1
2012/10/05 15:36:24.758471 [27403]: Freeze priority 2
2012/10/05 15:36:24.759120 [27403]: Freeze priority 3
2012/10/05 15:36:24.760228 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:24.760260 [27403]: Monitoring has been disabled
2012/10/05 15:36:24.760270 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:24.816705 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:24.816764 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:24.829750 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:24.830387 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:24.830760 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:24.831377 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:24.832552 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:24.832917 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:24.833811 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:24.833831 [27403]: Thawing priority 1
2012/10/05 15:36:24.833838 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:24.833853 [27403]: Thawing priority 2
2012/10/05 15:36:24.833860 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:24.833869 [27403]: Thawing priority 3
2012/10/05 15:36:24.833875 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:24.847258 [set_recmode:30536]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:24.847318 [set_recmode:30536]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:24.977083 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:24.977166 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:25.765522 [27403]: Freeze priority 1
2012/10/05 15:36:25.766267 [27403]: Freeze priority 2
2012/10/05 15:36:25.766933 [27403]: Freeze priority 3
2012/10/05 15:36:25.767994 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:25.768024 [27403]: Monitoring has been disabled
2012/10/05 15:36:25.768033 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:25.817320 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:25.817358 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:25.820629 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:25.821137 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:25.821493 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:25.822082 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:25.823120 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:25.823430 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:25.824317 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:25.824339 [27403]: Thawing priority 1
2012/10/05 15:36:25.824347 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:25.824361 [27403]: Thawing priority 2
2012/10/05 15:36:25.824367 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:25.824376 [27403]: Thawing priority 3
2012/10/05 15:36:25.824382 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:25.867837 [set_recmode:30602]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:25.867908 [set_recmode:30602]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:25.979105 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:25.979185 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:26.776405 [27403]: Freeze priority 1
2012/10/05 15:36:26.777141 [27403]: Freeze priority 2
2012/10/05 15:36:26.777801 [27403]: Freeze priority 3
2012/10/05 15:36:26.778863 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:26.778890 [27403]: Monitoring has been disabled
2012/10/05 15:36:26.778899 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:26.818333 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:26.818369 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:26.837722 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:26.838282 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:26.838594 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:26.839199 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:26.840374 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:26.840684 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:26.841497 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:26.841517 [27403]: Thawing priority 1
2012/10/05 15:36:26.841525 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:26.841541 [27403]: Thawing priority 2
2012/10/05 15:36:26.841548 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:26.841557 [27403]: Thawing priority 3
2012/10/05 15:36:26.841564 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:26.868490 [set_recmode:30668]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:26.868565 [set_recmode:30668]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:26.981143 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:26.981223 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:27.763633 [27403]: Freeze priority 1
2012/10/05 15:36:27.764490 [27403]: Freeze priority 2
2012/10/05 15:36:27.765214 [27403]: Freeze priority 3
2012/10/05 15:36:27.766429 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:27.766468 [27403]: Monitoring has been disabled
2012/10/05 15:36:27.766478 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:27.814776 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:27.816448 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:27.816797 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:27.817370 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:27.818422 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:27.818473 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:27.818485 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:27.818715 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:27.819563 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:27.819589 [27403]: Thawing priority 1
2012/10/05 15:36:27.819596 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:27.819614 [27403]: Thawing priority 2
2012/10/05 15:36:27.819621 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:27.819632 [27403]: Thawing priority 3
2012/10/05 15:36:27.819638 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:27.842121 [set_recmode:30734]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:27.842182 [set_recmode:30734]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:27.983203 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:27.983460 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:28.771976 [27403]: Freeze priority 1
2012/10/05 15:36:28.772711 [27403]: Freeze priority 2
2012/10/05 15:36:28.773338 [27403]: Freeze priority 3
2012/10/05 15:36:28.774404 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:28.774437 [27403]: Monitoring has been disabled
2012/10/05 15:36:28.774476 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:28.819487 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:28.819525 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:28.824621 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:28.835529 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:28.835939 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:28.836546 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:28.837709 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:28.838047 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:28.838797 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:28.838816 [27403]: Thawing priority 1
2012/10/05 15:36:28.838822 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:28.838839 [27403]: Thawing priority 2
2012/10/05 15:36:28.838845 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:28.838854 [27403]: Thawing priority 3
2012/10/05 15:36:28.838860 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:28.884658 [set_recmode:30800]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:28.884735 [set_recmode:30800]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:28.985222 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:28.985298 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:29.765882 [27403]: Freeze priority 1
2012/10/05 15:36:29.766629 [27403]: Freeze priority 2
2012/10/05 15:36:29.767266 [27403]: Freeze priority 3
2012/10/05 15:36:29.768328 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:29.768355 [27403]: Monitoring has been disabled
2012/10/05 15:36:29.768364 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:29.817222 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:29.817701 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:29.818019 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:29.818667 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:29.819768 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:29.819816 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:29.819827 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:29.820086 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:29.820931 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:29.820950 [27403]: Thawing priority 1
2012/10/05 15:36:29.820957 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:29.820972 [27403]: Thawing priority 2
2012/10/05 15:36:29.820979 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:29.820988 [27403]: Thawing priority 3
2012/10/05 15:36:29.820995 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:29.833754 [set_recmode:30866]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:29.833822 [set_recmode:30866]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:29.987271 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:29.987352 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:30.783375 [27403]: Freeze priority 1
2012/10/05 15:36:30.784081 [27403]: Freeze priority 2
2012/10/05 15:36:30.784786 [27403]: Freeze priority 3
2012/10/05 15:36:30.785823 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:30.785857 [27403]: Monitoring has been disabled
2012/10/05 15:36:30.785866 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:30.820628 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:30.820668 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:30.847961 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:30.848536 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:30.848892 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:30.849521 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:30.850839 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:30.851188 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:30.852081 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:30.852110 [27403]: Thawing priority 1
2012/10/05 15:36:30.852118 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:30.852135 [27403]: Thawing priority 2
2012/10/05 15:36:30.852142 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:30.852152 [27403]: Thawing priority 3
2012/10/05 15:36:30.852159 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:30.898349 [set_recmode:30932]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:30.898425 [set_recmode:30932]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:30.989347 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:30.989441 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:31.774503 [27403]: Freeze priority 1
2012/10/05 15:36:31.775329 [27403]: Freeze priority 2
2012/10/05 15:36:31.776041 [27403]: Freeze priority 3
2012/10/05 15:36:31.777218 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:31.777254 [27403]: Monitoring has been disabled
2012/10/05 15:36:31.777264 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:31.831391 [27403]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:36:31.831443 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:31.837769 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:31.838295 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:31.838642 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:31.839273 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:31.840395 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:31.840732 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:31.841577 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:31.841595 [27403]: Thawing priority 1
2012/10/05 15:36:31.841602 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:31.841619 [27403]: Thawing priority 2
2012/10/05 15:36:31.841626 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:31.841636 [27403]: Thawing priority 3
2012/10/05 15:36:31.841642 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:31.853994 [set_recmode:30998]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:31.854052 [set_recmode:30998]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:31.991468 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:31.991544 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:32.777737 [27403]: Freeze priority 1
2012/10/05 15:36:32.778493 [27403]: Freeze priority 2
2012/10/05 15:36:32.779210 [27403]: Freeze priority 3
2012/10/05 15:36:32.780387 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:32.780421 [27403]: Monitoring has been disabled
2012/10/05 15:36:32.780430 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:32.829375 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:32.831357 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:32.831713 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:32.831762 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:32.832331 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:32.833530 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:32.833849 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:32.834609 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:32.834626 [27403]: Thawing priority 1
2012/10/05 15:36:32.834633 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:32.834648 [27403]: Thawing priority 2
2012/10/05 15:36:32.834654 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:32.834663 [27403]: Thawing priority 3
2012/10/05 15:36:32.834669 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:32.878658 [set_recmode:31064]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:32.878745 [set_recmode:31064]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:32.993556 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:32.993819 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:33.789296 [27403]: Freeze priority 1
2012/10/05 15:36:33.790072 [27403]: Freeze priority 2
2012/10/05 15:36:33.790748 [27403]: Freeze priority 3
2012/10/05 15:36:33.791877 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:33.791905 [27403]: Monitoring has been disabled
2012/10/05 15:36:33.791915 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:33.832061 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:33.840590 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:33.841687 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:33.842024 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:33.842664 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:33.843990 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:33.844344 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:33.845229 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:33.845249 [27403]: Thawing priority 1
2012/10/05 15:36:33.845256 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:33.845273 [27403]: Thawing priority 2
2012/10/05 15:36:33.845280 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:33.845304 [27403]: Thawing priority 3
2012/10/05 15:36:33.845312 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:33.871955 [set_recmode:31130]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:33.872023 [set_recmode:31130]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:33.995683 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:33.995765 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:34.775391 [27403]: Freeze priority 1
2012/10/05 15:36:34.776169 [27403]: Freeze priority 2
2012/10/05 15:36:34.776835 [27403]: Freeze priority 3
2012/10/05 15:36:34.777996 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:34.778028 [27403]: Monitoring has been disabled
2012/10/05 15:36:34.778037 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:34.827816 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:34.833428 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:34.837677 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:34.838057 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:34.838681 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:34.839813 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:34.840135 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:34.840931 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:34.840952 [27403]: Thawing priority 1
2012/10/05 15:36:34.840959 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:34.840977 [27403]: Thawing priority 2
2012/10/05 15:36:34.840984 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:34.840994 [27403]: Thawing priority 3
2012/10/05 15:36:34.841001 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:34.870613 [set_recmode:31196]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:34.870681 [set_recmode:31196]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:34.997665 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:34.997755 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:35.783590 [27403]: Freeze priority 1
2012/10/05 15:36:35.784317 [27403]: Freeze priority 2
2012/10/05 15:36:35.784991 [27403]: Freeze priority 3
2012/10/05 15:36:35.786014 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:35.786043 [27403]: Monitoring has been disabled
2012/10/05 15:36:35.786052 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:35.834912 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:35.859440 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:35.860024 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:35.860356 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:35.860958 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:35.862136 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:35.862468 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:35.863400 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:35.863446 [27403]: Thawing priority 1
2012/10/05 15:36:35.863453 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:35.863468 [27403]: Thawing priority 2
2012/10/05 15:36:35.863474 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:35.863517 [27403]: Thawing priority 3
2012/10/05 15:36:35.863531 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:35.959657 [set_recmode:31262]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:35.959733 [set_recmode:31262]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:35.999658 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:35.999732 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:36.780790 [27403]: Freeze priority 1
2012/10/05 15:36:36.781588 [27403]: Freeze priority 2
2012/10/05 15:36:36.782278 [27403]: Freeze priority 3
2012/10/05 15:36:36.783396 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:36.783430 [27403]: Monitoring has been disabled
2012/10/05 15:36:36.783439 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:36.832757 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:36.835515 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:36.839891 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:36.840248 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:36.840844 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:36.841985 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:36.842310 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:36.843084 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:36.843104 [27403]: Thawing priority 1
2012/10/05 15:36:36.843111 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:36.843128 [27403]: Thawing priority 2
2012/10/05 15:36:36.843135 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:36.843145 [27403]: Thawing priority 3
2012/10/05 15:36:36.843152 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:36.855656 [set_recmode:31328]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:36.855714 [set_recmode:31328]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:37.001794 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:37.001895 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:37.797115 [27403]: Freeze priority 1
2012/10/05 15:36:37.797937 [27403]: Freeze priority 2
2012/10/05 15:36:37.798659 [27403]: Freeze priority 3
2012/10/05 15:36:37.799775 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:37.799805 [27403]: Monitoring has been disabled
2012/10/05 15:36:37.799814 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:37.836582 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:37.848447 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:37.848935 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:37.849257 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:37.849844 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:37.850872 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:37.851171 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:37.851951 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:37.851969 [27403]: Thawing priority 1
2012/10/05 15:36:37.851975 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:37.851990 [27403]: Thawing priority 2
2012/10/05 15:36:37.851997 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:37.852006 [27403]: Thawing priority 3
2012/10/05 15:36:37.852012 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:37.892028 [set_recmode:31394]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:37.892111 [set_recmode:31394]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:38.003855 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:38.004113 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:38.797340 [27403]: Freeze priority 1
2012/10/05 15:36:38.798077 [27403]: Freeze priority 2
2012/10/05 15:36:38.798774 [27403]: Freeze priority 3
2012/10/05 15:36:38.799879 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:38.799907 [27403]: Monitoring has been disabled
2012/10/05 15:36:38.799916 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:38.837034 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:38.865839 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:38.866435 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:38.866798 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:38.867410 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:38.868539 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:38.868875 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:38.869748 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:38.869769 [27403]: Thawing priority 1
2012/10/05 15:36:38.869777 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:38.869794 [27403]: Thawing priority 2
2012/10/05 15:36:38.869800 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:38.869811 [27403]: Thawing priority 3
2012/10/05 15:36:38.869817 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:38.899627 [set_recmode:31460]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:38.899698 [set_recmode:31460]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:39.005898 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:39.005992 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:39.784458 [27403]: Freeze priority 1
2012/10/05 15:36:39.785231 [27403]: Freeze priority 2
2012/10/05 15:36:39.785927 [27403]: Freeze priority 3
2012/10/05 15:36:39.787057 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:39.787087 [27403]: Monitoring has been disabled
2012/10/05 15:36:39.787097 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:39.835201 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:39.837738 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:39.850977 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:39.851403 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:39.852056 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:39.853269 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:39.853605 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:39.854434 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:39.854455 [27403]: Thawing priority 1
2012/10/05 15:36:39.854462 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:39.854481 [27403]: Thawing priority 2
2012/10/05 15:36:39.854488 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:39.854498 [27403]: Thawing priority 3
2012/10/05 15:36:39.854504 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:39.872966 [set_recmode:31526]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:39.873024 [set_recmode:31526]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:40.007973 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:40.008052 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:40.786055 [27403]: Freeze priority 1
2012/10/05 15:36:40.786860 [27403]: Freeze priority 2
2012/10/05 15:36:40.787639 [27403]: Freeze priority 3
2012/10/05 15:36:40.788795 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:40.788825 [27403]: Monitoring has been disabled
2012/10/05 15:36:40.788834 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:40.838155 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:40.838221 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:40.838667 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:40.839015 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:40.839672 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:40.840803 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:40.841118 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:40.841909 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:40.841926 [27403]: Thawing priority 1
2012/10/05 15:36:40.841932 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:40.841947 [27403]: Thawing priority 2
2012/10/05 15:36:40.841954 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:40.841963 [27403]: Thawing priority 3
2012/10/05 15:36:40.841969 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:40.881413 [set_recmode:31592]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:40.881482 [set_recmode:31592]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:41.010023 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:41.010104 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:41.809204 [27403]: Freeze priority 1
2012/10/05 15:36:41.809954 [27403]: Freeze priority 2
2012/10/05 15:36:41.810627 [27403]: Freeze priority 3
2012/10/05 15:36:41.811723 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:41.811755 [27403]: Monitoring has been disabled
2012/10/05 15:36:41.811764 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:41.838851 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:41.880383 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:41.880982 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:41.881328 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:41.881910 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:41.883058 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:41.883396 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:41.884309 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:41.884337 [27403]: Thawing priority 1
2012/10/05 15:36:41.884343 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:41.884359 [27403]: Thawing priority 2
2012/10/05 15:36:41.884366 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:41.884375 [27403]: Thawing priority 3
2012/10/05 15:36:41.884381 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:41.927266 [set_recmode:31658]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:41.927346 [set_recmode:31658]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:42.012182 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:42.012265 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:42.795985 [27403]: Freeze priority 1
2012/10/05 15:36:42.796758 [27403]: Freeze priority 2
2012/10/05 15:36:42.797545 [27403]: Freeze priority 3
2012/10/05 15:36:42.798687 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:42.798720 [27403]: Monitoring has been disabled
2012/10/05 15:36:42.798730 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:42.840326 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:42.846839 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:42.847311 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:42.847678 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:42.848303 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:42.849469 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:42.849795 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:42.850577 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:42.850595 [27403]: Thawing priority 1
2012/10/05 15:36:42.850602 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:42.850616 [27403]: Thawing priority 2
2012/10/05 15:36:42.850624 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:42.850634 [27403]: Thawing priority 3
2012/10/05 15:36:42.850641 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:42.863306 [set_recmode:31724]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:42.863395 [set_recmode:31724]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:43.014198 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:43.014532 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:43.790748 [27403]: Freeze priority 1
2012/10/05 15:36:43.791488 [27403]: Freeze priority 2
2012/10/05 15:36:43.792141 [27403]: Freeze priority 3
2012/10/05 15:36:43.793267 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:43.793302 [27403]: Monitoring has been disabled
2012/10/05 15:36:43.793311 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:43.840447 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:43.855797 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:43.858599 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:43.859011 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:43.859646 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:43.860842 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:43.861174 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:43.861920 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:43.861938 [27403]: Thawing priority 1
2012/10/05 15:36:43.861959 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:43.861975 [27403]: Thawing priority 2
2012/10/05 15:36:43.861982 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:43.861992 [27403]: Thawing priority 3
2012/10/05 15:36:43.861999 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:43.899870 [set_recmode:31790]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:43.899939 [set_recmode:31790]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:44.016757 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:44.016838 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:44.798982 [27403]: Freeze priority 1
2012/10/05 15:36:44.799747 [27403]: Freeze priority 2
2012/10/05 15:36:44.800412 [27403]: Freeze priority 3
2012/10/05 15:36:44.801534 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:44.801568 [27403]: Monitoring has been disabled
2012/10/05 15:36:44.801577 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:44.840688 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:44.848838 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:44.849959 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:44.850289 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:44.850874 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:44.852054 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:44.852394 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:44.853210 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:44.853230 [27403]: Thawing priority 1
2012/10/05 15:36:44.853237 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:44.853252 [27403]: Thawing priority 2
2012/10/05 15:36:44.853259 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:44.853269 [27403]: Thawing priority 3
2012/10/05 15:36:44.853276 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:44.865686 [set_recmode:31856]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:44.865745 [set_recmode:31856]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:45.018763 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:45.018844 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:45.809034 [27403]: Freeze priority 1
2012/10/05 15:36:45.809798 [27403]: Freeze priority 2
2012/10/05 15:36:45.810458 [27403]: Freeze priority 3
2012/10/05 15:36:45.811548 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:45.811588 [27403]: Monitoring has been disabled
2012/10/05 15:36:45.811599 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:45.841442 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:45.874641 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:45.875185 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:45.875499 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:45.876054 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:45.877141 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:45.877464 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:45.878352 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:45.878368 [27403]: Thawing priority 1
2012/10/05 15:36:45.878375 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:45.878389 [27403]: Thawing priority 2
2012/10/05 15:36:45.878396 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:45.878405 [27403]: Thawing priority 3
2012/10/05 15:36:45.878411 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:45.923335 [set_recmode:31922]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:45.923395 [set_recmode:31922]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:46.020877 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:46.020953 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:46.795714 [27403]: Freeze priority 1
2012/10/05 15:36:46.796430 [27403]: Freeze priority 2
2012/10/05 15:36:46.797053 [27403]: Freeze priority 3
2012/10/05 15:36:46.798099 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:46.798126 [27403]: Monitoring has been disabled
2012/10/05 15:36:46.798148 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:46.842175 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:46.846044 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:46.846548 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:46.846882 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:46.847550 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:46.848809 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:46.849143 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:46.850016 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:46.850032 [27403]: Thawing priority 1
2012/10/05 15:36:46.850039 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:46.850054 [27403]: Thawing priority 2
2012/10/05 15:36:46.850060 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:46.850069 [27403]: Thawing priority 3
2012/10/05 15:36:46.850075 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:46.862717 [set_recmode:31988]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:46.862783 [set_recmode:31988]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:47.022968 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:47.023043 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:47.803783 [27403]: Freeze priority 1
2012/10/05 15:36:47.804584 [27403]: Freeze priority 2
2012/10/05 15:36:47.805402 [27403]: Freeze priority 3
2012/10/05 15:36:47.806570 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:47.806603 [27403]: Monitoring has been disabled
2012/10/05 15:36:47.806612 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:47.843220 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:47.855468 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:47.866198 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:47.866561 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:47.867122 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:47.868172 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:47.868475 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:47.869223 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:47.869256 [27403]: Thawing priority 1
2012/10/05 15:36:47.869263 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:47.869281 [27403]: Thawing priority 2
2012/10/05 15:36:47.869287 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:47.869297 [27403]: Thawing priority 3
2012/10/05 15:36:47.869302 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:47.911334 [set_recmode:32054]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:47.911404 [set_recmode:32054]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:48.025046 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:48.025437 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:48.799366 [27403]: Freeze priority 1
2012/10/05 15:36:48.800161 [27403]: Freeze priority 2
2012/10/05 15:36:48.800884 [27403]: Freeze priority 3
2012/10/05 15:36:48.802160 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:48.802199 [27403]: Monitoring has been disabled
2012/10/05 15:36:48.802209 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:48.843782 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:48.859886 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:48.860422 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:48.860758 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:48.861330 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:48.862340 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:48.862628 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:48.863455 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:48.863472 [27403]: Thawing priority 1
2012/10/05 15:36:48.863479 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:48.863493 [27403]: Thawing priority 2
2012/10/05 15:36:48.863499 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:48.863508 [27403]: Thawing priority 3
2012/10/05 15:36:48.863514 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:48.876442 [set_recmode:32120]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:48.876504 [set_recmode:32120]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:49.027589 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:49.027666 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:49.801508 [27403]: Freeze priority 1
2012/10/05 15:36:49.802224 [27403]: Freeze priority 2
2012/10/05 15:36:49.802869 [27403]: Freeze priority 3
2012/10/05 15:36:49.803910 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:49.803938 [27403]: Monitoring has been disabled
2012/10/05 15:36:49.803951 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:49.844363 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:49.871526 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:49.872123 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:49.872498 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:49.873126 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:49.874306 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:49.874651 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:49.875580 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:49.875601 [27403]: Thawing priority 1
2012/10/05 15:36:49.875608 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:49.875624 [27403]: Thawing priority 2
2012/10/05 15:36:49.875630 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:49.875639 [27403]: Thawing priority 3
2012/10/05 15:36:49.875645 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:49.919440 [set_recmode:32186]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:49.919514 [set_recmode:32186]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:50.029669 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:50.029751 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:50.805856 [27403]: Freeze priority 1
2012/10/05 15:36:50.806619 [27403]: Freeze priority 2
2012/10/05 15:36:50.807291 [27403]: Freeze priority 3
2012/10/05 15:36:50.808681 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:50.808740 [27403]: Monitoring has been disabled
2012/10/05 15:36:50.808751 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:50.844945 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:50.858267 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:50.869688 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:50.870015 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:50.870672 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:50.871841 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:50.872188 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:50.872956 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:50.872977 [27403]: Thawing priority 1
2012/10/05 15:36:50.872984 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:50.873004 [27403]: Thawing priority 2
2012/10/05 15:36:50.873011 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:50.873021 [27403]: Thawing priority 3
2012/10/05 15:36:50.873028 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:50.886260 [set_recmode:32252]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:50.886332 [set_recmode:32252]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:51.031667 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:51.031743 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:51.826526 [27403]: Freeze priority 1
2012/10/05 15:36:51.827293 [27403]: Freeze priority 2
2012/10/05 15:36:51.828121 [27403]: Freeze priority 3
2012/10/05 15:36:51.829279 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:51.829310 [27403]: Monitoring has been disabled
2012/10/05 15:36:51.829319 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:51.852522 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:51.889384 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:51.889950 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:51.890299 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:51.890915 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:51.892118 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:51.892452 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:51.893319 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:51.893339 [27403]: Thawing priority 1
2012/10/05 15:36:51.893346 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:51.893360 [27403]: Thawing priority 2
2012/10/05 15:36:51.893366 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:51.893376 [27403]: Thawing priority 3
2012/10/05 15:36:51.893381 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:51.994009 [set_recmode:32318]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:51.994089 [set_recmode:32318]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:52.033812 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:52.033887 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:52.807923 [27403]: Freeze priority 1
2012/10/05 15:36:52.808691 [27403]: Freeze priority 2
2012/10/05 15:36:52.809351 [27403]: Freeze priority 3
2012/10/05 15:36:52.810435 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:52.810465 [27403]: Monitoring has been disabled
2012/10/05 15:36:52.810475 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:52.853082 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:52.859941 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:52.860785 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:52.861180 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:52.861794 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:52.863101 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:52.863432 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:52.864233 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:52.864252 [27403]: Thawing priority 1
2012/10/05 15:36:52.864259 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:52.864276 [27403]: Thawing priority 2
2012/10/05 15:36:52.864282 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:52.864292 [27403]: Thawing priority 3
2012/10/05 15:36:52.864298 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:52.876583 [set_recmode:32384]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:52.876640 [set_recmode:32384]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:53.035869 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:53.036131 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:53.810827 [27403]: Freeze priority 1
2012/10/05 15:36:53.811534 [27403]: Freeze priority 2
2012/10/05 15:36:53.812185 [27403]: Freeze priority 3
2012/10/05 15:36:53.813225 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:53.813261 [27403]: Monitoring has been disabled
2012/10/05 15:36:53.813272 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:53.854104 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:53.872912 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:53.873434 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:53.873764 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:53.874296 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:53.875311 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:53.875615 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:53.876448 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:53.876465 [27403]: Thawing priority 1
2012/10/05 15:36:53.876472 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:53.876486 [27403]: Thawing priority 2
2012/10/05 15:36:53.876492 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:53.876501 [27403]: Thawing priority 3
2012/10/05 15:36:53.876507 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:53.917331 [set_recmode:32451]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:53.917400 [set_recmode:32451]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:54.037907 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:54.037983 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:54.829992 [27403]: Freeze priority 1
2012/10/05 15:36:54.831215 [27403]: Freeze priority 2
2012/10/05 15:36:54.832405 [27403]: Freeze priority 3
2012/10/05 15:36:54.833583 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:54.833626 [27403]: Monitoring has been disabled
2012/10/05 15:36:54.833636 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:54.854695 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:54.885926 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:54.892919 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:54.893306 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:54.893824 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:54.894803 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:54.895070 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:54.895777 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:54.895797 [27403]: Thawing priority 1
2012/10/05 15:36:54.895803 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:54.895821 [27403]: Thawing priority 2
2012/10/05 15:36:54.895827 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:54.895837 [27403]: Thawing priority 3
2012/10/05 15:36:54.895843 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:54.925172 [set_recmode:32517]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:54.925251 [set_recmode:32517]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:55.040033 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:55.040112 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:55.812331 [27403]: Freeze priority 1
2012/10/05 15:36:55.813112 [27403]: Freeze priority 2
2012/10/05 15:36:55.813823 [27403]: Freeze priority 3
2012/10/05 15:36:55.814997 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:55.815030 [27403]: Monitoring has been disabled
2012/10/05 15:36:55.815039 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:55.855646 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:55.864324 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:55.864863 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:55.865197 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:55.865798 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:55.866984 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:55.867327 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:55.868142 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:55.868161 [27403]: Thawing priority 1
2012/10/05 15:36:55.868168 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:55.868183 [27403]: Thawing priority 2
2012/10/05 15:36:55.868190 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:55.868199 [27403]: Thawing priority 3
2012/10/05 15:36:55.868206 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:55.886755 [set_recmode:32583]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:55.886817 [set_recmode:32583]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:56.042002 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:56.042082 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:56.856942 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:56.873337 [27403]: Freeze priority 1
2012/10/05 15:36:56.874065 [27403]: Freeze priority 2
2012/10/05 15:36:56.874729 [27403]: Freeze priority 3
2012/10/05 15:36:56.875833 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:56.875868 [27403]: Monitoring has been disabled
2012/10/05 15:36:56.875878 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:56.924454 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:56.924999 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:56.925316 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:56.925819 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:56.926771 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:56.927046 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:56.927945 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:56.927972 [27403]: Thawing priority 1
2012/10/05 15:36:56.927978 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:56.927994 [27403]: Thawing priority 2
2012/10/05 15:36:56.928000 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:56.928009 [27403]: Thawing priority 3
2012/10/05 15:36:56.928015 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:56.972698 [set_recmode:32649]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:56.972773 [set_recmode:32649]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:57.044148 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:57.044239 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:57.816819 [27403]: Freeze priority 1
2012/10/05 15:36:57.817650 [27403]: Freeze priority 2
2012/10/05 15:36:57.818387 [27403]: Freeze priority 3
2012/10/05 15:36:57.819563 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:57.819600 [27403]: Monitoring has been disabled
2012/10/05 15:36:57.819609 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:57.857112 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:57.869480 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:57.869979 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:57.870300 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:57.870886 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:57.872065 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:57.872386 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:57.873197 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:57.873215 [27403]: Thawing priority 1
2012/10/05 15:36:57.873221 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:57.873235 [27403]: Thawing priority 2
2012/10/05 15:36:57.873241 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:57.873251 [27403]: Thawing priority 3
2012/10/05 15:36:57.873257 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:57.886446 [set_recmode:32715]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:57.886501 [set_recmode:32715]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:58.046211 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:58.046478 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:58.824135 [27403]: Freeze priority 1
2012/10/05 15:36:58.824919 [27403]: Freeze priority 2
2012/10/05 15:36:58.825662 [27403]: Freeze priority 3
2012/10/05 15:36:58.826876 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:58.826921 [27403]: Monitoring has been disabled
2012/10/05 15:36:58.826946 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:58.857938 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:58.886715 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:58.887267 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:58.887628 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:58.888238 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:58.889435 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:58.889776 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:58.890688 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:58.890708 [27403]: Thawing priority 1
2012/10/05 15:36:58.890715 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:58.890731 [27403]: Thawing priority 2
2012/10/05 15:36:58.890738 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:58.890748 [27403]: Thawing priority 3
2012/10/05 15:36:58.890754 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:58.932021 [set_recmode: 313]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:58.932087 [set_recmode: 313]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:36:59.048197 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:36:59.048274 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:59.819121 [27403]: Freeze priority 1
2012/10/05 15:36:59.819860 [27403]: Freeze priority 2
2012/10/05 15:36:59.820558 [27403]: Freeze priority 3
2012/10/05 15:36:59.821680 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:59.821717 [27403]: Monitoring has been disabled
2012/10/05 15:36:59.821726 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:59.858521 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:36:59.869965 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:59.870509 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:59.870831 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:59.871478 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:59.872799 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:59.873123 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:59.873921 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:59.873940 [27403]: Thawing priority 1
2012/10/05 15:36:59.873947 [27403]: Release freeze handler for prio 1
2012/10/05 15:36:59.873962 [27403]: Thawing priority 2
2012/10/05 15:36:59.873969 [27403]: Release freeze handler for prio 2
2012/10/05 15:36:59.873978 [27403]: Thawing priority 3
2012/10/05 15:36:59.873985 [27403]: Release freeze handler for prio 3
2012/10/05 15:36:59.886265 [set_recmode: 382]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:59.886341 [set_recmode: 382]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:00.050325 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:00.050406 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:00.836448 [27403]: Freeze priority 1
2012/10/05 15:37:00.837177 [27403]: Freeze priority 2
2012/10/05 15:37:00.837825 [27403]: Freeze priority 3
2012/10/05 15:37:00.838893 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:00.838928 [27403]: Monitoring has been disabled
2012/10/05 15:37:00.838938 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:00.859092 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:00.886684 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:00.887178 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:00.887542 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:00.888191 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:00.889401 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:00.889729 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:00.890618 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:00.890649 [27403]: Thawing priority 1
2012/10/05 15:37:00.890656 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:00.890675 [27403]: Thawing priority 2
2012/10/05 15:37:00.890682 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:00.890692 [27403]: Thawing priority 3
2012/10/05 15:37:00.890698 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:00.945995 [set_recmode: 449]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:00.946096 [set_recmode: 449]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:01.052396 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:01.052472 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:01.822559 [27403]: Freeze priority 1
2012/10/05 15:37:01.823354 [27403]: Freeze priority 2
2012/10/05 15:37:01.824025 [27403]: Freeze priority 3
2012/10/05 15:37:01.825169 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:01.825210 [27403]: Monitoring has been disabled
2012/10/05 15:37:01.825219 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:01.865687 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:01.899945 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:01.900574 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:01.900954 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:01.901576 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:01.902817 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:01.903163 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:01.904044 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:01.904062 [27403]: Thawing priority 1
2012/10/05 15:37:01.904069 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:01.904084 [27403]: Thawing priority 2
2012/10/05 15:37:01.904091 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:01.904101 [27403]: Thawing priority 3
2012/10/05 15:37:01.904107 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:01.930561 [set_recmode: 525]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:01.930650 [set_recmode: 525]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:02.054435 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:02.054512 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:02.824194 [27403]: Freeze priority 1
2012/10/05 15:37:02.825030 [27403]: Freeze priority 2
2012/10/05 15:37:02.825668 [27403]: Freeze priority 3
2012/10/05 15:37:02.826727 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:02.826760 [27403]: Monitoring has been disabled
2012/10/05 15:37:02.826769 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:02.866725 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:02.874966 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:02.884975 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:02.885318 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:02.885968 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:02.887097 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:02.887417 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:02.888216 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:02.888234 [27403]: Thawing priority 1
2012/10/05 15:37:02.888241 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:02.888258 [27403]: Thawing priority 2
2012/10/05 15:37:02.888265 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:02.888274 [27403]: Thawing priority 3
2012/10/05 15:37:02.888280 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:02.911254 [set_recmode: 592]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:02.911330 [set_recmode: 592]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:03.056522 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:03.056795 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:03.854791 [27403]: Freeze priority 1
2012/10/05 15:37:03.855507 [27403]: Freeze priority 2
2012/10/05 15:37:03.856120 [27403]: Freeze priority 3
2012/10/05 15:37:03.857180 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:03.857214 [27403]: Monitoring has been disabled
2012/10/05 15:37:03.857223 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:03.870164 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:03.915198 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:03.915799 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:03.916100 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:03.916683 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:03.917832 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:03.918223 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:03.919358 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:03.919392 [27403]: Thawing priority 1
2012/10/05 15:37:03.919399 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:03.919418 [27403]: Thawing priority 2
2012/10/05 15:37:03.919425 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:03.919434 [27403]: Thawing priority 3
2012/10/05 15:37:03.919440 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:03.963347 [set_recmode: 658]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:03.963427 [set_recmode: 658]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:04.059055 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:04.059134 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:04.827974 [27403]: Freeze priority 1
2012/10/05 15:37:04.828720 [27403]: Freeze priority 2
2012/10/05 15:37:04.829407 [27403]: Freeze priority 3
2012/10/05 15:37:04.830529 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:04.830566 [27403]: Monitoring has been disabled
2012/10/05 15:37:04.830576 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:04.870957 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:04.876958 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:04.889921 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:04.890261 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:04.890840 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:04.891936 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:04.892244 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:04.893020 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:04.893040 [27403]: Thawing priority 1
2012/10/05 15:37:04.893047 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:04.893066 [27403]: Thawing priority 2
2012/10/05 15:37:04.893073 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:04.893083 [27403]: Thawing priority 3
2012/10/05 15:37:04.893090 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:04.905614 [set_recmode: 724]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:04.905672 [set_recmode: 724]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:05.061169 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:05.061249 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:05.829099 [27403]: Freeze priority 1
2012/10/05 15:37:05.829860 [27403]: Freeze priority 2
2012/10/05 15:37:05.830564 [27403]: Freeze priority 3
2012/10/05 15:37:05.831691 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:05.831734 [27403]: Monitoring has been disabled
2012/10/05 15:37:05.831743 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:05.871390 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:05.878169 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:05.880179 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:05.880584 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:05.881207 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:05.882420 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:05.882747 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:05.883556 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:05.883576 [27403]: Thawing priority 1
2012/10/05 15:37:05.883583 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:05.883600 [27403]: Thawing priority 2
2012/10/05 15:37:05.883607 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:05.883617 [27403]: Thawing priority 3
2012/10/05 15:37:05.883623 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:05.926438 [set_recmode: 794]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:05.926500 [set_recmode: 794]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:06.063203 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:06.063283 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:06.851699 [27403]: Freeze priority 1
2012/10/05 15:37:06.852426 [27403]: Freeze priority 2
2012/10/05 15:37:06.853085 [27403]: Freeze priority 3
2012/10/05 15:37:06.854200 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:06.854230 [27403]: Monitoring has been disabled
2012/10/05 15:37:06.854240 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:06.871474 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:06.902152 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:06.902661 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:06.903027 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:06.903684 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:06.904823 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:06.905187 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:06.906054 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:06.906072 [27403]: Thawing priority 1
2012/10/05 15:37:06.906078 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:06.906092 [27403]: Thawing priority 2
2012/10/05 15:37:06.906098 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:06.906107 [27403]: Thawing priority 3
2012/10/05 15:37:06.906113 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:06.950184 [set_recmode: 860]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:06.950265 [set_recmode: 860]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:07.065250 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:07.065328 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:07.832343 [27403]: Freeze priority 1
2012/10/05 15:37:07.833052 [27403]: Freeze priority 2
2012/10/05 15:37:07.833677 [27403]: Freeze priority 3
2012/10/05 15:37:07.834748 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:07.834784 [27403]: Monitoring has been disabled
2012/10/05 15:37:07.834792 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:07.871579 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:07.882290 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:07.883606 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:07.883971 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:07.884608 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:07.885930 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:07.886262 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:07.887019 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:07.887049 [27403]: Thawing priority 1
2012/10/05 15:37:07.887057 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:07.887073 [27403]: Thawing priority 2
2012/10/05 15:37:07.887080 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:07.887090 [27403]: Thawing priority 3
2012/10/05 15:37:07.887097 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:07.899694 [set_recmode: 928]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:07.899748 [set_recmode: 928]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:08.067337 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:08.067593 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:08.834621 [27403]: Freeze priority 1
2012/10/05 15:37:08.835393 [27403]: Freeze priority 2
2012/10/05 15:37:08.836046 [27403]: Freeze priority 3
2012/10/05 15:37:08.837164 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:08.837193 [27403]: Monitoring has been disabled
2012/10/05 15:37:08.837210 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:08.872111 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:08.892841 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:08.893368 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:08.893698 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:08.894266 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:08.895298 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:08.895652 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:08.896531 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:08.896549 [27403]: Thawing priority 1
2012/10/05 15:37:08.896556 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:08.896570 [27403]: Thawing priority 2
2012/10/05 15:37:08.896577 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:08.896602 [27403]: Thawing priority 3
2012/10/05 15:37:08.896610 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:08.937774 [set_recmode: 994]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:08.937841 [set_recmode: 994]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:09.069446 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:09.069528 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:09.840828 [27403]: Freeze priority 1
2012/10/05 15:37:09.841599 [27403]: Freeze priority 2
2012/10/05 15:37:09.842311 [27403]: Freeze priority 3
2012/10/05 15:37:09.843422 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:09.843461 [27403]: Monitoring has been disabled
2012/10/05 15:37:09.843470 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:09.872213 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:09.920985 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:09.921527 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:09.921842 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:09.922371 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:09.923360 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:09.923664 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:09.924546 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:09.924563 [27403]: Thawing priority 1
2012/10/05 15:37:09.924570 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:09.924583 [27403]: Thawing priority 2
2012/10/05 15:37:09.924589 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:09.924598 [27403]: Thawing priority 3
2012/10/05 15:37:09.924604 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:09.967346 [set_recmode: 1069]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:09.967431 [set_recmode: 1069]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:10.071452 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:10.071528 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:10.837185 [27403]: Freeze priority 1
2012/10/05 15:37:10.837908 [27403]: Freeze priority 2
2012/10/05 15:37:10.838532 [27403]: Freeze priority 3
2012/10/05 15:37:10.839607 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:10.839635 [27403]: Monitoring has been disabled
2012/10/05 15:37:10.839644 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:10.873267 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:10.886434 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:10.898484 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:10.898848 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:10.899417 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:10.900517 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:10.900851 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:10.901605 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:10.901627 [27403]: Thawing priority 1
2012/10/05 15:37:10.901634 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:10.901654 [27403]: Thawing priority 2
2012/10/05 15:37:10.901660 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:10.901671 [27403]: Thawing priority 3
2012/10/05 15:37:10.901677 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:10.914904 [set_recmode: 1136]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:10.914964 [set_recmode: 1136]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:11.073550 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:11.073646 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:11.853951 [27403]: Freeze priority 1
2012/10/05 15:37:11.854716 [27403]: Freeze priority 2
2012/10/05 15:37:11.855411 [27403]: Freeze priority 3
2012/10/05 15:37:11.856526 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:11.856556 [27403]: Monitoring has been disabled
2012/10/05 15:37:11.856574 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:11.873575 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:11.916111 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:11.916619 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:11.916958 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:11.917545 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:11.918674 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:11.919019 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:11.919856 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:11.919875 [27403]: Thawing priority 1
2012/10/05 15:37:11.919882 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:11.919896 [27403]: Thawing priority 2
2012/10/05 15:37:11.919903 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:11.919912 [27403]: Thawing priority 3
2012/10/05 15:37:11.919918 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:11.962354 [set_recmode: 1203]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:11.962432 [set_recmode: 1203]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:12.075547 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:12.075623 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:12.839611 [27403]: Freeze priority 1
2012/10/05 15:37:12.840390 [27403]: Freeze priority 2
2012/10/05 15:37:12.841082 [27403]: Freeze priority 3
2012/10/05 15:37:12.842153 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:12.842182 [27403]: Monitoring has been disabled
2012/10/05 15:37:12.842191 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:12.873785 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:12.888061 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:12.890276 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:12.890608 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:12.891135 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:12.892119 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:12.892404 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:12.893227 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:12.893246 [27403]: Thawing priority 1
2012/10/05 15:37:12.893252 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:12.893267 [27403]: Thawing priority 2
2012/10/05 15:37:12.893274 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:12.893283 [27403]: Thawing priority 3
2012/10/05 15:37:12.893289 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:12.906712 [set_recmode: 1269]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:12.906764 [set_recmode: 1269]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:13.077643 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:13.077898 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:13.847560 [27403]: Freeze priority 1
2012/10/05 15:37:13.848280 [27403]: Freeze priority 2
2012/10/05 15:37:13.848958 [27403]: Freeze priority 3
2012/10/05 15:37:13.850017 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:13.850057 [27403]: Monitoring has been disabled
2012/10/05 15:37:13.850076 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:13.874857 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:13.896430 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:13.897719 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:13.898038 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:13.898583 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:13.899577 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:13.899852 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:13.900603 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:13.900619 [27403]: Thawing priority 1
2012/10/05 15:37:13.900625 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:13.900640 [27403]: Thawing priority 2
2012/10/05 15:37:13.900646 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:13.900655 [27403]: Thawing priority 3
2012/10/05 15:37:13.900661 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:13.989645 [set_recmode: 1336]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:13.989718 [set_recmode: 1336]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:14.079706 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:14.079782 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:14.843055 [27403]: Freeze priority 1
2012/10/05 15:37:14.843784 [27403]: Freeze priority 2
2012/10/05 15:37:14.844401 [27403]: Freeze priority 3
2012/10/05 15:37:14.845429 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:14.845460 [27403]: Monitoring has been disabled
2012/10/05 15:37:14.845480 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:14.875492 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:14.903939 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:14.904492 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:14.904806 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:14.905387 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:14.906525 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:14.906840 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:14.907656 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:14.907677 [27403]: Thawing priority 1
2012/10/05 15:37:14.907685 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:14.907700 [27403]: Thawing priority 2
2012/10/05 15:37:14.907707 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:14.907717 [27403]: Thawing priority 3
2012/10/05 15:37:14.907723 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:14.919964 [set_recmode: 1404]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:14.920022 [set_recmode: 1404]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:15.081789 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:15.081870 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:15.844293 [27403]: Freeze priority 1
2012/10/05 15:37:15.844986 [27403]: Freeze priority 2
2012/10/05 15:37:15.845636 [27403]: Freeze priority 3
2012/10/05 15:37:15.846672 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:15.846698 [27403]: Monitoring has been disabled
2012/10/05 15:37:15.846707 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:15.876066 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:15.892290 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:15.910067 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:15.910416 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:15.911002 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:15.912226 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:15.912579 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:15.913362 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:15.913382 [27403]: Thawing priority 1
2012/10/05 15:37:15.913388 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:15.913406 [27403]: Thawing priority 2
2012/10/05 15:37:15.913412 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:15.913422 [27403]: Thawing priority 3
2012/10/05 15:37:15.913428 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:15.956921 [set_recmode: 1470]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:15.956994 [set_recmode: 1470]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:16.083790 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:16.083860 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:16.867548 [27403]: Freeze priority 1
2012/10/05 15:37:16.868308 [27403]: Freeze priority 2
2012/10/05 15:37:16.868975 [27403]: Freeze priority 3
2012/10/05 15:37:16.870090 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:16.870141 [27403]: Monitoring has been disabled
2012/10/05 15:37:16.870151 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:16.876619 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:16.916582 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:16.917632 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:16.917968 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:16.918521 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:16.919603 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:16.919942 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:16.920786 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:16.920808 [27403]: Thawing priority 1
2012/10/05 15:37:16.920814 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:16.920831 [27403]: Thawing priority 2
2012/10/05 15:37:16.920837 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:16.920846 [27403]: Thawing priority 3
2012/10/05 15:37:16.920852 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:16.938843 [set_recmode: 1536]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:16.938901 [set_recmode: 1536]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:17.085942 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:17.086026 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:17.848026 [27403]: Freeze priority 1
2012/10/05 15:37:17.848834 [27403]: Freeze priority 2
2012/10/05 15:37:17.849519 [27403]: Freeze priority 3
2012/10/05 15:37:17.850683 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:17.850726 [27403]: Monitoring has been disabled
2012/10/05 15:37:17.850736 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:17.876881 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:17.901475 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:17.901929 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:17.902237 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:17.902834 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:17.903851 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:17.904138 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:17.904917 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:17.904935 [27403]: Thawing priority 1
2012/10/05 15:37:17.904942 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:17.904956 [27403]: Thawing priority 2
2012/10/05 15:37:17.904962 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:17.904972 [27403]: Thawing priority 3
2012/10/05 15:37:17.904978 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:17.934868 [set_recmode: 1603]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:17.934940 [set_recmode: 1603]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:18.087963 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:18.088299 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:18.849262 [27403]: Freeze priority 1
2012/10/05 15:37:18.850041 [27403]: Freeze priority 2
2012/10/05 15:37:18.850748 [27403]: Freeze priority 3
2012/10/05 15:37:18.851836 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:18.851866 [27403]: Monitoring has been disabled
2012/10/05 15:37:18.851875 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:18.877529 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:18.910974 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:18.911571 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:18.911943 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:18.912544 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:18.913543 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:18.913857 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:18.914722 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:18.914749 [27403]: Thawing priority 1
2012/10/05 15:37:18.914756 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:18.914771 [27403]: Thawing priority 2
2012/10/05 15:37:18.914777 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:18.914786 [27403]: Thawing priority 3
2012/10/05 15:37:18.914792 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:18.948129 [set_recmode: 1669]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:18.948190 [set_recmode: 1669]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:19.090533 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:19.090608 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:19.856836 [27403]: Freeze priority 1
2012/10/05 15:37:19.857620 [27403]: Freeze priority 2
2012/10/05 15:37:19.858294 [27403]: Freeze priority 3
2012/10/05 15:37:19.859401 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:19.859431 [27403]: Monitoring has been disabled
2012/10/05 15:37:19.859441 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:19.878329 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:19.907795 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:19.908286 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:19.908627 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:19.909222 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:19.910334 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:19.910659 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:19.911503 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:19.911523 [27403]: Thawing priority 1
2012/10/05 15:37:19.911530 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:19.911546 [27403]: Thawing priority 2
2012/10/05 15:37:19.911553 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:19.911563 [27403]: Thawing priority 3
2012/10/05 15:37:19.911570 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:19.929795 [set_recmode: 1737]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:19.929862 [set_recmode: 1737]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:20.092633 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:20.092712 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:20.868613 [27403]: Freeze priority 1
2012/10/05 15:37:20.869407 [27403]: Freeze priority 2
2012/10/05 15:37:20.870102 [27403]: Freeze priority 3
2012/10/05 15:37:20.871221 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:20.871252 [27403]: Monitoring has been disabled
2012/10/05 15:37:20.871268 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:20.878589 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:20.918188 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:20.919941 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:20.920299 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:20.920913 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:20.922124 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:20.922462 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:20.923307 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:20.923328 [27403]: Thawing priority 1
2012/10/05 15:37:20.923335 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:20.923353 [27403]: Thawing priority 2
2012/10/05 15:37:20.923360 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:20.923383 [27403]: Thawing priority 3
2012/10/05 15:37:20.923393 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:20.965720 [set_recmode: 1803]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:20.965801 [set_recmode: 1803]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:21.093743 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:21.093822 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:21.855283 [27403]: Freeze priority 1
2012/10/05 15:37:21.856014 [27403]: Freeze priority 2
2012/10/05 15:37:21.856679 [27403]: Freeze priority 3
2012/10/05 15:37:21.857692 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:21.857721 [27403]: Monitoring has been disabled
2012/10/05 15:37:21.857729 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:21.878643 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:21.903253 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:21.903813 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:21.904159 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:21.904801 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:21.905981 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:21.906308 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:21.907086 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:21.907103 [27403]: Thawing priority 1
2012/10/05 15:37:21.907110 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:21.907124 [27403]: Thawing priority 2
2012/10/05 15:37:21.907130 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:21.907140 [27403]: Thawing priority 3
2012/10/05 15:37:21.907145 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:21.919160 [set_recmode: 1871]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:21.919212 [set_recmode: 1871]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:22.095699 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:22.095782 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:22.862377 [27403]: Freeze priority 1
2012/10/05 15:37:22.863129 [27403]: Freeze priority 2
2012/10/05 15:37:22.863798 [27403]: Freeze priority 3
2012/10/05 15:37:22.864892 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:22.864922 [27403]: Monitoring has been disabled
2012/10/05 15:37:22.864932 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:22.879063 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:22.913113 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:22.913599 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:22.913936 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:22.914537 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:22.915610 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:22.915943 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:22.916786 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:22.916805 [27403]: Thawing priority 1
2012/10/05 15:37:22.916812 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:22.916827 [27403]: Thawing priority 2
2012/10/05 15:37:22.916833 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:22.916841 [27403]: Thawing priority 3
2012/10/05 15:37:22.916847 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:22.958612 [set_recmode: 1937]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:22.958684 [set_recmode: 1937]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:23.097800 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:23.098067 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:23.858664 [27403]: Freeze priority 1
2012/10/05 15:37:23.859417 [27403]: Freeze priority 2
2012/10/05 15:37:23.860098 [27403]: Freeze priority 3
2012/10/05 15:37:23.861257 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:23.861288 [27403]: Monitoring has been disabled
2012/10/05 15:37:23.861304 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:23.879597 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:23.922280 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:23.922863 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:23.923204 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:23.923913 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:23.925101 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:23.925413 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:23.926239 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:23.926257 [27403]: Thawing priority 1
2012/10/05 15:37:23.926263 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:23.926278 [27403]: Thawing priority 2
2012/10/05 15:37:23.926284 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:23.926293 [27403]: Thawing priority 3
2012/10/05 15:37:23.926299 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:24.008524 [set_recmode: 2003]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:24.008610 [set_recmode: 2003]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:24.099838 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:24.099910 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:24.858225 [27403]: Freeze priority 1
2012/10/05 15:37:24.858942 [27403]: Freeze priority 2
2012/10/05 15:37:24.859576 [27403]: Freeze priority 3
2012/10/05 15:37:24.860625 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:24.860689 [27403]: Monitoring has been disabled
2012/10/05 15:37:24.860702 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:24.880641 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:24.906974 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:24.908971 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:24.909327 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:24.909964 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:24.911216 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:24.911560 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:24.912349 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:24.912368 [27403]: Thawing priority 1
2012/10/05 15:37:24.912374 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:24.912390 [27403]: Thawing priority 2
2012/10/05 15:37:24.912396 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:24.912406 [27403]: Thawing priority 3
2012/10/05 15:37:24.912411 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:24.931155 [set_recmode: 2069]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:24.931207 [set_recmode: 2069]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:25.101977 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:25.102056 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:25.881798 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:25.883387 [27403]: Freeze priority 1
2012/10/05 15:37:25.884100 [27403]: Freeze priority 2
2012/10/05 15:37:25.884777 [27403]: Freeze priority 3
2012/10/05 15:37:25.885832 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:25.885861 [27403]: Monitoring has been disabled
2012/10/05 15:37:25.885869 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:25.932823 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:25.933561 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:25.933934 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:25.934510 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:25.935686 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:25.936021 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:25.936857 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:25.936875 [27403]: Thawing priority 1
2012/10/05 15:37:25.936881 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:25.936896 [27403]: Thawing priority 2
2012/10/05 15:37:25.936902 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:25.936911 [27403]: Thawing priority 3
2012/10/05 15:37:25.936917 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:25.980785 [set_recmode: 2136]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:25.980864 [set_recmode: 2136]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:26.104060 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:26.104139 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:26.863244 [27403]: Freeze priority 1
2012/10/05 15:37:26.864040 [27403]: Freeze priority 2
2012/10/05 15:37:26.864718 [27403]: Freeze priority 3
2012/10/05 15:37:26.865802 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:26.865833 [27403]: Monitoring has been disabled
2012/10/05 15:37:26.865842 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:26.882774 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:26.914687 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:26.915198 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:26.915553 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:26.916144 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:26.917373 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:26.917695 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:26.918502 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:26.918521 [27403]: Thawing priority 1
2012/10/05 15:37:26.918528 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:26.918543 [27403]: Thawing priority 2
2012/10/05 15:37:26.918550 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:26.918560 [27403]: Thawing priority 3
2012/10/05 15:37:26.918566 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:26.931015 [set_recmode: 2202]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:26.931072 [set_recmode: 2202]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:27.106117 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:27.106198 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:27.864443 [27403]: Freeze priority 1
2012/10/05 15:37:27.865216 [27403]: Freeze priority 2
2012/10/05 15:37:27.865904 [27403]: Freeze priority 3
2012/10/05 15:37:27.867037 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:27.867067 [27403]: Monitoring has been disabled
2012/10/05 15:37:27.867076 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:27.883355 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:27.927886 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:27.928436 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:27.928736 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:27.929293 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:27.930307 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:27.930612 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:27.931480 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:27.931502 [27403]: Thawing priority 1
2012/10/05 15:37:27.931509 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:27.931527 [27403]: Thawing priority 2
2012/10/05 15:37:27.931533 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:27.931543 [27403]: Thawing priority 3
2012/10/05 15:37:27.931550 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:27.973235 [set_recmode: 2268]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:27.973305 [set_recmode: 2268]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:28.108076 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:28.108331 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:28.872644 [27403]: Freeze priority 1
2012/10/05 15:37:28.873420 [27403]: Freeze priority 2
2012/10/05 15:37:28.874116 [27403]: Freeze priority 3
2012/10/05 15:37:28.875219 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:28.875254 [27403]: Monitoring has been disabled
2012/10/05 15:37:28.875263 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:28.884421 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:28.921182 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:28.922722 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:28.923063 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:28.923616 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:28.924709 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:28.925031 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:28.925859 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:28.925877 [27403]: Thawing priority 1
2012/10/05 15:37:28.925884 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:28.925912 [27403]: Thawing priority 2
2012/10/05 15:37:28.925921 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:28.925932 [27403]: Thawing priority 3
2012/10/05 15:37:28.925939 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:28.938814 [set_recmode: 2334]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:28.938876 [set_recmode: 2334]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:29.110155 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:29.110236 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:29.883446 [27403]: Freeze priority 1
2012/10/05 15:37:29.884186 [27403]: Freeze priority 2
2012/10/05 15:37:29.884813 [27403]: Freeze priority 3
2012/10/05 15:37:29.884920 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:29.885883 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:29.885906 [27403]: Monitoring has been disabled
2012/10/05 15:37:29.885921 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:29.931625 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:29.932162 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:29.932443 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:29.933020 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:29.934037 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:29.934327 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:29.935624 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:29.935650 [27403]: Thawing priority 1
2012/10/05 15:37:29.935657 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:29.935673 [27403]: Thawing priority 2
2012/10/05 15:37:29.935680 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:29.935689 [27403]: Thawing priority 3
2012/10/05 15:37:29.935739 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:29.979219 [set_recmode: 2400]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:29.979289 [set_recmode: 2400]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:30.112237 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:30.112313 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:30.872120 [27403]: Freeze priority 1
2012/10/05 15:37:30.872844 [27403]: Freeze priority 2
2012/10/05 15:37:30.873471 [27403]: Freeze priority 3
2012/10/05 15:37:30.874565 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:30.874594 [27403]: Monitoring has been disabled
2012/10/05 15:37:30.874604 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:30.885265 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:30.921917 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:30.922406 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:30.922798 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:30.923448 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:30.924590 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:30.924946 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:30.925802 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:30.925823 [27403]: Thawing priority 1
2012/10/05 15:37:30.925830 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:30.925845 [27403]: Thawing priority 2
2012/10/05 15:37:30.925851 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:30.925860 [27403]: Thawing priority 3
2012/10/05 15:37:30.925866 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:30.938367 [set_recmode: 2466]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:30.938426 [set_recmode: 2466]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:31.114330 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:31.114405 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:31.877524 [27403]: Freeze priority 1
2012/10/05 15:37:31.878265 [27403]: Freeze priority 2
2012/10/05 15:37:31.878900 [27403]: Freeze priority 3
2012/10/05 15:37:31.879940 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:31.879975 [27403]: Monitoring has been disabled
2012/10/05 15:37:31.879991 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:31.885629 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:31.927513 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:31.928036 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:31.928390 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:31.928989 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:31.930086 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:31.930400 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:31.931197 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:31.931216 [27403]: Thawing priority 1
2012/10/05 15:37:31.931223 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:31.931238 [27403]: Thawing priority 2
2012/10/05 15:37:31.931245 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:31.931254 [27403]: Thawing priority 3
2012/10/05 15:37:31.931260 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:31.973748 [set_recmode: 2532]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:31.973819 [set_recmode: 2532]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:32.116419 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:32.116500 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:32.874498 [27403]: Freeze priority 1
2012/10/05 15:37:32.875234 [27403]: Freeze priority 2
2012/10/05 15:37:32.875858 [27403]: Freeze priority 3
2012/10/05 15:37:32.876899 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:32.876934 [27403]: Monitoring has been disabled
2012/10/05 15:37:32.876944 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:32.886182 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:32.926763 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:32.927323 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:32.927755 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:32.928479 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:32.929576 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:32.929879 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:32.930710 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:32.930730 [27403]: Thawing priority 1
2012/10/05 15:37:32.930737 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:32.930755 [27403]: Thawing priority 2
2012/10/05 15:37:32.930762 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:32.930772 [27403]: Thawing priority 3
2012/10/05 15:37:32.930779 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:32.947130 [set_recmode: 2598]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:32.947233 [set_recmode: 2598]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:33.118483 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:33.118735 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:33.876443 [27403]: Freeze priority 1
2012/10/05 15:37:33.877196 [27403]: Freeze priority 2
2012/10/05 15:37:33.877895 [27403]: Freeze priority 3
2012/10/05 15:37:33.878985 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:33.879018 [27403]: Monitoring has been disabled
2012/10/05 15:37:33.879028 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:33.886262 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:33.925077 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:33.927136 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:33.927518 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:33.928105 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:33.929273 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:33.929646 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:33.930529 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:33.930549 [27403]: Thawing priority 1
2012/10/05 15:37:33.930556 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:33.930574 [27403]: Thawing priority 2
2012/10/05 15:37:33.930581 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:33.930591 [27403]: Thawing priority 3
2012/10/05 15:37:33.930597 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:33.969511 [set_recmode: 2664]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:33.969591 [set_recmode: 2664]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:34.120532 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:34.120611 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:34.887430 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:34.898723 [27403]: Freeze priority 1
2012/10/05 15:37:34.899499 [27403]: Freeze priority 2
2012/10/05 15:37:34.900189 [27403]: Freeze priority 3
2012/10/05 15:37:34.901314 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:34.901350 [27403]: Monitoring has been disabled
2012/10/05 15:37:34.901360 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:34.947604 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:34.959626 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:34.960014 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:34.960612 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:34.961763 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:34.962079 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:34.962873 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:34.962892 [27403]: Thawing priority 1
2012/10/05 15:37:34.962898 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:34.962917 [27403]: Thawing priority 2
2012/10/05 15:37:34.962923 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:34.962933 [27403]: Thawing priority 3
2012/10/05 15:37:34.962939 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:35.006852 [set_recmode: 2730]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:35.006915 [set_recmode: 2730]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:35.122524 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:35.122595 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:35.879083 [27403]: Freeze priority 1
2012/10/05 15:37:35.879842 [27403]: Freeze priority 2
2012/10/05 15:37:35.880557 [27403]: Freeze priority 3
2012/10/05 15:37:35.881663 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:35.881696 [27403]: Monitoring has been disabled
2012/10/05 15:37:35.881705 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:35.888381 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:35.930056 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:35.930668 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:35.931028 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:35.931654 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:35.932805 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:35.933632 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:35.934436 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:35.934454 [27403]: Thawing priority 1
2012/10/05 15:37:35.934461 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:35.934476 [27403]: Thawing priority 2
2012/10/05 15:37:35.934482 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:35.934492 [27403]: Thawing priority 3
2012/10/05 15:37:35.934498 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:35.946835 [set_recmode: 2796]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:35.946919 [set_recmode: 2796]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:36.124704 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:36.124791 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:36.880596 [27403]: Freeze priority 1
2012/10/05 15:37:36.881331 [27403]: Freeze priority 2
2012/10/05 15:37:36.882004 [27403]: Freeze priority 3
2012/10/05 15:37:36.883085 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:36.883117 [27403]: Monitoring has been disabled
2012/10/05 15:37:36.883127 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:36.889445 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:36.975894 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:36.976464 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:36.976818 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:36.977419 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:36.978599 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:36.978917 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:36.979781 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:36.979801 [27403]: Thawing priority 1
2012/10/05 15:37:36.979808 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:36.979824 [27403]: Thawing priority 2
2012/10/05 15:37:36.979831 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:36.979841 [27403]: Thawing priority 3
2012/10/05 15:37:36.979847 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:37.020723 [set_recmode: 2862]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:37.020784 [set_recmode: 2862]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:37.126718 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:37.126795 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:37.882062 [27403]: Freeze priority 1
2012/10/05 15:37:37.882815 [27403]: Freeze priority 2
2012/10/05 15:37:37.883459 [27403]: Freeze priority 3
2012/10/05 15:37:37.884559 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:37.884586 [27403]: Monitoring has been disabled
2012/10/05 15:37:37.884595 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:37.890011 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:37.939171 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:37.939730 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:37.940069 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:37.940674 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:37.941799 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:37.942123 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:37.943040 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:37.943057 [27403]: Thawing priority 1
2012/10/05 15:37:37.943064 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:37.943078 [27403]: Thawing priority 2
2012/10/05 15:37:37.943085 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:37.943094 [27403]: Thawing priority 3
2012/10/05 15:37:37.943100 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:37.955211 [set_recmode: 2928]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:37.955270 [set_recmode: 2928]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:38.128769 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:38.129034 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:38.890662 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:38.905307 [27403]: Freeze priority 1
2012/10/05 15:37:38.906027 [27403]: Freeze priority 2
2012/10/05 15:37:38.906714 [27403]: Freeze priority 3
2012/10/05 15:37:38.907809 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:38.907842 [27403]: Monitoring has been disabled
2012/10/05 15:37:38.907851 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:38.955800 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:38.956274 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:38.956631 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:38.957291 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:38.958503 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:38.958841 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:38.959706 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:38.959725 [27403]: Thawing priority 1
2012/10/05 15:37:38.959732 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:38.959748 [27403]: Thawing priority 2
2012/10/05 15:37:38.959755 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:38.959765 [27403]: Thawing priority 3
2012/10/05 15:37:38.959771 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:39.001739 [set_recmode: 2994]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:39.001805 [set_recmode: 2994]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:39.130915 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:39.130994 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:39.885320 [27403]: Freeze priority 1
2012/10/05 15:37:39.886076 [27403]: Freeze priority 2
2012/10/05 15:37:39.886785 [27403]: Freeze priority 3
2012/10/05 15:37:39.887905 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:39.887935 [27403]: Monitoring has been disabled
2012/10/05 15:37:39.887945 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:39.891625 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:39.947003 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:39.947606 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:39.947974 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:39.948558 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:39.949754 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:39.950105 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:39.951354 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:39.951413 [27403]: Thawing priority 1
2012/10/05 15:37:39.951420 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:39.951465 [27403]: Thawing priority 2
2012/10/05 15:37:39.951473 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:39.951484 [27403]: Thawing priority 3
2012/10/05 15:37:39.951490 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:39.964279 [set_recmode: 3060]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:39.964357 [set_recmode: 3060]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:40.132973 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:40.133049 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:40.892814 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:40.895137 [27403]: Freeze priority 1
2012/10/05 15:37:40.895868 [27403]: Freeze priority 2
2012/10/05 15:37:40.896531 [27403]: Freeze priority 3
2012/10/05 15:37:40.897642 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:40.897672 [27403]: Monitoring has been disabled
2012/10/05 15:37:40.897697 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:40.944924 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:40.945719 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:40.946056 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:40.946573 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:40.947615 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:40.947924 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:40.948693 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:40.948713 [27403]: Thawing priority 1
2012/10/05 15:37:40.948719 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:40.948734 [27403]: Thawing priority 2
2012/10/05 15:37:40.948741 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:40.948750 [27403]: Thawing priority 3
2012/10/05 15:37:40.948756 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:40.992328 [set_recmode: 3126]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:40.992397 [set_recmode: 3126]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:41.134148 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:41.134239 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:41.893377 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:41.897800 [27403]: Freeze priority 1
2012/10/05 15:37:41.898695 [27403]: Freeze priority 2
2012/10/05 15:37:41.899449 [27403]: Freeze priority 3
2012/10/05 15:37:41.900577 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:41.900608 [27403]: Monitoring has been disabled
2012/10/05 15:37:41.900618 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:41.950687 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:41.951287 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:41.951575 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:41.952073 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:41.953205 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:41.953540 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:41.954354 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:41.954377 [27403]: Thawing priority 1
2012/10/05 15:37:41.954384 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:41.954400 [27403]: Thawing priority 2
2012/10/05 15:37:41.954407 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:41.954416 [27403]: Thawing priority 3
2012/10/05 15:37:41.954422 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:41.966457 [set_recmode: 3192]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:41.966532 [set_recmode: 3192]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:42.136128 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:42.136207 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:42.894951 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:42.907925 [27403]: Freeze priority 1
2012/10/05 15:37:42.908723 [27403]: Freeze priority 2
2012/10/05 15:37:42.909445 [27403]: Freeze priority 3
2012/10/05 15:37:42.910614 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:42.910649 [27403]: Monitoring has been disabled
2012/10/05 15:37:42.910660 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:42.959002 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:42.960140 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:42.960502 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:42.961140 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:42.962312 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:42.962698 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:42.963479 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:42.963498 [27403]: Thawing priority 1
2012/10/05 15:37:42.963505 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:42.963520 [27403]: Thawing priority 2
2012/10/05 15:37:42.963527 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:42.963536 [27403]: Thawing priority 3
2012/10/05 15:37:42.963543 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:43.005335 [set_recmode: 3258]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:43.005395 [set_recmode: 3258]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:43.138115 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:43.138392 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:43.892716 [27403]: Freeze priority 1
2012/10/05 15:37:43.893555 [27403]: Freeze priority 2
2012/10/05 15:37:43.894253 [27403]: Freeze priority 3
2012/10/05 15:37:43.895394 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:43.895424 [27403]: Monitoring has been disabled
2012/10/05 15:37:43.895434 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:43.895676 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:43.943975 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:43.944463 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:43.944790 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:43.945405 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:43.946661 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:43.946991 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:43.947806 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:43.947824 [27403]: Thawing priority 1
2012/10/05 15:37:43.947831 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:43.947847 [27403]: Thawing priority 2
2012/10/05 15:37:43.947854 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:43.947863 [27403]: Thawing priority 3
2012/10/05 15:37:43.947870 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:43.960651 [set_recmode: 3324]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:43.960709 [set_recmode: 3324]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:44.140247 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:44.140327 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:44.896593 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:44.899468 [27403]: Freeze priority 1
2012/10/05 15:37:44.900186 [27403]: Freeze priority 2
2012/10/05 15:37:44.900851 [27403]: Freeze priority 3
2012/10/05 15:37:44.901939 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:44.901980 [27403]: Monitoring has been disabled
2012/10/05 15:37:44.901989 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:44.950364 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:44.950829 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:44.951112 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:44.951624 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:44.952620 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:44.952927 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:44.953752 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:44.953772 [27403]: Thawing priority 1
2012/10/05 15:37:44.953779 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:44.953794 [27403]: Thawing priority 2
2012/10/05 15:37:44.953801 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:44.953811 [27403]: Thawing priority 3
2012/10/05 15:37:44.953818 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:44.997666 [set_recmode: 3390]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:44.997737 [set_recmode: 3390]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:45.142229 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:45.142303 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:45.897665 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:45.911887 [27403]: Freeze priority 1
2012/10/05 15:37:45.912610 [27403]: Freeze priority 2
2012/10/05 15:37:45.913318 [27403]: Freeze priority 3
2012/10/05 15:37:45.914454 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:45.914485 [27403]: Monitoring has been disabled
2012/10/05 15:37:45.914502 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:45.962727 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:45.963256 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:45.963586 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:45.964185 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:45.965319 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:45.965651 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:45.966478 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:45.966495 [27403]: Thawing priority 1
2012/10/05 15:37:45.966502 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:45.966526 [27403]: Thawing priority 2
2012/10/05 15:37:45.966534 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:45.966543 [27403]: Thawing priority 3
2012/10/05 15:37:45.966549 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:45.998696 [set_recmode: 3456]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:45.998769 [set_recmode: 3456]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:46.144350 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:46.144428 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:46.896566 [27403]: Freeze priority 1
2012/10/05 15:37:46.897354 [27403]: Freeze priority 2
2012/10/05 15:37:46.898041 [27403]: Freeze priority 3
2012/10/05 15:37:46.898190 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:46.899202 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:46.899227 [27403]: Monitoring has been disabled
2012/10/05 15:37:46.899236 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:46.959513 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:46.960087 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:46.960392 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:46.960955 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:46.961965 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:46.962263 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:46.963102 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:46.963121 [27403]: Thawing priority 1
2012/10/05 15:37:46.963128 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:46.963144 [27403]: Thawing priority 2
2012/10/05 15:37:46.963151 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:46.963161 [27403]: Thawing priority 3
2012/10/05 15:37:46.963167 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:46.981336 [set_recmode: 3522]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:46.981393 [set_recmode: 3522]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:47.146368 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:47.146444 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:47.899289 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:47.903903 [27403]: Freeze priority 1
2012/10/05 15:37:47.904661 [27403]: Freeze priority 2
2012/10/05 15:37:47.905323 [27403]: Freeze priority 3
2012/10/05 15:37:47.906402 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:47.906432 [27403]: Monitoring has been disabled
2012/10/05 15:37:47.906441 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:47.953085 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:47.953633 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:47.953987 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:47.954662 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:47.955698 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:47.956003 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:47.956778 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:47.956795 [27403]: Thawing priority 1
2012/10/05 15:37:47.956801 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:47.956815 [27403]: Thawing priority 2
2012/10/05 15:37:47.956822 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:47.956831 [27403]: Thawing priority 3
2012/10/05 15:37:47.956837 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:47.988345 [set_recmode: 3588]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:47.988413 [set_recmode: 3588]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:48.148463 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:48.148718 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:48.899859 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:48.917915 [27403]: Freeze priority 1
2012/10/05 15:37:48.918644 [27403]: Freeze priority 2
2012/10/05 15:37:48.919277 [27403]: Freeze priority 3
2012/10/05 15:37:48.920326 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:48.920357 [27403]: Monitoring has been disabled
2012/10/05 15:37:48.920366 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:48.968055 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:48.969523 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:48.969870 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:48.970565 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:48.971809 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:48.972132 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:48.972984 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:48.973005 [27403]: Thawing priority 1
2012/10/05 15:37:48.973011 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:48.973028 [27403]: Thawing priority 2
2012/10/05 15:37:48.973035 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:48.973044 [27403]: Thawing priority 3
2012/10/05 15:37:48.973050 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:49.016157 [set_recmode: 3654]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:49.016266 [set_recmode: 3654]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:49.150228 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:49.150313 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:49.901087 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:49.902292 [27403]: Freeze priority 1
2012/10/05 15:37:49.903016 [27403]: Freeze priority 2
2012/10/05 15:37:49.903670 [27403]: Freeze priority 3
2012/10/05 15:37:49.904743 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:49.904795 [27403]: Monitoring has been disabled
2012/10/05 15:37:49.904806 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:49.952480 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:49.953385 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:49.953748 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:49.954363 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:49.955384 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:49.955675 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:49.956432 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:49.956451 [27403]: Thawing priority 1
2012/10/05 15:37:49.956458 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:49.956475 [27403]: Thawing priority 2
2012/10/05 15:37:49.956481 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:49.956492 [27403]: Thawing priority 3
2012/10/05 15:37:49.956498 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:49.968989 [set_recmode: 3720]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:49.969051 [set_recmode: 3720]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:50.152567 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:50.152644 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:50.902456 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:50.909401 [27403]: Freeze priority 1
2012/10/05 15:37:50.910161 [27403]: Freeze priority 2
2012/10/05 15:37:50.910916 [27403]: Freeze priority 3
2012/10/05 15:37:50.912025 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:50.912060 [27403]: Monitoring has been disabled
2012/10/05 15:37:50.912069 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:50.959585 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:50.960170 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:50.960521 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:50.961139 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:50.962136 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:50.962409 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:50.963141 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:50.963158 [27403]: Thawing priority 1
2012/10/05 15:37:50.963165 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:50.963181 [27403]: Thawing priority 2
2012/10/05 15:37:50.963187 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:50.963196 [27403]: Thawing priority 3
2012/10/05 15:37:50.963202 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:51.055146 [set_recmode: 3786]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:51.055224 [set_recmode: 3786]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:51.154549 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:51.154617 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:51.903147 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:51.904157 [27403]: Freeze priority 1
2012/10/05 15:37:51.904881 [27403]: Freeze priority 2
2012/10/05 15:37:51.905603 [27403]: Freeze priority 3
2012/10/05 15:37:51.906703 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:51.906733 [27403]: Monitoring has been disabled
2012/10/05 15:37:51.906742 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:51.953645 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:51.954265 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:51.954655 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:51.955332 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:51.956590 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:51.956948 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:51.957764 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:51.957789 [27403]: Thawing priority 1
2012/10/05 15:37:51.957796 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:51.957811 [27403]: Thawing priority 2
2012/10/05 15:37:51.957817 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:51.957826 [27403]: Thawing priority 3
2012/10/05 15:37:51.957832 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:51.970662 [set_recmode: 3852]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:51.970727 [set_recmode: 3852]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:52.156736 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:52.156813 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:52.904649 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:52.921925 [27403]: Freeze priority 1
2012/10/05 15:37:52.922670 [27403]: Freeze priority 2
2012/10/05 15:37:52.923342 [27403]: Freeze priority 3
2012/10/05 15:37:52.925020 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:52.925078 [27403]: Monitoring has been disabled
2012/10/05 15:37:52.925088 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:52.977369 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:52.978438 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:52.978889 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:52.979547 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:52.980653 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:52.980996 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:52.981849 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:52.981871 [27403]: Thawing priority 1
2012/10/05 15:37:52.981878 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:52.981896 [27403]: Thawing priority 2
2012/10/05 15:37:52.981904 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:52.981914 [27403]: Thawing priority 3
2012/10/05 15:37:52.981921 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:53.028435 [set_recmode: 3918]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:53.028546 [set_recmode: 3918]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:53.158771 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:53.159106 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:53.906169 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:53.908776 [27403]: Freeze priority 1
2012/10/05 15:37:53.909459 [27403]: Freeze priority 2
2012/10/05 15:37:53.910116 [27403]: Freeze priority 3
2012/10/05 15:37:53.911114 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:53.911142 [27403]: Monitoring has been disabled
2012/10/05 15:37:53.911151 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:53.962645 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:53.963353 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:53.963943 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:53.964553 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:53.966112 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:53.966502 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:53.967497 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:53.967533 [27403]: Thawing priority 1
2012/10/05 15:37:53.967541 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:53.967561 [27403]: Thawing priority 2
2012/10/05 15:37:53.967568 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:53.967579 [27403]: Thawing priority 3
2012/10/05 15:37:53.967586 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:53.980690 [set_recmode: 3984]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:53.980778 [set_recmode: 3984]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:54.160886 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:54.160961 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:54.907224 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:54.916021 [27403]: Freeze priority 1
2012/10/05 15:37:54.916770 [27403]: Freeze priority 2
2012/10/05 15:37:54.917438 [27403]: Freeze priority 3
2012/10/05 15:37:54.918520 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:54.918550 [27403]: Monitoring has been disabled
2012/10/05 15:37:54.918559 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:54.975558 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:54.976095 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:54.976477 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:54.977125 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:54.978308 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:54.978663 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:54.979499 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:54.979519 [27403]: Thawing priority 1
2012/10/05 15:37:54.979527 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:54.979543 [27403]: Thawing priority 2
2012/10/05 15:37:54.979550 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:54.979560 [27403]: Thawing priority 3
2012/10/05 15:37:54.979566 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:55.023240 [set_recmode: 4050]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:55.023304 [set_recmode: 4050]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:55.162934 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:55.163017 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:55.907774 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:55.911413 [27403]: Freeze priority 1
2012/10/05 15:37:55.912122 [27403]: Freeze priority 2
2012/10/05 15:37:55.912773 [27403]: Freeze priority 3
2012/10/05 15:37:55.913864 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:55.913896 [27403]: Monitoring has been disabled
2012/10/05 15:37:55.913912 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:55.964921 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:55.967005 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:55.967375 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:55.967926 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:55.969068 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:55.969405 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:55.970240 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:55.970262 [27403]: Thawing priority 1
2012/10/05 15:37:55.970268 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:55.970286 [27403]: Thawing priority 2
2012/10/05 15:37:55.970292 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:55.970302 [27403]: Thawing priority 3
2012/10/05 15:37:55.970308 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:55.983027 [set_recmode: 4116]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:55.983107 [set_recmode: 4116]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:56.165069 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:56.165149 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:56.908840 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:56.913758 [27403]: Freeze priority 1
2012/10/05 15:37:56.914505 [27403]: Freeze priority 2
2012/10/05 15:37:56.915182 [27403]: Freeze priority 3
2012/10/05 15:37:56.916391 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:56.916427 [27403]: Monitoring has been disabled
2012/10/05 15:37:56.916444 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:56.964388 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:56.964854 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:56.965565 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:56.966192 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:56.967334 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:56.967679 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:56.968541 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:56.968562 [27403]: Thawing priority 1
2012/10/05 15:37:56.968569 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:56.968585 [27403]: Thawing priority 2
2012/10/05 15:37:56.968592 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:56.968602 [27403]: Thawing priority 3
2012/10/05 15:37:56.968608 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:57.012061 [set_recmode: 4182]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:57.012136 [set_recmode: 4182]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:57.167049 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:57.167127 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:57.909967 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:57.936049 [27403]: Freeze priority 1
2012/10/05 15:37:57.936785 [27403]: Freeze priority 2
2012/10/05 15:37:57.937461 [27403]: Freeze priority 3
2012/10/05 15:37:57.938523 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:57.938553 [27403]: Monitoring has been disabled
2012/10/05 15:37:57.938563 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:57.987276 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:57.987843 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:57.988182 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:57.988793 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:57.989965 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:57.990269 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:57.991079 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:57.991099 [27403]: Thawing priority 1
2012/10/05 15:37:57.991106 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:57.991122 [27403]: Thawing priority 2
2012/10/05 15:37:57.991128 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:57.991137 [27403]: Thawing priority 3
2012/10/05 15:37:57.991143 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:58.034203 [set_recmode: 4248]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:58.034282 [set_recmode: 4248]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:58.169161 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:58.169490 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:58.911020 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:58.915923 [27403]: Freeze priority 1
2012/10/05 15:37:58.916637 [27403]: Freeze priority 2
2012/10/05 15:37:58.917301 [27403]: Freeze priority 3
2012/10/05 15:37:58.918448 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:58.918484 [27403]: Monitoring has been disabled
2012/10/05 15:37:58.918500 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:58.967358 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:58.967841 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:58.968141 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:58.968685 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:58.969835 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:58.970129 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:58.970932 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:58.970951 [27403]: Thawing priority 1
2012/10/05 15:37:58.970958 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:58.970973 [27403]: Thawing priority 2
2012/10/05 15:37:58.970980 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:58.970990 [27403]: Thawing priority 3
2012/10/05 15:37:58.970996 [27403]: Release freeze handler for prio 3
2012/10/05 15:37:58.983142 [set_recmode: 4314]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:58.983194 [set_recmode: 4314]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:37:59.171137 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:37:59.171212 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:59.912083 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:37:59.917526 [27403]: Freeze priority 1
2012/10/05 15:37:59.918292 [27403]: Freeze priority 2
2012/10/05 15:37:59.918982 [27403]: Freeze priority 3
2012/10/05 15:37:59.920084 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:59.920114 [27403]: Monitoring has been disabled
2012/10/05 15:37:59.920130 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:59.968346 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:59.984896 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:59.985292 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:59.985890 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:59.987019 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:59.987351 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:59.988139 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:59.988160 [27403]: Thawing priority 1
2012/10/05 15:37:59.988166 [27403]: Release freeze handler for prio 1
2012/10/05 15:37:59.988185 [27403]: Thawing priority 2
2012/10/05 15:37:59.988192 [27403]: Release freeze handler for prio 2
2012/10/05 15:37:59.988202 [27403]: Thawing priority 3
2012/10/05 15:37:59.988209 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:00.029682 [set_recmode: 4380]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:00.029759 [set_recmode: 4380]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:00.173195 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:00.173273 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:00.913639 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:00.925427 [27403]: Freeze priority 1
2012/10/05 15:38:00.926193 [27403]: Freeze priority 2
2012/10/05 15:38:00.926884 [27403]: Freeze priority 3
2012/10/05 15:38:00.928081 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:00.928116 [27403]: Monitoring has been disabled
2012/10/05 15:38:00.928125 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:00.976191 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:00.976661 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:00.976970 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:00.977513 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:00.978532 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:00.978860 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:00.979665 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:00.979684 [27403]: Thawing priority 1
2012/10/05 15:38:00.979691 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:00.979705 [27403]: Thawing priority 2
2012/10/05 15:38:00.979712 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:00.979723 [27403]: Thawing priority 3
2012/10/05 15:38:00.979729 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:00.992023 [set_recmode: 4446]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:00.992095 [set_recmode: 4446]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:01.175306 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:01.175391 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:01.915172 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:01.935410 [27403]: Freeze priority 1
2012/10/05 15:38:01.936119 [27403]: Freeze priority 2
2012/10/05 15:38:01.936782 [27403]: Freeze priority 3
2012/10/05 15:38:01.937874 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:01.937907 [27403]: Monitoring has been disabled
2012/10/05 15:38:01.937917 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:02.010030 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:02.010605 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:02.010921 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:02.011495 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:02.012587 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:02.012936 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:02.013819 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:02.013833 [27403]: Thawing priority 1
2012/10/05 15:38:02.013840 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:02.013856 [27403]: Thawing priority 2
2012/10/05 15:38:02.013864 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:02.013874 [27403]: Thawing priority 3
2012/10/05 15:38:02.013880 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:02.114406 [set_recmode: 4512]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:02.114494 [set_recmode: 4512]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:02.177396 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:02.177480 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:02.916292 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:02.926233 [27403]: Freeze priority 1
2012/10/05 15:38:02.927013 [27403]: Freeze priority 2
2012/10/05 15:38:02.928523 [27403]: Freeze priority 3
2012/10/05 15:38:02.929792 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:02.929860 [27403]: Monitoring has been disabled
2012/10/05 15:38:02.929870 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:02.984863 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:02.985301 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:02.985568 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:02.986050 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:02.986957 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:02.987232 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:02.987965 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:02.987981 [27403]: Thawing priority 1
2012/10/05 15:38:02.987987 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:02.988001 [27403]: Thawing priority 2
2012/10/05 15:38:02.988007 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:02.988016 [27403]: Thawing priority 3
2012/10/05 15:38:02.988022 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:03.000672 [set_recmode: 4578]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:03.000989 [set_recmode: 4578]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:03.179483 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:03.179760 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:03.916794 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:03.931082 [27403]: Freeze priority 1
2012/10/05 15:38:03.931811 [27403]: Freeze priority 2
2012/10/05 15:38:03.932477 [27403]: Freeze priority 3
2012/10/05 15:38:03.933581 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:03.933622 [27403]: Monitoring has been disabled
2012/10/05 15:38:03.933632 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:03.982021 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:03.992033 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:03.992399 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:03.993023 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:03.994191 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:03.994543 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:03.995364 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:03.995384 [27403]: Thawing priority 1
2012/10/05 15:38:03.995391 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:03.995410 [27403]: Thawing priority 2
2012/10/05 15:38:03.995417 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:03.995427 [27403]: Thawing priority 3
2012/10/05 15:38:03.995433 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:04.038691 [set_recmode: 4644]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:04.038769 [set_recmode: 4644]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:04.181557 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:04.181636 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:04.918294 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:04.926289 [27403]: Freeze priority 1
2012/10/05 15:38:04.927025 [27403]: Freeze priority 2
2012/10/05 15:38:04.927692 [27403]: Freeze priority 3
2012/10/05 15:38:04.928781 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:04.928810 [27403]: Monitoring has been disabled
2012/10/05 15:38:04.928826 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:04.976960 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:04.978066 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:04.978434 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:04.979077 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:04.980261 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:04.980599 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:04.981429 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:04.981449 [27403]: Thawing priority 1
2012/10/05 15:38:04.981456 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:04.981471 [27403]: Thawing priority 2
2012/10/05 15:38:04.981478 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:04.981488 [27403]: Thawing priority 3
2012/10/05 15:38:04.981494 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:04.993816 [set_recmode: 4710]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:04.993881 [set_recmode: 4710]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:05.183651 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:05.183732 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:05.918946 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:05.943680 [27403]: Freeze priority 1
2012/10/05 15:38:05.944435 [27403]: Freeze priority 2
2012/10/05 15:38:05.945136 [27403]: Freeze priority 3
2012/10/05 15:38:05.946277 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:05.946309 [27403]: Monitoring has been disabled
2012/10/05 15:38:05.946338 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:05.993980 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:05.995120 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:05.995422 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:05.995995 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:05.997351 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:05.997677 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:05.998517 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:05.998538 [27403]: Thawing priority 1
2012/10/05 15:38:05.998545 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:05.998562 [27403]: Thawing priority 2
2012/10/05 15:38:05.998569 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:05.998579 [27403]: Thawing priority 3
2012/10/05 15:38:05.998586 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:06.043020 [set_recmode: 4776]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:06.043097 [set_recmode: 4776]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:06.185645 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:06.185725 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:06.920009 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:06.929368 [27403]: Freeze priority 1
2012/10/05 15:38:06.930100 [27403]: Freeze priority 2
2012/10/05 15:38:06.930762 [27403]: Freeze priority 3
2012/10/05 15:38:06.931836 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:06.931865 [27403]: Monitoring has been disabled
2012/10/05 15:38:06.931875 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:06.979911 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:06.980467 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:06.980778 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:06.981351 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:06.982406 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:06.982723 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:06.983519 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:06.983538 [27403]: Thawing priority 1
2012/10/05 15:38:06.983545 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:06.983559 [27403]: Thawing priority 2
2012/10/05 15:38:06.983566 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:06.983575 [27403]: Thawing priority 3
2012/10/05 15:38:06.983582 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:06.996485 [set_recmode: 4842]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:06.996554 [set_recmode: 4842]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:07.187740 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:07.187816 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:07.921113 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:07.937224 [27403]: Freeze priority 1
2012/10/05 15:38:07.937953 [27403]: Freeze priority 2
2012/10/05 15:38:07.938583 [27403]: Freeze priority 3
2012/10/05 15:38:07.939623 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:07.939654 [27403]: Monitoring has been disabled
2012/10/05 15:38:07.939664 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:07.987261 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:07.987722 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:07.988054 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:07.988658 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:07.989835 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:07.990173 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:07.991000 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:07.991028 [27403]: Thawing priority 1
2012/10/05 15:38:07.991035 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:07.991050 [27403]: Thawing priority 2
2012/10/05 15:38:07.991057 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:07.991067 [27403]: Thawing priority 3
2012/10/05 15:38:07.991074 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:08.034210 [set_recmode: 4908]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:08.034288 [set_recmode: 4908]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:08.189798 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:08.190047 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:08.922147 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:08.933821 [27403]: Freeze priority 1
2012/10/05 15:38:08.934573 [27403]: Freeze priority 2
2012/10/05 15:38:08.935273 [27403]: Freeze priority 3
2012/10/05 15:38:08.936344 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:08.936376 [27403]: Monitoring has been disabled
2012/10/05 15:38:08.936385 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:08.982936 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:08.984087 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:08.984454 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:08.985119 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:08.986341 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:08.986701 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:08.987499 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:08.987529 [27403]: Thawing priority 1
2012/10/05 15:38:08.987540 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:08.987557 [27403]: Thawing priority 2
2012/10/05 15:38:08.987564 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:08.987574 [27403]: Thawing priority 3
2012/10/05 15:38:08.987581 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:09.000659 [set_recmode: 4974]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:09.000732 [set_recmode: 4974]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:09.191862 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:09.191943 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:09.923693 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:09.950815 [27403]: Freeze priority 1
2012/10/05 15:38:09.951552 [27403]: Freeze priority 2
2012/10/05 15:38:09.952265 [27403]: Freeze priority 3
2012/10/05 15:38:09.953462 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:09.953495 [27403]: Monitoring has been disabled
2012/10/05 15:38:09.953504 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:10.001622 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:10.002088 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:10.002420 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:10.003020 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:10.004155 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:10.004472 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:10.005278 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:10.005303 [27403]: Thawing priority 1
2012/10/05 15:38:10.005310 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:10.005325 [27403]: Thawing priority 2
2012/10/05 15:38:10.005332 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:10.005342 [27403]: Thawing priority 3
2012/10/05 15:38:10.005349 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:10.043289 [set_recmode: 5040]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:10.043363 [set_recmode: 5040]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:10.193799 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:10.193877 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:10.925228 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:10.937314 [27403]: Freeze priority 1
2012/10/05 15:38:10.938007 [27403]: Freeze priority 2
2012/10/05 15:38:10.938622 [27403]: Freeze priority 3
2012/10/05 15:38:10.939717 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:10.939747 [27403]: Monitoring has been disabled
2012/10/05 15:38:10.939756 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:11.017816 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:11.018439 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:11.018818 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:11.019494 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:11.020719 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:11.021060 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:11.021941 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:11.021956 [27403]: Thawing priority 1
2012/10/05 15:38:11.021963 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:11.021979 [27403]: Thawing priority 2
2012/10/05 15:38:11.021986 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:11.021996 [27403]: Thawing priority 3
2012/10/05 15:38:11.022003 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:11.064038 [set_recmode: 5106]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:11.064103 [set_recmode: 5106]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:11.195950 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:11.196029 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:11.925775 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:11.940123 [27403]: Freeze priority 1
2012/10/05 15:38:11.940923 [27403]: Freeze priority 2
2012/10/05 15:38:11.941537 [27403]: Freeze priority 3
2012/10/05 15:38:11.942557 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:11.942595 [27403]: Monitoring has been disabled
2012/10/05 15:38:11.942605 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:11.990986 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:11.992265 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:11.992666 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:11.993243 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:11.994262 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:11.994552 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:11.995311 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:11.995340 [27403]: Thawing priority 1
2012/10/05 15:38:11.995347 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:11.995362 [27403]: Thawing priority 2
2012/10/05 15:38:11.995369 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:11.995378 [27403]: Thawing priority 3
2012/10/05 15:38:11.995385 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:12.007947 [set_recmode: 5172]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:12.008014 [set_recmode: 5172]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:12.198038 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:12.198116 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:12.926406 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:12.963236 [27403]: Freeze priority 1
2012/10/05 15:38:12.964009 [27403]: Freeze priority 2
2012/10/05 15:38:12.964682 [27403]: Freeze priority 3
2012/10/05 15:38:12.965851 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:12.965882 [27403]: Monitoring has been disabled
2012/10/05 15:38:12.965892 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:13.014933 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:13.015403 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:13.015724 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:13.016313 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:13.017474 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:13.017802 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:13.018612 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:13.018627 [27403]: Thawing priority 1
2012/10/05 15:38:13.018634 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:13.018649 [27403]: Thawing priority 2
2012/10/05 15:38:13.018656 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:13.018666 [27403]: Thawing priority 3
2012/10/05 15:38:13.018673 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:13.121597 [set_recmode: 5238]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:13.121685 [set_recmode: 5238]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:13.200139 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:13.200403 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:13.927403 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:13.942339 [27403]: Freeze priority 1
2012/10/05 15:38:13.943112 [27403]: Freeze priority 2
2012/10/05 15:38:13.943761 [27403]: Freeze priority 3
2012/10/05 15:38:13.944958 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:13.944992 [27403]: Monitoring has been disabled
2012/10/05 15:38:13.945002 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:13.991941 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:14.008483 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:14.008844 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:14.009441 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:14.010499 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:14.010787 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:14.011526 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:14.011552 [27403]: Thawing priority 1
2012/10/05 15:38:14.011559 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:14.011578 [27403]: Thawing priority 2
2012/10/05 15:38:14.011585 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:14.011596 [27403]: Thawing priority 3
2012/10/05 15:38:14.011603 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:14.023858 [set_recmode: 5304]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:14.023915 [set_recmode: 5304]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:14.202170 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:14.202247 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:14.928544 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:14.943775 [27403]: Freeze priority 1
2012/10/05 15:38:14.944502 [27403]: Freeze priority 2
2012/10/05 15:38:14.945173 [27403]: Freeze priority 3
2012/10/05 15:38:14.946280 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:14.946321 [27403]: Monitoring has been disabled
2012/10/05 15:38:14.946332 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:14.994081 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:14.994536 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:14.994847 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:14.995434 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:14.996592 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:14.996952 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:14.997767 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:14.997786 [27403]: Thawing priority 1
2012/10/05 15:38:14.997792 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:14.997807 [27403]: Thawing priority 2
2012/10/05 15:38:14.997814 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:14.997824 [27403]: Thawing priority 3
2012/10/05 15:38:14.997830 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:15.031565 [set_recmode: 5370]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:15.031628 [set_recmode: 5370]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:15.204208 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:15.204290 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:15.929604 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:15.968515 [27403]: Freeze priority 1
2012/10/05 15:38:15.969261 [27403]: Freeze priority 2
2012/10/05 15:38:15.969928 [27403]: Freeze priority 3
2012/10/05 15:38:15.971051 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:15.971082 [27403]: Monitoring has been disabled
2012/10/05 15:38:15.971092 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:16.018242 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:16.019445 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:16.019788 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:16.020348 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:16.021537 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:16.021862 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:16.022672 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:16.022687 [27403]: Thawing priority 1
2012/10/05 15:38:16.022703 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:16.022718 [27403]: Thawing priority 2
2012/10/05 15:38:16.022726 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:16.022736 [27403]: Thawing priority 3
2012/10/05 15:38:16.022743 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:16.065628 [set_recmode: 5436]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:16.065699 [set_recmode: 5436]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:16.206274 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:16.206352 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:16.930668 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:16.948476 [27403]: Freeze priority 1
2012/10/05 15:38:16.949202 [27403]: Freeze priority 2
2012/10/05 15:38:16.949853 [27403]: Freeze priority 3
2012/10/05 15:38:16.950934 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:16.950968 [27403]: Monitoring has been disabled
2012/10/05 15:38:16.950978 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:16.998486 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:16.998960 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:16.999271 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:16.999820 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:17.000923 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:17.001268 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:17.002117 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:17.002130 [27403]: Thawing priority 1
2012/10/05 15:38:17.002137 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:17.002152 [27403]: Thawing priority 2
2012/10/05 15:38:17.002159 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:17.002169 [27403]: Thawing priority 3
2012/10/05 15:38:17.002176 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:17.014553 [set_recmode: 5502]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:17.014610 [set_recmode: 5502]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:17.208367 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:17.208447 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:17.932223 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:17.950222 [27403]: Freeze priority 1
2012/10/05 15:38:17.950984 [27403]: Freeze priority 2
2012/10/05 15:38:17.951702 [27403]: Freeze priority 3
2012/10/05 15:38:17.952838 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:17.952870 [27403]: Monitoring has been disabled
2012/10/05 15:38:17.952879 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:17.999641 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:18.000165 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:18.000529 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:18.001218 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:18.002341 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:18.002671 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:18.003438 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:18.003452 [27403]: Thawing priority 1
2012/10/05 15:38:18.003466 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:18.003481 [27403]: Thawing priority 2
2012/10/05 15:38:18.003487 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:18.003496 [27403]: Thawing priority 3
2012/10/05 15:38:18.003502 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:18.041804 [set_recmode: 5568]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:18.041869 [set_recmode: 5568]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:18.210413 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:18.210667 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:18.933281 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:18.957443 [27403]: Freeze priority 1
2012/10/05 15:38:18.958208 [27403]: Freeze priority 2
2012/10/05 15:38:18.958895 [27403]: Freeze priority 3
2012/10/05 15:38:18.959975 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:18.960010 [27403]: Monitoring has been disabled
2012/10/05 15:38:18.960020 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:19.016050 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:19.016572 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:19.016927 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:19.017524 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:19.018696 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:19.019019 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:19.019872 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:19.019885 [27403]: Thawing priority 1
2012/10/05 15:38:19.019892 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:19.019907 [27403]: Thawing priority 2
2012/10/05 15:38:19.019914 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:19.019924 [27403]: Thawing priority 3
2012/10/05 15:38:19.019931 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:19.053788 [set_recmode: 5634]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:19.053856 [set_recmode: 5634]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:19.212489 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:19.212567 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:19.934340 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:19.953048 [27403]: Freeze priority 1
2012/10/05 15:38:19.953798 [27403]: Freeze priority 2
2012/10/05 15:38:19.954460 [27403]: Freeze priority 3
2012/10/05 15:38:19.955533 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:19.955560 [27403]: Monitoring has been disabled
2012/10/05 15:38:19.955569 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:20.002249 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:20.004651 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:20.005027 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:20.005727 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:20.006989 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:20.007322 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:20.008096 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:20.008109 [27403]: Thawing priority 1
2012/10/05 15:38:20.008115 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:20.008142 [27403]: Thawing priority 2
2012/10/05 15:38:20.008149 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:20.008159 [27403]: Thawing priority 3
2012/10/05 15:38:20.008165 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:20.027671 [set_recmode: 5700]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:20.027739 [set_recmode: 5700]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:20.214497 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:20.214578 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:20.935413 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:20.971532 [27403]: Freeze priority 1
2012/10/05 15:38:20.972252 [27403]: Freeze priority 2
2012/10/05 15:38:20.972987 [27403]: Freeze priority 3
2012/10/05 15:38:20.974116 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:20.974146 [27403]: Monitoring has been disabled
2012/10/05 15:38:20.974155 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:21.022528 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:21.023936 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:21.024294 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:21.024950 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:21.026099 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:21.026446 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:21.027238 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:21.027253 [27403]: Thawing priority 1
2012/10/05 15:38:21.027259 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:21.027275 [27403]: Thawing priority 2
2012/10/05 15:38:21.027282 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:21.027292 [27403]: Thawing priority 3
2012/10/05 15:38:21.027299 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:21.070531 [set_recmode: 5766]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:21.070604 [set_recmode: 5766]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:21.216652 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:21.216732 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:21.935713 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:21.957447 [27403]: Freeze priority 1
2012/10/05 15:38:21.958232 [27403]: Freeze priority 2
2012/10/05 15:38:21.958909 [27403]: Freeze priority 3
2012/10/05 15:38:21.960000 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:21.960035 [27403]: Monitoring has been disabled
2012/10/05 15:38:21.960044 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:22.010084 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:22.010622 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:22.010953 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:22.011579 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:22.012704 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:22.013007 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:22.013860 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:22.013880 [27403]: Thawing priority 1
2012/10/05 15:38:22.013887 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:22.013913 [27403]: Thawing priority 2
2012/10/05 15:38:22.013920 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:22.013929 [27403]: Thawing priority 3
2012/10/05 15:38:22.013935 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:22.026817 [set_recmode: 5832]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:22.026893 [set_recmode: 5832]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:22.218681 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:22.218760 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:22.937200 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:22.964315 [27403]: Freeze priority 1
2012/10/05 15:38:22.965031 [27403]: Freeze priority 2
2012/10/05 15:38:22.965692 [27403]: Freeze priority 3
2012/10/05 15:38:22.966762 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:22.966792 [27403]: Monitoring has been disabled
2012/10/05 15:38:22.966801 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:23.012881 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:23.013416 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:23.013746 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:23.014241 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:23.015175 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:23.015456 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:23.016306 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:23.016322 [27403]: Thawing priority 1
2012/10/05 15:38:23.016328 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:23.016342 [27403]: Thawing priority 2
2012/10/05 15:38:23.016348 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:23.016358 [27403]: Thawing priority 3
2012/10/05 15:38:23.016364 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:23.057914 [set_recmode: 5898]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:23.057999 [set_recmode: 5898]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:23.220756 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:23.221104 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:23.938156 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:23.960748 [27403]: Freeze priority 1
2012/10/05 15:38:23.961487 [27403]: Freeze priority 2
2012/10/05 15:38:23.962223 [27403]: Freeze priority 3
2012/10/05 15:38:23.963326 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:23.963362 [27403]: Monitoring has been disabled
2012/10/05 15:38:23.963371 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:24.012024 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:24.012546 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:24.012928 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:24.013612 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:24.014858 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:24.015225 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:24.016060 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:24.016075 [27403]: Thawing priority 1
2012/10/05 15:38:24.016082 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:24.016096 [27403]: Thawing priority 2
2012/10/05 15:38:24.016111 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:24.016122 [27403]: Thawing priority 3
2012/10/05 15:38:24.016128 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:24.029335 [set_recmode: 5964]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:24.029389 [set_recmode: 5964]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:24.222811 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:24.222887 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:24.938725 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:24.977060 [27403]: Freeze priority 1
2012/10/05 15:38:24.977818 [27403]: Freeze priority 2
2012/10/05 15:38:24.978476 [27403]: Freeze priority 3
2012/10/05 15:38:24.979589 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:24.979624 [27403]: Monitoring has been disabled
2012/10/05 15:38:24.979634 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:25.026030 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:25.029800 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:25.030206 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:25.030815 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:25.031885 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:25.032168 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:25.032932 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:25.032946 [27403]: Thawing priority 1
2012/10/05 15:38:25.032952 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:25.032970 [27403]: Thawing priority 2
2012/10/05 15:38:25.032976 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:25.032985 [27403]: Thawing priority 3
2012/10/05 15:38:25.032991 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:25.072440 [set_recmode: 6030]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:25.072522 [set_recmode: 6030]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:25.224877 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:25.224954 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:25.939798 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:25.963296 [27403]: Freeze priority 1
2012/10/05 15:38:25.964053 [27403]: Freeze priority 2
2012/10/05 15:38:25.964722 [27403]: Freeze priority 3
2012/10/05 15:38:25.965812 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:25.965853 [27403]: Monitoring has been disabled
2012/10/05 15:38:25.965862 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:26.013485 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:26.013996 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:26.014347 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:26.014977 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:26.016108 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:26.016431 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:26.017268 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:26.017283 [27403]: Thawing priority 1
2012/10/05 15:38:26.017290 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:26.017306 [27403]: Thawing priority 2
2012/10/05 15:38:26.017321 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:26.017332 [27403]: Thawing priority 3
2012/10/05 15:38:26.017339 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:26.029546 [set_recmode: 6096]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:26.029610 [set_recmode: 6096]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:26.226948 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:26.227030 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:26.940872 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:26.970675 [27403]: Freeze priority 1
2012/10/05 15:38:26.971430 [27403]: Freeze priority 2
2012/10/05 15:38:26.972080 [27403]: Freeze priority 3
2012/10/05 15:38:26.973171 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:26.973210 [27403]: Monitoring has been disabled
2012/10/05 15:38:26.973220 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:27.019102 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:27.021759 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:27.022124 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:27.022720 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:27.023894 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:27.024218 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:27.025026 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:27.025040 [27403]: Thawing priority 1
2012/10/05 15:38:27.025046 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:27.025066 [27403]: Thawing priority 2
2012/10/05 15:38:27.025072 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:27.025082 [27403]: Thawing priority 3
2012/10/05 15:38:27.025088 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:27.064231 [set_recmode: 6162]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:27.064290 [set_recmode: 6162]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:27.229035 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:27.229110 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:27.941437 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:27.968100 [27403]: Freeze priority 1
2012/10/05 15:38:27.968871 [27403]: Freeze priority 2
2012/10/05 15:38:27.969554 [27403]: Freeze priority 3
2012/10/05 15:38:27.970667 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:27.970696 [27403]: Monitoring has been disabled
2012/10/05 15:38:27.970705 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:28.018093 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:28.018755 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:28.019120 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:28.019811 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:28.021084 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:28.021451 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:28.022234 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:28.022260 [27403]: Thawing priority 1
2012/10/05 15:38:28.022269 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:28.022285 [27403]: Thawing priority 2
2012/10/05 15:38:28.022292 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:28.022312 [27403]: Thawing priority 3
2012/10/05 15:38:28.022319 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:28.034814 [set_recmode: 6228]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:28.034874 [set_recmode: 6228]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:28.231127 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:28.231387 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:28.942510 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:28.983968 [27403]: Freeze priority 1
2012/10/05 15:38:28.984698 [27403]: Freeze priority 2
2012/10/05 15:38:28.985408 [27403]: Freeze priority 3
2012/10/05 15:38:28.986555 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:28.986585 [27403]: Monitoring has been disabled
2012/10/05 15:38:28.986594 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:29.034540 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:29.035068 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:29.035350 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:29.035822 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:29.036779 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:29.037074 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:29.037875 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:29.037888 [27403]: Thawing priority 1
2012/10/05 15:38:29.037894 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:29.037909 [27403]: Thawing priority 2
2012/10/05 15:38:29.037915 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:29.037924 [27403]: Thawing priority 3
2012/10/05 15:38:29.037930 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:29.081366 [set_recmode: 6296]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:29.081440 [set_recmode: 6296]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:29.233111 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:29.233188 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:29.943553 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:29.970338 [27403]: Freeze priority 1
2012/10/05 15:38:29.971099 [27403]: Freeze priority 2
2012/10/05 15:38:29.971759 [27403]: Freeze priority 3
2012/10/05 15:38:29.972831 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:29.972860 [27403]: Monitoring has been disabled
2012/10/05 15:38:29.972870 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:30.019815 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:30.020790 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:30.021173 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:30.021787 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:30.022912 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:30.023230 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:30.023996 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:30.024008 [27403]: Thawing priority 1
2012/10/05 15:38:30.024015 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:30.024029 [27403]: Thawing priority 2
2012/10/05 15:38:30.024035 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:30.024054 [27403]: Thawing priority 3
2012/10/05 15:38:30.024061 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:30.036368 [set_recmode: 6366]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:30.036438 [set_recmode: 6366]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:30.235212 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:30.235291 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:30.944098 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:30.978449 [27403]: Freeze priority 1
2012/10/05 15:38:30.979224 [27403]: Freeze priority 2
2012/10/05 15:38:30.979919 [27403]: Freeze priority 3
2012/10/05 15:38:30.981073 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:30.981104 [27403]: Monitoring has been disabled
2012/10/05 15:38:30.981113 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:31.028845 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:31.029305 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:31.029625 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:31.030152 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:31.031165 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:31.031449 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:31.032205 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:31.032219 [27403]: Thawing priority 1
2012/10/05 15:38:31.032226 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:31.032240 [27403]: Thawing priority 2
2012/10/05 15:38:31.032248 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:31.032258 [27403]: Thawing priority 3
2012/10/05 15:38:31.032264 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:31.069217 [set_recmode: 6432]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:31.069287 [set_recmode: 6432]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:31.237187 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:31.237263 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:31.945155 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:31.975421 [27403]: Freeze priority 1
2012/10/05 15:38:31.976162 [27403]: Freeze priority 2
2012/10/05 15:38:31.976842 [27403]: Freeze priority 3
2012/10/05 15:38:31.978013 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:31.978059 [27403]: Monitoring has been disabled
2012/10/05 15:38:31.978069 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:32.025770 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:32.026317 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:32.026676 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:32.027331 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:32.028455 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:32.028820 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:32.029642 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:32.029658 [27403]: Thawing priority 1
2012/10/05 15:38:32.029664 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:32.029679 [27403]: Thawing priority 2
2012/10/05 15:38:32.029686 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:32.029706 [27403]: Thawing priority 3
2012/10/05 15:38:32.029713 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:32.041992 [set_recmode: 6498]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:32.042081 [set_recmode: 6498]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:32.239343 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:32.239423 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:32.946226 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:32.989282 [27403]: Freeze priority 1
2012/10/05 15:38:32.990063 [27403]: Freeze priority 2
2012/10/05 15:38:32.990774 [27403]: Freeze priority 3
2012/10/05 15:38:32.991941 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:32.991972 [27403]: Monitoring has been disabled
2012/10/05 15:38:32.991982 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:33.051139 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:33.051699 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:33.052040 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:33.052679 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:33.053872 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:33.054205 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:33.055040 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:33.055054 [27403]: Thawing priority 1
2012/10/05 15:38:33.055061 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:33.055075 [27403]: Thawing priority 2
2012/10/05 15:38:33.055082 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:33.055092 [27403]: Thawing priority 3
2012/10/05 15:38:33.055113 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:33.097817 [set_recmode: 6564]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:33.097890 [set_recmode: 6564]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:33.241445 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:33.241697 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:33.946796 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:33.977452 [27403]: Freeze priority 1
2012/10/05 15:38:33.978237 [27403]: Freeze priority 2
2012/10/05 15:38:33.978928 [27403]: Freeze priority 3
2012/10/05 15:38:33.980047 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:33.980077 [27403]: Monitoring has been disabled
2012/10/05 15:38:33.980087 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:34.028165 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:34.028704 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:34.029097 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:34.029749 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:34.031025 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:34.031406 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:34.032203 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:34.032218 [27403]: Thawing priority 1
2012/10/05 15:38:34.032225 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:34.032240 [27403]: Thawing priority 2
2012/10/05 15:38:34.032247 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:34.032257 [27403]: Thawing priority 3
2012/10/05 15:38:34.032273 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:34.044816 [set_recmode: 6630]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:34.044874 [set_recmode: 6630]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:34.243497 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:34.243577 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:34.947391 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:34.990203 [27403]: Freeze priority 1
2012/10/05 15:38:34.990977 [27403]: Freeze priority 2
2012/10/05 15:38:34.991668 [27403]: Freeze priority 3
2012/10/05 15:38:34.992788 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:34.992819 [27403]: Monitoring has been disabled
2012/10/05 15:38:34.992828 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:35.040312 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:35.040807 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:35.041150 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:35.041712 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:35.042742 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:35.043041 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:35.043852 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:35.043866 [27403]: Thawing priority 1
2012/10/05 15:38:35.043873 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:35.043887 [27403]: Thawing priority 2
2012/10/05 15:38:35.043893 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:35.043903 [27403]: Thawing priority 3
2012/10/05 15:38:35.043909 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:35.087861 [set_recmode: 6696]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:35.087929 [set_recmode: 6696]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:35.245543 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:35.245619 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:35.948423 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:35.981380 [27403]: Freeze priority 1
2012/10/05 15:38:35.982158 [27403]: Freeze priority 2
2012/10/05 15:38:35.982857 [27403]: Freeze priority 3
2012/10/05 15:38:35.983984 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:35.984013 [27403]: Monitoring has been disabled
2012/10/05 15:38:35.984022 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:36.030258 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:36.031901 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:36.032277 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:36.032910 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:36.034058 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:36.034384 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:36.035158 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:36.035173 [27403]: Thawing priority 1
2012/10/05 15:38:36.035180 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:36.035195 [27403]: Thawing priority 2
2012/10/05 15:38:36.035203 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:36.035213 [27403]: Thawing priority 3
2012/10/05 15:38:36.035219 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:36.047900 [set_recmode: 6762]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:36.047959 [set_recmode: 6762]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:36.247526 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:36.247607 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:36.948973 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:36.982329 [27403]: Freeze priority 1
2012/10/05 15:38:36.983085 [27403]: Freeze priority 2
2012/10/05 15:38:36.983752 [27403]: Freeze priority 3
2012/10/05 15:38:36.984825 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:36.984864 [27403]: Monitoring has been disabled
2012/10/05 15:38:36.984875 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:37.046238 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:37.046818 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:37.047171 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:37.047783 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:37.048966 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:37.049295 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:37.050167 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:37.050180 [27403]: Thawing priority 1
2012/10/05 15:38:37.050187 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:37.050201 [27403]: Thawing priority 2
2012/10/05 15:38:37.050208 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:37.050217 [27403]: Thawing priority 3
2012/10/05 15:38:37.050223 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:37.089620 [set_recmode: 6828]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:37.089693 [set_recmode: 6828]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:37.249670 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:37.249744 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:37.950002 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:37.983450 [27403]: Freeze priority 1
2012/10/05 15:38:37.984186 [27403]: Freeze priority 2
2012/10/05 15:38:37.984847 [27403]: Freeze priority 3
2012/10/05 15:38:37.985956 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:37.985994 [27403]: Monitoring has been disabled
2012/10/05 15:38:37.986005 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:38.033089 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:38.033544 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:38.033856 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:38.034412 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:38.035499 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:38.035797 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:38.036583 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:38.036597 [27403]: Thawing priority 1
2012/10/05 15:38:38.036603 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:38.036617 [27403]: Thawing priority 2
2012/10/05 15:38:38.036624 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:38.036633 [27403]: Thawing priority 3
2012/10/05 15:38:38.036639 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:38.049856 [set_recmode: 6894]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:38.049910 [set_recmode: 6894]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:38.251745 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:38.252010 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:38.950644 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:39.006709 [27403]: Freeze priority 1
2012/10/05 15:38:39.007447 [27403]: Freeze priority 2
2012/10/05 15:38:39.008129 [27403]: Freeze priority 3
2012/10/05 15:38:39.009222 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:39.009247 [27403]: Monitoring has been disabled
2012/10/05 15:38:39.009255 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:39.065400 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:39.065953 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:39.066290 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:39.066882 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:39.068000 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:39.068312 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:39.069163 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:39.069183 [27403]: Thawing priority 1
2012/10/05 15:38:39.069189 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:39.069205 [27403]: Thawing priority 2
2012/10/05 15:38:39.069211 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:39.069220 [27403]: Thawing priority 3
2012/10/05 15:38:39.069226 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:39.113031 [set_recmode: 6960]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:39.113100 [set_recmode: 6960]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:39.253802 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:39.253879 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:39.951692 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:39.987212 [27403]: Freeze priority 1
2012/10/05 15:38:39.987965 [27403]: Freeze priority 2
2012/10/05 15:38:39.988655 [27403]: Freeze priority 3
2012/10/05 15:38:39.989756 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:39.989786 [27403]: Monitoring has been disabled
2012/10/05 15:38:39.989795 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:40.036554 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:40.038827 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:40.039162 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:40.039838 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:40.041166 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:40.041492 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:40.042261 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:40.042276 [27403]: Thawing priority 1
2012/10/05 15:38:40.042283 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:40.042299 [27403]: Thawing priority 2
2012/10/05 15:38:40.042306 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:40.042316 [27403]: Thawing priority 3
2012/10/05 15:38:40.042323 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:40.054711 [set_recmode: 7026]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:40.054767 [set_recmode: 7026]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:40.255772 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:40.255852 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:40.952705 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:40.989594 [27403]: Freeze priority 1
2012/10/05 15:38:40.990373 [27403]: Freeze priority 2
2012/10/05 15:38:40.991074 [27403]: Freeze priority 3
2012/10/05 15:38:40.992221 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:40.992259 [27403]: Monitoring has been disabled
2012/10/05 15:38:40.992269 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:41.048217 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:41.048849 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:41.049222 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:41.049804 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:41.050874 [27403]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:41.051211 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:41.052174 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:41.052193 [27403]: Thawing priority 1
2012/10/05 15:38:41.052201 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:41.052219 [27403]: Thawing priority 2
2012/10/05 15:38:41.052226 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:41.052237 [27403]: Thawing priority 3
2012/10/05 15:38:41.052244 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:41.093340 [set_recmode: 7092]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:41.093422 [set_recmode: 7092]: ERROR: recovery lock file /mnt/gluster/ctdb/lock not locked when recovering!
2012/10/05 15:38:41.257920 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:41.257995 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:41.680207 [27403]: 10.100.55.205:4379: node 10.100.55.209:4379 is dead: 0 connected
2012/10/05 15:38:41.680259 [27403]: Tearing down connection to dead node :0
2012/10/05 15:38:41.953836 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:42.259545 [recoverd:27474]: Recmaster node 0 is disconnected. Force reelection
2012/10/05 15:38:42.259584 [recoverd:27474]: server/ctdb_recoverd.c:2160 Force an election
2012/10/05 15:38:42.259604 [27403]: Freeze priority 1
2012/10/05 15:38:42.259863 [27403]: Freeze priority 2
2012/10/05 15:38:42.260051 [27403]: Freeze priority 3
2012/10/05 15:38:42.260284 [recoverd:27474]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/05 15:38:42.954830 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:43.955982 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:44.956506 [27403]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/05 15:38:45.261571 [recoverd:27474]: server/ctdb_recoverd.c:913 Election timed out
2012/10/05 15:38:45.261954 [recoverd:27474]: The interfaces status has changed on local node 1 - force takeover run
2012/10/05 15:38:45.262039 [recoverd:27474]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:45.262090 [recoverd:27474]: server/ctdb_recoverd.c:2344 Node:1 was in recovery mode. Restart recovery process
2012/10/05 15:38:45.262113 [recoverd:27474]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:45.262133 [recoverd:27474]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:45.262141 [recoverd:27474]: Take the recovery lock
2012/10/05 15:38:45.299354 [recoverd:27474]: Recovery lock taken successfully
2012/10/05 15:38:45.299392 [recoverd:27474]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:45.299474 [recoverd:27474]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:45.299485 [recoverd:27474]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:45.299521 [recoverd:27474]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:45.299560 [recoverd:27474]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:45.299636 [27403]: Freeze priority 1
2012/10/05 15:38:45.299672 [27403]: Freeze priority 2
2012/10/05 15:38:45.299704 [27403]: Freeze priority 3
2012/10/05 15:38:45.299755 [27403]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:45.299777 [27403]: Monitoring has been disabled
2012/10/05 15:38:45.299786 [27403]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:45.345572 [27403]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:45.345720 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:45.345747 [recoverd:27474]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:45.345811 [recoverd:27474]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:45.345822 [recoverd:27474]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:45.345840 [27403]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:45.345863 [recoverd:27474]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:45.345929 [recoverd:27474]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:45.345964 [recoverd:27474]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:45.346023 [27403]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:45.346046 [recoverd:27474]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:45.346067 [27403]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:45.346082 [27403]: Thawing priority 1
2012/10/05 15:38:45.346088 [27403]: Release freeze handler for prio 1
2012/10/05 15:38:45.346102 [27403]: Thawing priority 2
2012/10/05 15:38:45.346108 [27403]: Release freeze handler for prio 2
2012/10/05 15:38:45.346118 [27403]: Thawing priority 3
2012/10/05 15:38:45.346124 [27403]: Release freeze handler for prio 3
2012/10/05 15:38:45.383887 [recoverd:27474]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/05 15:38:45.384051 [recoverd:27474]: Failed to find node to cover ip 10.100.55.198
2012/10/05 15:38:45.384102 [27403]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/05 15:38:45.384147 [27403]: Forced running of eventscripts with arguments ipreallocated
2012/10/05 15:38:45.384156 [27403]: Monitoring has been disabled
2012/10/05 15:38:45.384164 [27403]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/05 15:38:45.443507 [27403]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/05 15:38:45.443541 [27403]: Monitoring has been enabled
2012/10/05 15:38:45.443606 [27403]: Recovery has finished
2012/10/05 15:38:45.443621 [27403]: Monitoring has been disabled
2012/10/05 15:38:45.443634 [27403]: server/eventscript.c:754 Starting eventscript recovered
2012/10/05 15:38:45.465485 [27403]: Shutting down recovery daemon
2012/10/05 15:38:45.465525 [27403]: Monitoring has been stopped
2012/10/05 15:38:45.465557 [27403]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/05 15:38:45.465570 [27403]: server/eventscript.c:754 Starting eventscript shutdown
-------------- next part --------------
2012/10/05 15:35:21.589864 [ 5751]: CTDB starting on node
2012/10/05 15:35:21.639865 [ 5752]: Starting CTDBD as pid : 5752
2012/10/05 15:35:21.639934 [ 5752]: Set scheduler to SCHED_FIFO
2012/10/05 15:35:21.640131 [ 5752]: ctdb chose network address 10.100.55.209:4379 pnn 0
2012/10/05 15:35:21.643410 [ 5752]: server/eventscript.c:754 Starting eventscript init
2012/10/05 15:35:21.811705 [ 5752]: server/eventscript.c:485 Eventscript init finished with state 0
2012/10/05 15:35:21.812264 [ 5752]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface 'eth0' refs[0])
2012/10/05 15:35:21.819336 [ 5752]: Keepalive monitoring has been started
2012/10/05 15:35:21.819352 [ 5752]: Monitoring has been started
2012/10/05 15:35:21.819386 [ 5752]: server/eventscript.c:754 Starting eventscript setup
2012/10/05 15:35:21.872997 [recoverd: 5823]: monitor_cluster starting
2012/10/05 15:35:21.873130 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=17870283321406128128
2012/10/05 15:35:21.873217 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=17365880163140632576
2012/10/05 15:35:21.873254 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=17582052945254416384
2012/10/05 15:35:21.873288 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=17942340915444056064
2012/10/05 15:35:21.873319 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=17798225727368200192
2012/10/05 15:35:21.873351 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=18014398509481984000
2012/10/05 15:35:21.873382 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=18086456103519911936
2012/10/05 15:35:21.873412 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=18158513697557839872
2012/10/05 15:35:21.873443 [ 5752]: server/ctdb_daemon.c:187 Registered message handler for srvid=17654110539292344320
2012/10/05 15:35:21.900337 [recoverd: 5823]: server/ctdb_recoverd.c:3029 Initial recovery master set - forcing election
2012/10/05 15:35:21.900381 [recoverd: 5823]: server/ctdb_recoverd.c:2160 Force an election
2012/10/05 15:35:21.900436 [ 5752]: Freeze priority 1
2012/10/05 15:35:21.900475 [ 5752]: Freeze priority 2
2012/10/05 15:35:21.900508 [ 5752]: Freeze priority 3
2012/10/05 15:35:21.900572 [recoverd: 5823]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/05 15:35:22.086601 [ 5752]: server/eventscript.c:485 Eventscript setup finished with state 0
2012/10/05 15:35:22.820413 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:23.821975 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:24.822527 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:24.904056 [recoverd: 5823]: server/ctdb_recoverd.c:913 Election timed out
2012/10/05 15:35:24.904426 [recoverd: 5823]: The interfaces status has changed on local node 0 - force takeover run
2012/10/05 15:35:24.904482 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:24.904535 [recoverd: 5823]: server/ctdb_recoverd.c:2344 Node:0 was in recovery mode. Restart recovery process
2012/10/05 15:35:24.904571 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:24.904584 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:24.904593 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:24.942163 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:24.942205 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:24.942276 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:24.942286 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:24.942324 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:24.942339 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:24.942416 [ 5752]: Freeze priority 1
2012/10/05 15:35:24.942454 [ 5752]: Freeze priority 2
2012/10/05 15:35:24.942487 [ 5752]: Freeze priority 3
2012/10/05 15:35:24.942539 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:24.942550 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:24.942558 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:24.989814 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:24.989993 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:24.990021 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:24.990079 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:24.990090 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:24.990110 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:24.990132 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:24.990194 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:24.990227 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:24.990288 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:24.990311 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:24.990332 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:24.990347 [ 5752]: Thawing priority 1
2012/10/05 15:35:24.990355 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:25.019343 [ 5752]: Thawing priority 2
2012/10/05 15:35:25.019370 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:25.019388 [ 5752]: Thawing priority 3
2012/10/05 15:35:25.019395 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:25.081838 [recoverd: 5823]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/05 15:35:25.081975 [recoverd: 5823]: Failed to find node to cover ip 10.100.55.198
2012/10/05 15:35:25.082022 [ 5752]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/05 15:35:25.082071 [ 5752]: Forced running of eventscripts with arguments ipreallocated
2012/10/05 15:35:25.082081 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:25.082089 [ 5752]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/05 15:35:25.142051 [ 5752]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/05 15:35:25.142104 [ 5752]: Monitoring has been enabled
2012/10/05 15:35:25.142187 [ 5752]: Recovery has finished
2012/10/05 15:35:25.142219 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:25.142232 [ 5752]: server/eventscript.c:754 Starting eventscript recovered
2012/10/05 15:35:25.186164 [ 5752]: server/eventscript.c:485 Eventscript recovered finished with state 0
2012/10/05 15:35:25.186203 [ 5752]: Monitoring has been enabled
2012/10/05 15:35:25.186261 [recoverd: 5823]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/05 15:35:25.186290 [recoverd: 5823]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/05 15:35:25.186298 [recoverd: 5823]: Resetting ban count to 0 for all nodes
2012/10/05 15:35:25.186304 [recoverd: 5823]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/05 15:35:25.823124 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:25.823174 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:26.824650 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:26.824700 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:27.825238 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:27.825283 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:28.825785 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:28.825834 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:29.826338 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:29.826406 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:30.826927 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:30.826978 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:31.827965 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:31.828017 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:32.828492 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:32.828542 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:33.829128 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:33.829177 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:33.916684 [recoverd: 5823]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/05 15:35:34.830675 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:34.830724 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:35.188223 [recoverd: 5823]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/05 15:35:35.831260 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:35.831308 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:36.822831 [ 5752]: 10.100.55.209:4379: connected to 10.100.55.205:4379 - 1 connected
2012/10/05 15:35:36.832306 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:36.832339 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:36.917849 [recoverd: 5823]: server/ctdb_recoverd.c:913 Election timed out
2012/10/05 15:35:37.422441 [recoverd: 5823]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/05 15:35:37.833405 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:37.833458 [ 5752]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/05 15:35:38.833975 [ 5752]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/05 15:35:38.834029 [ 5752]: ctdb_recheck_presistent_health: OK[0] FAIL[0]
2012/10/05 15:35:38.834036 [ 5752]: server/ctdb_monitor.c:306 Recoveries finished. Running the "startup" event.
2012/10/05 15:35:38.834050 [ 5752]: server/eventscript.c:754 Starting eventscript startup
2012/10/05 15:35:38.881038 [ 5752]: iface[eth0] has changed it's link status down => up
2012/10/05 15:35:40.433336 [recoverd: 5823]: server/ctdb_recoverd.c:913 Election timed out
2012/10/05 15:35:41.200595 [recoverd: 5823]: The interfaces status has changed on local node 0 - force takeover run
2012/10/05 15:35:41.200713 [recoverd: 5823]: Trigger takeoverrun
2012/10/05 15:35:41.207195 [recoverd: 5823]: server/ctdb_recoverd.c:2344 Node:1 was in recovery mode. Restart recovery process
2012/10/05 15:35:41.207236 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:41.207243 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:41.207249 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:41.267164 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:41.267198 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:41.267265 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:41.267274 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:41.277002 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:41.277035 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:41.286690 [ 5752]: Freeze priority 1
2012/10/05 15:35:41.300091 [ 5752]: Freeze priority 2
2012/10/05 15:35:41.304647 [ 5752]: Freeze priority 3
2012/10/05 15:35:41.305300 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:41.315456 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:41.315493 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:41.315502 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:41.949066 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:41.949237 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:41.949898 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:41.950177 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:41.950619 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:41.950633 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:41.950654 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:41.951051 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:41.951611 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:41.952114 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:41.952182 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:41.952663 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:41.953112 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:41.953140 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:41.953149 [ 5752]: Thawing priority 1
2012/10/05 15:35:41.953156 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:41.953168 [ 5752]: Thawing priority 2
2012/10/05 15:35:41.953175 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:41.953186 [ 5752]: Thawing priority 3
2012/10/05 15:35:41.953192 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:41.958830 [ 5752]: server/eventscript.c:485 Eventscript startup finished with state 0
2012/10/05 15:35:41.958888 [ 5752]: startup event OK - enabling monitoring
2012/10/05 15:35:42.000779 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:42.000822 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:42.000850 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:42.000867 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:42.000875 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:42.000882 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:42.201982 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:42.202644 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:42.202657 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:42.202663 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:42.221766 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:42.221789 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:42.221838 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:42.221847 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:42.222393 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:42.222406 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:42.222679 [ 5752]: Freeze priority 1
2012/10/05 15:35:42.223428 [ 5752]: Freeze priority 2
2012/10/05 15:35:42.224202 [ 5752]: Freeze priority 3
2012/10/05 15:35:42.225060 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:42.225393 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:42.225414 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:42.225424 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:42.312955 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:42.313125 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:42.313602 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:42.313877 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:42.314174 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:42.314190 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:42.314212 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:42.314450 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:42.314897 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:42.315123 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:42.315186 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:42.315459 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:42.315694 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:42.315718 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:42.315727 [ 5752]: Thawing priority 1
2012/10/05 15:35:42.315733 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:42.315746 [ 5752]: Thawing priority 2
2012/10/05 15:35:42.315752 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:42.315761 [ 5752]: Thawing priority 3
2012/10/05 15:35:42.315767 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:42.329019 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:42.329065 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:42.329075 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:42.371518 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:42.371564 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:42.371574 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:43.204024 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:43.204668 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:43.204682 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:43.204689 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:43.225389 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:43.225425 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:43.225493 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:43.225503 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:43.226130 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:43.226144 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:43.226418 [ 5752]: Freeze priority 1
2012/10/05 15:35:43.227368 [ 5752]: Freeze priority 2
2012/10/05 15:35:43.228471 [ 5752]: Freeze priority 3
2012/10/05 15:35:43.229487 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:43.229868 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:43.229884 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:43.229893 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:43.280175 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:43.294026 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:43.294399 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:43.294688 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:43.294976 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:43.294989 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:43.295008 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:43.295275 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:43.295780 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:43.296025 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:43.296086 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:43.296382 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:43.296636 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:43.296660 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:43.296668 [ 5752]: Thawing priority 1
2012/10/05 15:35:43.296674 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:43.296690 [ 5752]: Thawing priority 2
2012/10/05 15:35:43.296696 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:43.296706 [ 5752]: Thawing priority 3
2012/10/05 15:35:43.296712 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:43.335205 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:43.335235 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:43.335243 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:43.335250 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:43.335256 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:43.335263 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:44.205698 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:44.206434 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:44.206452 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:44.206459 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:44.246782 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:44.246810 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:44.246868 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:44.246878 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:44.247465 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:44.247481 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:44.247773 [ 5752]: Freeze priority 1
2012/10/05 15:35:44.248586 [ 5752]: Freeze priority 2
2012/10/05 15:35:44.249283 [ 5752]: Freeze priority 3
2012/10/05 15:35:44.249970 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:44.250292 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:44.250315 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:44.250324 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:44.307034 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:44.333536 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:44.333941 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:44.334214 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:44.334483 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:44.334496 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:44.334515 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:44.334791 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:44.335344 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:44.335612 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:44.335697 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:44.336009 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:44.336330 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:44.336360 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:44.336370 [ 5752]: Thawing priority 1
2012/10/05 15:35:44.336376 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:44.336394 [ 5752]: Thawing priority 2
2012/10/05 15:35:44.336401 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:44.336412 [ 5752]: Thawing priority 3
2012/10/05 15:35:44.336418 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:44.380513 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:44.380544 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:44.380553 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:44.380560 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:44.380566 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:44.380573 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:45.207210 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:45.207917 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:45.207931 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:45.207938 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:45.227782 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:45.227799 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:45.227840 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:45.227850 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:45.228372 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:45.228386 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:45.228662 [ 5752]: Freeze priority 1
2012/10/05 15:35:45.229492 [ 5752]: Freeze priority 2
2012/10/05 15:35:45.230167 [ 5752]: Freeze priority 3
2012/10/05 15:35:45.230869 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:45.231167 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:45.231180 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:45.231188 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:45.292215 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:45.292377 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:45.292771 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:45.293056 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:45.293454 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:45.293470 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:45.293492 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:45.293793 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:45.294377 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:45.294682 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:45.294767 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:45.295126 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:45.295422 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:45.295448 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:45.295458 [ 5752]: Thawing priority 1
2012/10/05 15:35:45.295464 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:45.295477 [ 5752]: Thawing priority 2
2012/10/05 15:35:45.295483 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:45.295492 [ 5752]: Thawing priority 3
2012/10/05 15:35:45.295498 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:45.310168 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:45.310195 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:45.310203 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:45.357283 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:45.357310 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:45.357318 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:46.208753 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:46.209438 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:46.209451 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:46.209457 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:46.230300 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:46.230320 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:46.230361 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:46.230369 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:46.230868 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:46.230881 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:46.231150 [ 5752]: Freeze priority 1
2012/10/05 15:35:46.231916 [ 5752]: Freeze priority 2
2012/10/05 15:35:46.232545 [ 5752]: Freeze priority 3
2012/10/05 15:35:46.233172 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:46.233446 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:46.233458 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:46.233466 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:46.306220 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:46.306459 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:46.307068 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:46.307470 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:46.307831 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:46.307846 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:46.307866 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:46.308176 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:46.308788 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:46.309064 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:46.309131 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:46.309506 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:46.309793 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:46.309833 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:46.309843 [ 5752]: Thawing priority 1
2012/10/05 15:35:46.309849 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:46.309865 [ 5752]: Thawing priority 2
2012/10/05 15:35:46.309872 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:46.309881 [ 5752]: Thawing priority 3
2012/10/05 15:35:46.309887 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:46.352992 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:46.353030 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:46.353039 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:46.415147 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:46.415180 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:46.415188 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:47.210815 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:47.211526 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:47.211539 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:47.211545 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:47.236958 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:47.236994 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:47.237061 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:47.237070 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:47.237694 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:47.237708 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:47.238001 [ 5752]: Freeze priority 1
2012/10/05 15:35:47.238810 [ 5752]: Freeze priority 2
2012/10/05 15:35:47.239496 [ 5752]: Freeze priority 3
2012/10/05 15:35:47.240165 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:47.240453 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:47.240466 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:47.240474 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:47.308010 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:47.314265 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:47.314673 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:47.314941 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:47.315231 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:47.315245 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:47.315264 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:47.315537 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:47.316074 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:47.316335 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:47.316411 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:47.316722 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:47.317007 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:47.317035 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:47.317045 [ 5752]: Thawing priority 1
2012/10/05 15:35:47.317051 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:47.317068 [ 5752]: Thawing priority 2
2012/10/05 15:35:47.317075 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:47.317085 [ 5752]: Thawing priority 3
2012/10/05 15:35:47.317091 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:47.618935 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:47.618974 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:47.618982 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:47.618989 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:47.618995 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:47.619002 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:48.212323 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:48.213062 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:48.213085 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:48.213092 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:48.247065 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:48.247088 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:48.247142 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:48.247152 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:48.247722 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:48.247736 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:48.248027 [ 5752]: Freeze priority 1
2012/10/05 15:35:48.248840 [ 5752]: Freeze priority 2
2012/10/05 15:35:48.249536 [ 5752]: Freeze priority 3
2012/10/05 15:35:48.250228 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:48.250557 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:48.250570 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:48.250578 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:48.299277 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:48.309474 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:48.309867 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:48.310161 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:48.310467 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:48.310501 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:48.310525 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:48.310831 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:48.311394 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:48.311677 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:48.311754 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:48.312148 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:48.312446 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:48.312478 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:48.312489 [ 5752]: Thawing priority 1
2012/10/05 15:35:48.312495 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:48.312514 [ 5752]: Thawing priority 2
2012/10/05 15:35:48.312521 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:48.312532 [ 5752]: Thawing priority 3
2012/10/05 15:35:48.312539 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:48.332008 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:48.332045 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:48.332054 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:48.356805 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:48.356834 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:48.356842 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:49.214057 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:49.214787 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:49.215092 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:49.215104 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:49.234725 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:49.234752 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:49.234809 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:49.234820 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:49.235421 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:49.235436 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:49.235708 [ 5752]: Freeze priority 1
2012/10/05 15:35:49.236505 [ 5752]: Freeze priority 2
2012/10/05 15:35:49.237223 [ 5752]: Freeze priority 3
2012/10/05 15:35:49.237897 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:49.238215 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:49.238229 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:49.238237 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:49.308928 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:49.309130 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:49.309623 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:49.309893 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:49.310158 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:49.310171 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:49.310191 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:49.310446 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:49.310978 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:49.311266 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:49.311331 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:49.311697 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:49.311965 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:49.312002 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:49.312015 [ 5752]: Thawing priority 1
2012/10/05 15:35:49.312021 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:49.312035 [ 5752]: Thawing priority 2
2012/10/05 15:35:49.312042 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:49.312052 [ 5752]: Thawing priority 3
2012/10/05 15:35:49.312058 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:49.342285 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:49.342310 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:49.342318 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:49.368609 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:49.368630 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:49.368638 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:50.216020 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:50.216780 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:50.216795 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:50.216802 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:50.242287 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:50.242325 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:50.242394 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:50.242405 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:50.243049 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:50.243075 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:50.243366 [ 5752]: Freeze priority 1
2012/10/05 15:35:50.244216 [ 5752]: Freeze priority 2
2012/10/05 15:35:50.244885 [ 5752]: Freeze priority 3
2012/10/05 15:35:50.245544 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:50.245876 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:50.245896 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:50.245905 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:50.294708 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:50.294846 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:50.295210 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:50.295513 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:50.295823 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:50.295836 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:50.295856 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:50.296161 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:50.296656 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:50.296955 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:50.297016 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:50.297325 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:50.297594 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:50.297619 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:50.297628 [ 5752]: Thawing priority 1
2012/10/05 15:35:50.297634 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:50.297647 [ 5752]: Thawing priority 2
2012/10/05 15:35:50.297653 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:50.297662 [ 5752]: Thawing priority 3
2012/10/05 15:35:50.297668 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:50.316641 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:50.316677 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:50.316686 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:50.335868 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:50.335899 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:50.335907 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:51.217562 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:51.218264 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:51.218277 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:51.218283 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:51.252609 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:51.252660 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:51.252741 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:51.252750 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:51.253510 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:51.253539 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:51.253838 [ 5752]: Freeze priority 1
2012/10/05 15:35:51.254735 [ 5752]: Freeze priority 2
2012/10/05 15:35:51.255558 [ 5752]: Freeze priority 3
2012/10/05 15:35:51.256372 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:51.256681 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:51.256694 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:51.256702 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:51.319217 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:51.319361 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:51.319835 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:51.320148 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:51.320467 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:51.320483 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:51.320505 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:51.320804 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:51.321323 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:51.321569 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:51.321664 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:51.321992 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:51.322259 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:51.322286 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:51.322296 [ 5752]: Thawing priority 1
2012/10/05 15:35:51.322302 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:51.322317 [ 5752]: Thawing priority 2
2012/10/05 15:35:51.322324 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:51.322334 [ 5752]: Thawing priority 3
2012/10/05 15:35:51.322340 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:51.366440 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:51.366476 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:51.366485 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:51.366492 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:51.366498 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:51.366505 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:52.219077 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:52.219731 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:52.219745 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:52.219752 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:52.239470 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:52.239499 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:52.239557 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:52.239567 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:52.240175 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:52.240200 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:52.240491 [ 5752]: Freeze priority 1
2012/10/05 15:35:52.241310 [ 5752]: Freeze priority 2
2012/10/05 15:35:52.241998 [ 5752]: Freeze priority 3
2012/10/05 15:35:52.242680 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:52.243017 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:52.243030 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:52.243038 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:52.297096 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:52.317052 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:52.317495 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:52.317769 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:52.318019 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:52.318033 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:52.318054 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:52.318320 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:52.318773 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:52.319003 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:52.319070 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:52.319359 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:52.319605 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:52.319631 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:52.319641 [ 5752]: Thawing priority 1
2012/10/05 15:35:52.319648 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:52.319665 [ 5752]: Thawing priority 2
2012/10/05 15:35:52.319672 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:52.319682 [ 5752]: Thawing priority 3
2012/10/05 15:35:52.319702 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:52.333081 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:52.333109 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:52.333117 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:52.375402 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:52.375423 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:52.375431 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:53.220715 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:53.221480 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:53.221495 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:53.221502 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:53.246825 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:53.246849 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:53.246901 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:53.246911 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:53.247480 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:53.247495 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:53.247786 [ 5752]: Freeze priority 1
2012/10/05 15:35:53.248619 [ 5752]: Freeze priority 2
2012/10/05 15:35:53.249331 [ 5752]: Freeze priority 3
2012/10/05 15:35:53.250035 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:53.250364 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:53.250378 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:53.250386 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:53.308506 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:53.308658 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:53.309100 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:53.309409 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:53.309720 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:53.309734 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:53.309767 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:53.310054 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:53.310582 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:53.310842 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:53.310907 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:53.311221 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:53.311486 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:53.311513 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:53.311531 [ 5752]: Thawing priority 1
2012/10/05 15:35:53.311538 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:53.311551 [ 5752]: Thawing priority 2
2012/10/05 15:35:53.311558 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:53.311568 [ 5752]: Thawing priority 3
2012/10/05 15:35:53.311575 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:53.355451 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:53.355483 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:53.355492 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:53.355499 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:53.355506 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:53.355512 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:54.222311 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:54.223019 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:54.223224 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:54.223237 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:54.257926 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:54.257949 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:54.258000 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:54.258010 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:54.258580 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:54.258595 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:54.258886 [ 5752]: Freeze priority 1
2012/10/05 15:35:54.259702 [ 5752]: Freeze priority 2
2012/10/05 15:35:54.260386 [ 5752]: Freeze priority 3
2012/10/05 15:35:54.261072 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:54.261389 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:54.261404 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:54.261412 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:54.322052 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:54.339983 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:54.340385 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:54.340686 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:54.340999 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:54.341014 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:54.341035 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:54.341341 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:54.341861 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:54.342115 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:54.342176 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:54.342518 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:54.342786 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:54.342811 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:54.342819 [ 5752]: Thawing priority 1
2012/10/05 15:35:54.342838 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:54.342855 [ 5752]: Thawing priority 2
2012/10/05 15:35:54.342872 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:54.342883 [ 5752]: Thawing priority 3
2012/10/05 15:35:54.342889 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:54.386540 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:54.386567 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:54.386575 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:54.386581 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:54.386587 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:54.386593 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:55.224313 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:55.224969 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:55.224984 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:55.224991 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:55.244946 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:55.244965 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:55.245008 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:55.245018 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:55.245569 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:55.245584 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:55.245878 [ 5752]: Freeze priority 1
2012/10/05 15:35:55.246711 [ 5752]: Freeze priority 2
2012/10/05 15:35:55.247415 [ 5752]: Freeze priority 3
2012/10/05 15:35:55.248120 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:55.248429 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:55.248442 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:55.248451 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:55.297140 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:55.297278 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:55.297681 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:55.297962 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:55.298259 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:55.298275 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:55.298296 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:55.298572 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:55.299157 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:55.299465 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:55.299530 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:55.299871 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:55.300181 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:55.300206 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:55.300215 [ 5752]: Thawing priority 1
2012/10/05 15:35:55.300221 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:55.300234 [ 5752]: Thawing priority 2
2012/10/05 15:35:55.300240 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:55.300249 [ 5752]: Thawing priority 3
2012/10/05 15:35:55.300255 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:55.313791 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:55.313814 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:55.313821 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:55.355781 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:55.355802 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:55.355810 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:56.226438 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:56.227128 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:56.227144 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:56.227151 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:56.252375 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:56.252395 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:56.252453 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:56.252465 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:56.252999 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:56.253012 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:56.253269 [ 5752]: Freeze priority 1
2012/10/05 15:35:56.254114 [ 5752]: Freeze priority 2
2012/10/05 15:35:56.254775 [ 5752]: Freeze priority 3
2012/10/05 15:35:56.255490 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:56.255807 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:56.255821 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:56.255829 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:56.341445 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:56.341605 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:56.341998 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:56.342281 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:56.342600 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:56.342617 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:56.342638 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:56.342938 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:56.343483 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:56.343752 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:56.343815 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:56.344141 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:56.344411 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:56.344436 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:56.344445 [ 5752]: Thawing priority 1
2012/10/05 15:35:56.344464 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:56.344478 [ 5752]: Thawing priority 2
2012/10/05 15:35:56.344485 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:56.344496 [ 5752]: Thawing priority 3
2012/10/05 15:35:56.344502 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:56.385276 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:56.385328 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:56.385337 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:56.396688 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:56.396718 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:56.396726 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:57.228483 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:57.229154 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:57.229168 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:57.229175 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:57.248221 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:57.248253 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:57.248317 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:57.248325 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:57.248944 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:57.248957 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:57.249234 [ 5752]: Freeze priority 1
2012/10/05 15:35:57.250029 [ 5752]: Freeze priority 2
2012/10/05 15:35:57.250684 [ 5752]: Freeze priority 3
2012/10/05 15:35:57.251344 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:57.251660 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:57.251672 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:57.251680 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:57.310095 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:57.310237 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:57.310600 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:57.310882 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:57.311173 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:57.311189 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:57.311212 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:57.311478 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:57.311999 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:57.312261 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:57.312325 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:57.312670 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:57.312926 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:57.312952 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:57.312961 [ 5752]: Thawing priority 1
2012/10/05 15:35:57.312966 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:57.312979 [ 5752]: Thawing priority 2
2012/10/05 15:35:57.312985 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:57.312994 [ 5752]: Thawing priority 3
2012/10/05 15:35:57.313000 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:57.326406 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:57.326445 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:57.326454 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:57.355903 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:57.355934 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:57.355942 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:58.230549 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:58.231253 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:58.231268 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:58.231274 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:58.265875 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:58.265897 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:58.265945 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:58.265955 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:58.266506 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:58.266519 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:58.266807 [ 5752]: Freeze priority 1
2012/10/05 15:35:58.267605 [ 5752]: Freeze priority 2
2012/10/05 15:35:58.268252 [ 5752]: Freeze priority 3
2012/10/05 15:35:58.268908 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:58.269224 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:58.269238 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:58.269245 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:58.334341 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:58.334492 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:58.334894 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:58.335190 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:58.335449 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:58.335463 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:58.335482 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:58.335779 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:58.336276 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:58.336551 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:58.336641 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:58.336942 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:58.337199 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:58.337227 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:58.337237 [ 5752]: Thawing priority 1
2012/10/05 15:35:58.337243 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:58.337256 [ 5752]: Thawing priority 2
2012/10/05 15:35:58.337263 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:58.337273 [ 5752]: Thawing priority 3
2012/10/05 15:35:58.337279 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:58.382574 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:58.382618 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:58.382628 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:58.382635 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:58.382642 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:58.382658 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:35:59.232140 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:35:59.232959 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:35:59.233661 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:35:59.233676 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:35:59.253124 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:35:59.253170 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:35:59.253239 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:35:59.253248 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:35:59.253924 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:35:59.253937 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:35:59.254233 [ 5752]: Freeze priority 1
2012/10/05 15:35:59.255036 [ 5752]: Freeze priority 2
2012/10/05 15:35:59.255704 [ 5752]: Freeze priority 3
2012/10/05 15:35:59.256372 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:35:59.256708 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:35:59.256720 [ 5752]: Monitoring has been disabled
2012/10/05 15:35:59.256728 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:35:59.307331 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:35:59.309570 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:59.309946 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:59.310224 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:35:59.310517 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:35:59.310531 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:35:59.310551 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:35:59.310828 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:35:59.311400 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:35:59.311704 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:35:59.311769 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:35:59.312097 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:35:59.312459 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:35:59.312491 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:35:59.312502 [ 5752]: Thawing priority 1
2012/10/05 15:35:59.312508 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:35:59.312524 [ 5752]: Thawing priority 2
2012/10/05 15:35:59.312531 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:35:59.312541 [ 5752]: Thawing priority 3
2012/10/05 15:35:59.312548 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:35:59.325982 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:59.326017 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:35:59.326025 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:35:59.369004 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:35:59.369039 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:35:59.369047 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:00.233672 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:00.234394 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:00.234410 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:00.234416 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:00.259585 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:00.259604 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:00.259650 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:00.259659 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:00.260177 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:00.260190 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:00.260468 [ 5752]: Freeze priority 1
2012/10/05 15:36:00.261266 [ 5752]: Freeze priority 2
2012/10/05 15:36:00.261945 [ 5752]: Freeze priority 3
2012/10/05 15:36:00.262610 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:00.262933 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:00.262946 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:00.262954 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:00.310071 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:00.325148 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:00.325511 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:00.325807 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:00.326091 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:00.326105 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:00.326125 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:00.326401 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:00.326920 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:00.327182 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:00.327262 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:00.327580 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:00.327853 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:00.327880 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:00.327889 [ 5752]: Thawing priority 1
2012/10/05 15:36:00.327895 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:00.327913 [ 5752]: Thawing priority 2
2012/10/05 15:36:00.327920 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:00.327931 [ 5752]: Thawing priority 3
2012/10/05 15:36:00.327938 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:00.370648 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:00.370685 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:00.370694 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:00.383070 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:00.383085 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:00.383092 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:01.235669 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:01.236454 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:01.236508 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:01.236515 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:01.255701 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:01.255735 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:01.255814 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:01.255824 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:01.256423 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:01.256436 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:01.256704 [ 5752]: Freeze priority 1
2012/10/05 15:36:01.257470 [ 5752]: Freeze priority 2
2012/10/05 15:36:01.258106 [ 5752]: Freeze priority 3
2012/10/05 15:36:01.258739 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:01.259041 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:01.259053 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:01.259061 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:01.313006 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:01.313131 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:01.313459 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:01.313737 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:01.314001 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:01.314015 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:01.314036 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:01.314267 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:01.314774 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:01.315014 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:01.315080 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:01.315366 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:01.315598 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:01.315625 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:01.315635 [ 5752]: Thawing priority 1
2012/10/05 15:36:01.315641 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:01.315654 [ 5752]: Thawing priority 2
2012/10/05 15:36:01.315661 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:01.315671 [ 5752]: Thawing priority 3
2012/10/05 15:36:01.315678 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:01.332524 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:01.332551 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:01.332559 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:01.359838 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:01.359872 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:01.359880 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:02.237303 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:02.238046 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:02.238062 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:02.238081 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:02.275692 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:02.275720 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:02.275777 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:02.275786 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:02.276354 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:02.276368 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:02.276616 [ 5752]: Freeze priority 1
2012/10/05 15:36:02.277446 [ 5752]: Freeze priority 2
2012/10/05 15:36:02.278078 [ 5752]: Freeze priority 3
2012/10/05 15:36:02.278695 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:02.278975 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:02.278988 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:02.278996 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:02.325859 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:02.328719 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:02.329042 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:02.329304 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:02.329594 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:02.329609 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:02.329630 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:02.329909 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:02.330427 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:02.330683 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:02.330749 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:02.331074 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:02.331340 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:02.331379 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:02.331389 [ 5752]: Thawing priority 1
2012/10/05 15:36:02.331395 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:02.331410 [ 5752]: Thawing priority 2
2012/10/05 15:36:02.331417 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:02.331427 [ 5752]: Thawing priority 3
2012/10/05 15:36:02.331434 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:02.375860 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:02.375896 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:02.375905 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:02.375911 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:02.375917 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:02.375923 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:03.239438 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:03.240076 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:03.240093 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:03.240100 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:03.259616 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:03.259658 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:03.259745 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:03.259756 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:03.260344 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:03.260365 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:03.260645 [ 5752]: Freeze priority 1
2012/10/05 15:36:03.261423 [ 5752]: Freeze priority 2
2012/10/05 15:36:03.262137 [ 5752]: Freeze priority 3
2012/10/05 15:36:03.262754 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:03.263016 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:03.263030 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:03.263040 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:03.311801 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:03.312352 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:03.312696 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:03.312961 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:03.313239 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:03.313253 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:03.313274 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:03.313557 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:03.314085 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:03.314348 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:03.314429 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:03.314744 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:03.315079 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:03.315105 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:03.315115 [ 5752]: Thawing priority 1
2012/10/05 15:36:03.315121 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:03.315135 [ 5752]: Thawing priority 2
2012/10/05 15:36:03.315142 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:03.315152 [ 5752]: Thawing priority 3
2012/10/05 15:36:03.315159 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:03.328643 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:03.328669 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:03.328677 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:03.369757 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:03.369791 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:03.369799 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:04.241429 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:04.242130 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:04.242327 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:04.242340 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:04.267509 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:04.267550 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:04.267621 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:04.267645 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:04.268271 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:04.268284 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:04.268562 [ 5752]: Freeze priority 1
2012/10/05 15:36:04.269361 [ 5752]: Freeze priority 2
2012/10/05 15:36:04.270030 [ 5752]: Freeze priority 3
2012/10/05 15:36:04.270689 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:04.271002 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:04.271016 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:04.271024 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:04.318906 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:04.337508 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:04.337864 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:04.338120 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:04.338346 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:04.338359 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:04.338378 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:04.338634 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:04.339076 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:04.339290 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:04.339352 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:04.339629 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:04.339865 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:04.339890 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:04.339898 [ 5752]: Thawing priority 1
2012/10/05 15:36:04.339905 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:04.339921 [ 5752]: Thawing priority 2
2012/10/05 15:36:04.339927 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:04.339936 [ 5752]: Thawing priority 3
2012/10/05 15:36:04.339943 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:04.383138 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:04.383174 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:04.383183 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:04.391885 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:04.391898 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:04.391905 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:05.243506 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:05.244186 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:05.244202 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:05.244208 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:05.263516 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:05.263566 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:05.263639 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:05.263648 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:05.264225 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:05.264250 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:05.264525 [ 5752]: Freeze priority 1
2012/10/05 15:36:05.265373 [ 5752]: Freeze priority 2
2012/10/05 15:36:05.266185 [ 5752]: Freeze priority 3
2012/10/05 15:36:05.266914 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:05.267246 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:05.267261 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:05.267269 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:05.340630 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:05.340763 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:05.341133 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:05.341345 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:05.341587 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:05.341601 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:05.341621 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:05.341883 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:05.342345 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:05.342581 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:05.342647 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:05.342931 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:05.343171 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:05.343198 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:05.343207 [ 5752]: Thawing priority 1
2012/10/05 15:36:05.343214 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:05.343227 [ 5752]: Thawing priority 2
2012/10/05 15:36:05.343234 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:05.343244 [ 5752]: Thawing priority 3
2012/10/05 15:36:05.343250 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:05.356809 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:05.356850 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:05.356859 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:05.405192 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:05.405226 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:05.405235 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:06.244526 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:06.245154 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:06.245171 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:06.245177 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:06.264461 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:06.264496 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:06.264563 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:06.264572 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:06.265192 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:06.265205 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:06.265491 [ 5752]: Freeze priority 1
2012/10/05 15:36:06.266337 [ 5752]: Freeze priority 2
2012/10/05 15:36:06.266998 [ 5752]: Freeze priority 3
2012/10/05 15:36:06.267690 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:06.267991 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:06.268004 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:06.268012 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:06.316620 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:06.316750 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:06.317162 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:06.317475 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:06.317803 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:06.317818 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:06.317840 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:06.318146 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:06.318757 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:06.319044 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:06.319137 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:06.319518 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:06.319825 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:06.319855 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:06.319865 [ 5752]: Thawing priority 1
2012/10/05 15:36:06.319871 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:06.319889 [ 5752]: Thawing priority 2
2012/10/05 15:36:06.319895 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:06.319905 [ 5752]: Thawing priority 3
2012/10/05 15:36:06.319911 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:06.358770 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:06.358805 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:06.358814 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:06.358820 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:06.358826 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:06.358832 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:07.246142 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:07.246845 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:07.246857 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:07.246864 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:07.287342 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:07.287380 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:07.287446 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:07.287455 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:07.288052 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:07.288066 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:07.288383 [ 5752]: Freeze priority 1
2012/10/05 15:36:07.289250 [ 5752]: Freeze priority 2
2012/10/05 15:36:07.290076 [ 5752]: Freeze priority 3
2012/10/05 15:36:07.290952 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:07.291294 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:07.291308 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:07.291317 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:07.351323 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:07.351487 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:07.352036 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:07.352344 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:07.352671 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:07.352687 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:07.352709 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:07.353004 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:07.353571 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:07.353882 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:07.353953 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:07.354309 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:07.354590 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:07.354617 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:07.354626 [ 5752]: Thawing priority 1
2012/10/05 15:36:07.354632 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:07.354645 [ 5752]: Thawing priority 2
2012/10/05 15:36:07.354651 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:07.354660 [ 5752]: Thawing priority 3
2012/10/05 15:36:07.354666 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:07.399235 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:07.399270 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:07.399279 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:07.399286 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:07.399292 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:07.399298 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:08.247706 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:08.248430 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:08.248444 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:08.248450 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:08.268035 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:08.268052 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:08.268095 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:08.268106 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:08.268637 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:08.268651 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:08.268952 [ 5752]: Freeze priority 1
2012/10/05 15:36:08.269782 [ 5752]: Freeze priority 2
2012/10/05 15:36:08.270480 [ 5752]: Freeze priority 3
2012/10/05 15:36:08.271166 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:08.271478 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:08.271500 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:08.271509 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:08.335569 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:08.335721 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:08.336148 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:08.336445 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:08.336726 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:08.336751 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:08.336775 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:08.337035 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:08.337572 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:08.337848 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:08.337915 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:08.338253 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:08.338542 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:08.338569 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:08.338579 [ 5752]: Thawing priority 1
2012/10/05 15:36:08.338585 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:08.338598 [ 5752]: Thawing priority 2
2012/10/05 15:36:08.338605 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:08.338615 [ 5752]: Thawing priority 3
2012/10/05 15:36:08.338621 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:08.351852 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:08.351875 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:08.351883 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:08.401019 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:08.401046 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:08.401054 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:09.249743 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:09.250457 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:09.250929 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:09.250941 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:09.270246 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:09.270261 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:09.270313 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:09.270323 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:09.270832 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:09.270845 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:09.271127 [ 5752]: Freeze priority 1
2012/10/05 15:36:09.271906 [ 5752]: Freeze priority 2
2012/10/05 15:36:09.272552 [ 5752]: Freeze priority 3
2012/10/05 15:36:09.273195 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:09.273498 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:09.273510 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:09.273518 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:09.319795 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:09.322475 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:09.322830 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:09.323082 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:09.323355 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:09.323368 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:09.323387 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:09.323640 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:09.324132 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:09.324396 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:09.324458 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:09.324753 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:09.325035 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:09.325066 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:09.325076 [ 5752]: Thawing priority 1
2012/10/05 15:36:09.325083 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:09.325096 [ 5752]: Thawing priority 2
2012/10/05 15:36:09.325103 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:09.325112 [ 5752]: Thawing priority 3
2012/10/05 15:36:09.325118 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:09.363672 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:09.363707 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:09.363715 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:09.363722 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:09.363728 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:09.363734 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:10.251324 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:10.251995 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:10.252008 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:10.252015 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:10.293842 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:10.293864 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:10.293923 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:10.293933 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:10.294445 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:10.294457 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:10.294731 [ 5752]: Freeze priority 1
2012/10/05 15:36:10.295513 [ 5752]: Freeze priority 2
2012/10/05 15:36:10.296158 [ 5752]: Freeze priority 3
2012/10/05 15:36:10.296783 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:10.297067 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:10.297079 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:10.297087 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:10.355852 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:10.356012 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:10.356485 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:10.356755 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:10.357033 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:10.357046 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:10.357065 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:10.357328 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:10.357862 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:10.358160 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:10.358231 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:10.358562 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:10.358806 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:10.358832 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:10.358840 [ 5752]: Thawing priority 1
2012/10/05 15:36:10.358846 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:10.358859 [ 5752]: Thawing priority 2
2012/10/05 15:36:10.358865 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:10.358887 [ 5752]: Thawing priority 3
2012/10/05 15:36:10.358895 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:10.403563 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:10.403593 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:10.403602 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:10.403608 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:10.403614 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:10.403620 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:11.252918 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:11.253603 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:11.253617 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:11.253623 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:11.273632 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:11.273665 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:11.273730 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:11.273740 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:11.274366 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:11.274383 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:11.274666 [ 5752]: Freeze priority 1
2012/10/05 15:36:11.275406 [ 5752]: Freeze priority 2
2012/10/05 15:36:11.276075 [ 5752]: Freeze priority 3
2012/10/05 15:36:11.276714 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:11.276997 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:11.277010 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:11.277018 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:11.327745 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:11.327888 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:11.328265 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:11.328545 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:11.328852 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:11.328867 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:11.328889 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:11.329185 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:11.329757 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:11.330037 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:11.330104 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:11.330448 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:11.330696 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:11.330723 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:11.330733 [ 5752]: Thawing priority 1
2012/10/05 15:36:11.330739 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:11.330753 [ 5752]: Thawing priority 2
2012/10/05 15:36:11.330760 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:11.330769 [ 5752]: Thawing priority 3
2012/10/05 15:36:11.330776 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:11.344159 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:11.344192 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:11.344201 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:11.393012 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:11.393046 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:11.393054 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:12.254455 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:12.255210 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:12.255226 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:12.255232 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:12.274521 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:12.274547 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:12.274600 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:12.274609 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:12.275190 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:12.275203 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:12.275475 [ 5752]: Freeze priority 1
2012/10/05 15:36:12.276315 [ 5752]: Freeze priority 2
2012/10/05 15:36:12.276985 [ 5752]: Freeze priority 3
2012/10/05 15:36:12.277661 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:12.277959 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:12.277972 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:12.277980 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:12.335212 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:12.335359 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:12.335780 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:12.336041 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:12.336303 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:12.336324 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:12.336344 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:12.336643 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:12.337189 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:12.337427 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:12.337506 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:12.337829 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:12.338072 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:12.338098 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:12.338107 [ 5752]: Thawing priority 1
2012/10/05 15:36:12.338113 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:12.338126 [ 5752]: Thawing priority 2
2012/10/05 15:36:12.338133 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:12.338142 [ 5752]: Thawing priority 3
2012/10/05 15:36:12.338148 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:12.381867 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:12.381902 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:12.381911 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:12.394831 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:12.394846 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:12.394853 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:13.256545 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:13.257272 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:13.257287 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:13.257293 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:13.282408 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:13.282424 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:13.282464 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:13.282474 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:13.282999 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:13.283013 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:13.283307 [ 5752]: Freeze priority 1
2012/10/05 15:36:13.284104 [ 5752]: Freeze priority 2
2012/10/05 15:36:13.284765 [ 5752]: Freeze priority 3
2012/10/05 15:36:13.285482 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:13.285792 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:13.285808 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:13.285816 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:13.334078 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:13.347992 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:13.348374 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:13.348700 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:13.348993 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:13.349007 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:13.349026 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:13.349304 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:13.349827 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:13.350103 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:13.350169 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:13.350493 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:13.350779 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:13.350806 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:13.350815 [ 5752]: Thawing priority 1
2012/10/05 15:36:13.350821 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:13.350838 [ 5752]: Thawing priority 2
2012/10/05 15:36:13.350845 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:13.350855 [ 5752]: Thawing priority 3
2012/10/05 15:36:13.350862 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:13.382325 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:13.382352 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:13.382360 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:13.406776 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:13.406794 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:13.406802 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:14.258112 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:14.258771 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:14.259160 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:14.259172 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:14.278598 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:14.278613 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:14.278666 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:14.278676 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:14.279181 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:14.279194 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:14.279482 [ 5752]: Freeze priority 1
2012/10/05 15:36:14.280316 [ 5752]: Freeze priority 2
2012/10/05 15:36:14.280990 [ 5752]: Freeze priority 3
2012/10/05 15:36:14.281683 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:14.281991 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:14.282004 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:14.282012 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:14.358552 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:14.358724 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:14.359176 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:14.359412 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:14.359661 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:14.359675 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:14.359695 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:14.359926 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:14.360380 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:14.360618 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:14.360696 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:14.360973 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:14.361250 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:14.361277 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:14.361287 [ 5752]: Thawing priority 1
2012/10/05 15:36:14.361293 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:14.361306 [ 5752]: Thawing priority 2
2012/10/05 15:36:14.361313 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:14.361323 [ 5752]: Thawing priority 3
2012/10/05 15:36:14.361329 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:14.461199 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:14.461232 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:14.461241 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:14.461248 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:14.461254 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:14.461260 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:15.259638 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:15.260338 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:15.260351 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:15.260357 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:15.279902 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:15.279939 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:15.280010 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:15.280020 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:15.280659 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:15.280673 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:15.280978 [ 5752]: Freeze priority 1
2012/10/05 15:36:15.281762 [ 5752]: Freeze priority 2
2012/10/05 15:36:15.282397 [ 5752]: Freeze priority 3
2012/10/05 15:36:15.283044 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:15.283326 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:15.283339 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:15.283346 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:15.329287 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:15.329684 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:15.330127 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:15.330657 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:15.330963 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:15.330977 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:15.330997 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:15.331298 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:15.331822 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:15.332063 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:15.332128 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:15.332432 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:15.332672 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:15.332715 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:15.332726 [ 5752]: Thawing priority 1
2012/10/05 15:36:15.332732 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:15.332747 [ 5752]: Thawing priority 2
2012/10/05 15:36:15.332753 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:15.332763 [ 5752]: Thawing priority 3
2012/10/05 15:36:15.332769 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:15.346239 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:15.346268 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:15.346277 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:15.370710 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:15.370745 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:15.370753 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:16.261211 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:16.261926 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:16.261939 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:16.261946 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:16.303184 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:16.303222 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:16.303308 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:16.303319 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:16.303951 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:16.303965 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:16.304254 [ 5752]: Freeze priority 1
2012/10/05 15:36:16.305102 [ 5752]: Freeze priority 2
2012/10/05 15:36:16.305828 [ 5752]: Freeze priority 3
2012/10/05 15:36:16.306519 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:16.306824 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:16.306837 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:16.306845 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:16.363149 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:16.366612 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:16.366983 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:16.367289 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:16.367596 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:16.367611 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:16.367631 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:16.367927 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:16.368566 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:16.368858 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:16.368931 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:16.369319 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:16.369605 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:16.369648 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:16.369658 [ 5752]: Thawing priority 1
2012/10/05 15:36:16.369665 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:16.369683 [ 5752]: Thawing priority 2
2012/10/05 15:36:16.369690 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:16.369701 [ 5752]: Thawing priority 3
2012/10/05 15:36:16.369708 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:16.412753 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:16.412820 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:16.412831 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:16.412839 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:16.412845 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:16.412852 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:17.263277 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:17.264006 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:17.264020 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:17.264027 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:17.283547 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:17.283564 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:17.283604 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:17.283614 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:17.284131 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:17.284145 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:17.284462 [ 5752]: Freeze priority 1
2012/10/05 15:36:17.285288 [ 5752]: Freeze priority 2
2012/10/05 15:36:17.286035 [ 5752]: Freeze priority 3
2012/10/05 15:36:17.286763 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:17.287063 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:17.287077 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:17.287085 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:17.335668 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:17.335808 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:17.336191 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:17.336425 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:17.336662 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:17.336676 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:17.336697 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:17.336926 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:17.337397 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:17.337622 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:17.337687 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:17.338003 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:17.338257 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:17.338284 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:17.338302 [ 5752]: Thawing priority 1
2012/10/05 15:36:17.338309 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:17.338322 [ 5752]: Thawing priority 2
2012/10/05 15:36:17.338342 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:17.338355 [ 5752]: Thawing priority 3
2012/10/05 15:36:17.338362 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:17.351084 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:17.351108 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:17.351116 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:17.399926 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:17.399959 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:17.399967 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:18.264323 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:18.265091 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:18.265106 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:18.265113 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:18.284136 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:18.284151 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:18.284189 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:18.284198 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:18.284711 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:18.284724 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:18.285027 [ 5752]: Freeze priority 1
2012/10/05 15:36:18.285851 [ 5752]: Freeze priority 2
2012/10/05 15:36:18.286568 [ 5752]: Freeze priority 3
2012/10/05 15:36:18.287256 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:18.287559 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:18.287572 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:18.287580 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:18.337174 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:18.337306 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:18.337698 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:18.338052 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:18.338327 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:18.338342 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:18.338363 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:18.338616 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:18.339107 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:18.339353 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:18.339436 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:18.339736 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:18.339986 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:18.340013 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:18.340022 [ 5752]: Thawing priority 1
2012/10/05 15:36:18.340028 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:18.340042 [ 5752]: Thawing priority 2
2012/10/05 15:36:18.340057 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:18.340069 [ 5752]: Thawing priority 3
2012/10/05 15:36:18.340076 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:18.383136 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:18.383168 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:18.383179 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:18.451077 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:18.451111 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:18.451119 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:19.265389 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:19.266078 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:19.266539 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:19.266553 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:19.292138 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:19.292159 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:19.292209 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:19.292219 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:19.292780 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:19.292794 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:19.293099 [ 5752]: Freeze priority 1
2012/10/05 15:36:19.293904 [ 5752]: Freeze priority 2
2012/10/05 15:36:19.294604 [ 5752]: Freeze priority 3
2012/10/05 15:36:19.295301 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:19.295644 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:19.295658 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:19.295666 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:19.343356 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:19.345976 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:19.346286 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:19.346579 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:19.346834 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:19.346849 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:19.346870 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:19.347132 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:19.347589 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:19.347814 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:19.347880 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:19.348220 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:19.348476 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:19.348502 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:19.348512 [ 5752]: Thawing priority 1
2012/10/05 15:36:19.348518 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:19.348532 [ 5752]: Thawing priority 2
2012/10/05 15:36:19.348539 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:19.348549 [ 5752]: Thawing priority 3
2012/10/05 15:36:19.348556 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:19.367387 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:19.367413 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:19.367422 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:19.385348 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:19.385362 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:19.385369 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:20.266948 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:20.267557 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:20.267572 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:20.267578 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:20.303051 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:20.303100 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:20.303158 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:20.303168 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:20.303762 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:20.303775 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:20.304069 [ 5752]: Freeze priority 1
2012/10/05 15:36:20.304853 [ 5752]: Freeze priority 2
2012/10/05 15:36:20.305533 [ 5752]: Freeze priority 3
2012/10/05 15:36:20.306213 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:20.306505 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:20.306530 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:20.306539 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:20.356063 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:20.368388 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:20.368771 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:20.369064 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:20.369353 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:20.369368 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:20.369388 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:20.369720 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:20.370309 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:20.370571 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:20.370638 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:20.370947 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:20.371216 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:20.371247 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:20.371258 [ 5752]: Thawing priority 1
2012/10/05 15:36:20.371264 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:20.371282 [ 5752]: Thawing priority 2
2012/10/05 15:36:20.371288 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:20.371297 [ 5752]: Thawing priority 3
2012/10/05 15:36:20.371303 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:20.414775 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:20.414808 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:20.414829 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:20.415561 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:20.415594 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:20.415603 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:21.269053 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:21.269787 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:21.269802 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:21.269809 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:21.289286 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:21.289304 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:21.289349 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:21.289359 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:21.289876 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:21.289891 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:21.290188 [ 5752]: Freeze priority 1
2012/10/05 15:36:21.291013 [ 5752]: Freeze priority 2
2012/10/05 15:36:21.291707 [ 5752]: Freeze priority 3
2012/10/05 15:36:21.292393 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:21.292717 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:21.292730 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:21.292739 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:21.351120 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:21.351268 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:21.351681 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:21.352005 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:21.352327 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:21.352342 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:21.352362 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:21.352653 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:21.353230 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:21.353509 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:21.353575 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:21.353917 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:21.354202 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:21.354228 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:21.354238 [ 5752]: Thawing priority 1
2012/10/05 15:36:21.354244 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:21.354258 [ 5752]: Thawing priority 2
2012/10/05 15:36:21.354264 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:21.354274 [ 5752]: Thawing priority 3
2012/10/05 15:36:21.354281 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:21.367502 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:21.367531 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:21.367540 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:21.403485 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:21.403525 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:21.403535 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:22.271088 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:22.271813 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:22.271827 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:22.271834 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:22.298387 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:22.298423 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:22.298490 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:22.298500 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:22.299106 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:22.299119 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:22.299411 [ 5752]: Freeze priority 1
2012/10/05 15:36:22.300216 [ 5752]: Freeze priority 2
2012/10/05 15:36:22.300895 [ 5752]: Freeze priority 3
2012/10/05 15:36:22.301555 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:22.301841 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:22.301854 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:22.301862 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:22.350659 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:22.380798 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:22.381197 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:22.381467 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:22.381774 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:22.381789 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:22.381809 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:22.382078 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:22.382596 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:22.382859 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:22.382925 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:22.383268 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:22.383596 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:22.383651 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:22.383665 [ 5752]: Thawing priority 1
2012/10/05 15:36:22.383671 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:22.383690 [ 5752]: Thawing priority 2
2012/10/05 15:36:22.383697 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:22.383708 [ 5752]: Thawing priority 3
2012/10/05 15:36:22.383714 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:22.427319 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:22.427362 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:22.427372 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:22.437286 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:22.437304 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:22.437321 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:23.272690 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:23.273459 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:23.273476 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:23.273483 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:23.293415 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:23.293448 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:23.293508 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:23.293518 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:23.294131 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:23.294146 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:23.294539 [ 5752]: Freeze priority 1
2012/10/05 15:36:23.295393 [ 5752]: Freeze priority 2
2012/10/05 15:36:23.296062 [ 5752]: Freeze priority 3
2012/10/05 15:36:23.296758 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:23.297085 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:23.297099 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:23.297117 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:23.357777 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:23.359747 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:23.360105 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:23.360382 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:23.360678 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:23.360693 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:23.360714 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:23.361009 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:23.361562 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:23.361835 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:23.361901 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:23.362243 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:23.362522 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:23.362548 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:23.362558 [ 5752]: Thawing priority 1
2012/10/05 15:36:23.362564 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:23.362579 [ 5752]: Thawing priority 2
2012/10/05 15:36:23.362586 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:23.362596 [ 5752]: Thawing priority 3
2012/10/05 15:36:23.362603 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:23.376031 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:23.376061 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:23.376069 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:23.406470 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:23.406518 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:23.406528 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:24.274224 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:24.274881 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:24.275213 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:24.275225 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:24.310451 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:24.310487 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:24.310555 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:24.310565 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:24.311169 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:24.311186 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:24.311592 [ 5752]: Freeze priority 1
2012/10/05 15:36:24.312410 [ 5752]: Freeze priority 2
2012/10/05 15:36:24.313055 [ 5752]: Freeze priority 3
2012/10/05 15:36:24.313704 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:24.314000 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:24.314013 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:24.314021 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:24.359211 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:24.388166 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:24.388527 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:24.388765 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:24.389030 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:24.389045 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:24.389067 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:24.389316 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:24.389779 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:24.390031 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:24.390097 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:24.390395 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:24.390636 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:24.390663 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:24.390673 [ 5752]: Thawing priority 1
2012/10/05 15:36:24.390679 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:24.390696 [ 5752]: Thawing priority 2
2012/10/05 15:36:24.390703 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:24.390714 [ 5752]: Thawing priority 3
2012/10/05 15:36:24.390721 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:24.435384 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:24.435417 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:24.435425 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:24.435432 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:24.435438 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:24.435445 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:25.275818 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:25.276494 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:25.276519 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:25.276526 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:25.297025 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:25.297053 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:25.297113 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:25.297124 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:25.297668 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:25.297684 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:25.297975 [ 5752]: Freeze priority 1
2012/10/05 15:36:25.298789 [ 5752]: Freeze priority 2
2012/10/05 15:36:25.299459 [ 5752]: Freeze priority 3
2012/10/05 15:36:25.300122 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:25.300439 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:25.300452 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:25.300460 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:25.358504 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:25.370841 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:25.371257 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:25.371544 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:25.371838 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:25.371863 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:25.371888 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:25.372168 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:25.372717 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:25.372999 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:25.373065 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:25.373419 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:25.373712 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:25.373739 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:25.373749 [ 5752]: Thawing priority 1
2012/10/05 15:36:25.373755 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:25.373773 [ 5752]: Thawing priority 2
2012/10/05 15:36:25.373780 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:25.373790 [ 5752]: Thawing priority 3
2012/10/05 15:36:25.373797 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:25.388205 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:25.388232 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:25.388241 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:25.430643 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:25.430673 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:25.430682 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:26.277354 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:26.278041 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:26.278054 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:26.278072 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:26.304902 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:26.304932 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:26.304978 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:26.304987 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:26.305500 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:26.305513 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:26.305798 [ 5752]: Freeze priority 1
2012/10/05 15:36:26.306593 [ 5752]: Freeze priority 2
2012/10/05 15:36:26.307265 [ 5752]: Freeze priority 3
2012/10/05 15:36:26.307935 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:26.308211 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:26.308223 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:26.308231 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:26.355203 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:26.361618 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:26.362029 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:26.362322 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:26.362586 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:26.362601 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:26.362621 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:26.362868 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:26.363340 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:26.363595 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:26.363661 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:26.363965 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:26.364227 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:26.364253 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:26.364263 [ 5752]: Thawing priority 1
2012/10/05 15:36:26.364269 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:26.364285 [ 5752]: Thawing priority 2
2012/10/05 15:36:26.364292 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:26.364303 [ 5752]: Thawing priority 3
2012/10/05 15:36:26.364309 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:26.408798 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:26.408830 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:26.408839 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:26.408846 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:26.408852 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:26.408858 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:27.279383 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:27.280035 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:27.280048 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:27.280054 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:27.315737 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:27.315763 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:27.315830 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:27.315840 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:27.316401 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:27.316414 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:27.316671 [ 5752]: Freeze priority 1
2012/10/05 15:36:27.317444 [ 5752]: Freeze priority 2
2012/10/05 15:36:27.318130 [ 5752]: Freeze priority 3
2012/10/05 15:36:27.318780 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:27.319097 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:27.319109 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:27.319117 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:27.366417 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:27.378760 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:27.379105 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:27.379362 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:27.379667 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:27.379684 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:27.379706 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:27.379985 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:27.380508 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:27.380810 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:27.380874 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:27.381196 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:27.381456 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:27.381493 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:27.381502 [ 5752]: Thawing priority 1
2012/10/05 15:36:27.381508 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:27.381524 [ 5752]: Thawing priority 2
2012/10/05 15:36:27.381530 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:27.381540 [ 5752]: Thawing priority 3
2012/10/05 15:36:27.381546 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:27.409421 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:27.409453 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:27.409461 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:27.426800 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:27.426821 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:27.426829 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:28.281488 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:28.282128 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:28.282142 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:28.282148 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:28.302596 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:28.302635 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:28.302706 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:28.302728 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:28.303384 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:28.303399 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:28.303729 [ 5752]: Freeze priority 1
2012/10/05 15:36:28.304756 [ 5752]: Freeze priority 2
2012/10/05 15:36:28.305493 [ 5752]: Freeze priority 3
2012/10/05 15:36:28.306232 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:28.306588 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:28.306603 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:28.306612 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:28.356816 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:28.356954 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:28.357341 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:28.357581 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:28.357867 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:28.357882 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:28.357905 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:28.358155 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:28.358656 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:28.358893 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:28.358961 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:28.359251 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:28.359497 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:28.359523 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:28.359533 [ 5752]: Thawing priority 1
2012/10/05 15:36:28.359539 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:28.359552 [ 5752]: Thawing priority 2
2012/10/05 15:36:28.359571 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:28.359585 [ 5752]: Thawing priority 3
2012/10/05 15:36:28.359592 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:28.383134 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:28.383170 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:28.383179 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:28.417386 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:28.417410 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:28.417418 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:29.283072 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:29.283777 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:29.284032 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:29.284043 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:29.311181 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:29.311219 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:29.311289 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:29.311299 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:29.311936 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:29.311962 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:29.312254 [ 5752]: Freeze priority 1
2012/10/05 15:36:29.313021 [ 5752]: Freeze priority 2
2012/10/05 15:36:29.313687 [ 5752]: Freeze priority 3
2012/10/05 15:36:29.314314 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:29.314636 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:29.314649 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:29.314657 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:29.375771 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:29.375929 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:29.376443 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:29.376729 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:29.377019 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:29.377033 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:29.377055 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:29.377345 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:29.377902 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:29.378188 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:29.378254 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:29.378552 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:29.378829 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:29.378855 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:29.378865 [ 5752]: Thawing priority 1
2012/10/05 15:36:29.378871 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:29.378884 [ 5752]: Thawing priority 2
2012/10/05 15:36:29.378891 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:29.378901 [ 5752]: Thawing priority 3
2012/10/05 15:36:29.378908 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:29.425663 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:29.425703 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:29.425711 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:29.484526 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:29.484561 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:29.484570 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:30.285125 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:30.285838 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:30.285851 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:30.285858 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:30.305314 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:30.305330 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:30.305370 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:30.305379 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:30.305870 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:30.305883 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:30.306169 [ 5752]: Freeze priority 1
2012/10/05 15:36:30.306940 [ 5752]: Freeze priority 2
2012/10/05 15:36:30.307586 [ 5752]: Freeze priority 3
2012/10/05 15:36:30.308231 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:30.308531 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:30.308543 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:30.308551 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:30.357087 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:30.358182 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:30.358526 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:30.358856 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:30.359136 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:30.359151 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:30.359172 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:30.359443 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:30.359952 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:30.360216 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:30.360282 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:30.360590 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:30.360858 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:30.360885 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:30.360894 [ 5752]: Thawing priority 1
2012/10/05 15:36:30.360900 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:30.360914 [ 5752]: Thawing priority 2
2012/10/05 15:36:30.360921 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:30.360930 [ 5752]: Thawing priority 3
2012/10/05 15:36:30.360937 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:30.374722 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:30.374753 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:30.374762 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:30.403969 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:30.403994 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:30.404002 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:31.286676 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:31.287363 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:31.287380 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:31.287387 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:31.322555 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:31.322595 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:31.322666 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:31.322675 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:31.323329 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:31.323345 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:31.323630 [ 5752]: Freeze priority 1
2012/10/05 15:36:31.324417 [ 5752]: Freeze priority 2
2012/10/05 15:36:31.325057 [ 5752]: Freeze priority 3
2012/10/05 15:36:31.325754 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:31.326043 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:31.326056 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:31.326064 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:31.375059 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:31.389015 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:31.389404 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:31.389689 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:31.389995 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:31.390009 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:31.390030 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:31.390325 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:31.390980 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:31.391281 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:31.391348 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:31.391689 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:31.391996 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:31.392024 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:31.392033 [ 5752]: Thawing priority 1
2012/10/05 15:36:31.392040 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:31.392057 [ 5752]: Thawing priority 2
2012/10/05 15:36:31.392064 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:31.392075 [ 5752]: Thawing priority 3
2012/10/05 15:36:31.392082 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:31.439285 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:31.439316 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:31.439324 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:31.439331 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:31.439337 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:31.439343 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:32.288783 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:32.289527 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:32.289543 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:32.289551 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:32.313786 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:32.313806 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:32.313883 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:32.313894 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:32.314445 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:32.314458 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:32.314748 [ 5752]: Freeze priority 1
2012/10/05 15:36:32.315609 [ 5752]: Freeze priority 2
2012/10/05 15:36:32.316342 [ 5752]: Freeze priority 3
2012/10/05 15:36:32.317066 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:32.317402 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:32.317427 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:32.317435 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:32.365868 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:32.378756 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:32.379127 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:32.379436 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:32.379749 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:32.379763 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:32.379784 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:32.380054 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:32.380573 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:32.380843 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:32.380909 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:32.381227 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:32.381509 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:32.381536 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:32.381545 [ 5752]: Thawing priority 1
2012/10/05 15:36:32.381551 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:32.381567 [ 5752]: Thawing priority 2
2012/10/05 15:36:32.381574 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:32.381585 [ 5752]: Thawing priority 3
2012/10/05 15:36:32.381592 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:32.394933 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:32.394960 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:32.394968 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:32.440099 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:32.440129 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:32.440137 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:33.290773 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:33.291438 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:33.291454 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:33.291461 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:33.317017 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:33.317050 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:33.317110 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:33.317121 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:33.317697 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:33.317713 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:33.318009 [ 5752]: Freeze priority 1
2012/10/05 15:36:33.318795 [ 5752]: Freeze priority 2
2012/10/05 15:36:33.319486 [ 5752]: Freeze priority 3
2012/10/05 15:36:33.320246 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:33.320547 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:33.320570 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:33.320579 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:33.371691 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:33.371829 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:33.372213 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:33.372500 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:33.372804 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:33.372819 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:33.372840 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:33.373131 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:33.373691 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:33.373977 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:33.374044 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:33.374385 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:33.374632 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:33.374658 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:33.374668 [ 5752]: Thawing priority 1
2012/10/05 15:36:33.374674 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:33.374688 [ 5752]: Thawing priority 2
2012/10/05 15:36:33.374695 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:33.374706 [ 5752]: Thawing priority 3
2012/10/05 15:36:33.374712 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:33.419664 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:33.419700 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:33.419709 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:33.419716 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:33.419722 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:33.419729 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:34.292418 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:34.293239 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:34.294042 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:34.294055 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:34.328536 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:34.328561 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:34.328617 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:34.328627 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:34.329237 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:34.329251 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:34.329548 [ 5752]: Freeze priority 1
2012/10/05 15:36:34.330370 [ 5752]: Freeze priority 2
2012/10/05 15:36:34.331062 [ 5752]: Freeze priority 3
2012/10/05 15:36:34.331739 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:34.332075 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:34.332089 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:34.332097 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:34.382043 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:34.382169 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:34.382532 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:34.382820 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:34.383135 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:34.383150 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:34.383171 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:34.383462 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:34.384122 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:34.384412 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:34.384494 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:34.384828 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:34.385146 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:34.385178 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:34.385189 [ 5752]: Thawing priority 1
2012/10/05 15:36:34.385195 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:34.385209 [ 5752]: Thawing priority 2
2012/10/05 15:36:34.385216 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:34.385227 [ 5752]: Thawing priority 3
2012/10/05 15:36:34.385233 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:34.412918 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:34.412960 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:34.412971 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:34.428725 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:34.428747 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:34.428755 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:35.294477 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:35.295214 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:35.295231 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:35.295237 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:35.314674 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:35.314697 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:35.314749 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:35.314759 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:35.315325 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:35.315340 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:35.315656 [ 5752]: Freeze priority 1
2012/10/05 15:36:35.316462 [ 5752]: Freeze priority 2
2012/10/05 15:36:35.317162 [ 5752]: Freeze priority 3
2012/10/05 15:36:35.317850 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:35.318197 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:35.318211 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:35.318219 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:35.377934 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:35.378104 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:35.378563 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:35.378850 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:35.379160 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:35.379173 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:35.379192 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:35.379464 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:35.379982 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:35.380264 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:35.380325 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:35.380643 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:35.380917 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:35.380942 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:35.380950 [ 5752]: Thawing priority 1
2012/10/05 15:36:35.380956 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:35.380968 [ 5752]: Thawing priority 2
2012/10/05 15:36:35.380974 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:35.380983 [ 5752]: Thawing priority 3
2012/10/05 15:36:35.381001 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:35.411594 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:35.411627 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:35.411636 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:35.437751 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:35.437775 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:35.437783 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:36.296507 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:36.297218 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:36.297234 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:36.297240 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:36.322841 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:36.322870 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:36.322927 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:36.322936 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:36.323526 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:36.323539 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:36.323841 [ 5752]: Freeze priority 1
2012/10/05 15:36:36.324656 [ 5752]: Freeze priority 2
2012/10/05 15:36:36.325304 [ 5752]: Freeze priority 3
2012/10/05 15:36:36.325957 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:36.326258 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:36.326271 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:36.326279 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:36.374360 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:36.400490 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:36.400873 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:36.401132 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:36.401404 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:36.401430 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:36.401450 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:36.401760 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:36.402305 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:36.402588 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:36.402649 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:36.402965 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:36.403257 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:36.403286 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:36.403296 [ 5752]: Thawing priority 1
2012/10/05 15:36:36.403302 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:36.403319 [ 5752]: Thawing priority 2
2012/10/05 15:36:36.403325 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:36.403335 [ 5752]: Thawing priority 3
2012/10/05 15:36:36.403342 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:36.500659 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:36.500695 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:36.500704 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:36.500711 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:36.500717 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:36.500723 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:37.298110 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:37.298829 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:37.298843 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:37.298850 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:37.319999 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:37.320029 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:37.320087 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:37.320097 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:37.320702 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:37.320716 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:37.321004 [ 5752]: Freeze priority 1
2012/10/05 15:36:37.321891 [ 5752]: Freeze priority 2
2012/10/05 15:36:37.322566 [ 5752]: Freeze priority 3
2012/10/05 15:36:37.323267 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:37.323573 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:37.323586 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:37.323594 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:37.380179 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:37.380337 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:37.380752 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:37.381020 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:37.381306 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:37.381321 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:37.381342 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:37.381636 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:37.382159 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:37.382426 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:37.382492 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:37.382816 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:37.383090 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:37.383116 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:37.383139 [ 5752]: Thawing priority 1
2012/10/05 15:36:37.383146 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:37.383159 [ 5752]: Thawing priority 2
2012/10/05 15:36:37.383166 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:37.383176 [ 5752]: Thawing priority 3
2012/10/05 15:36:37.383183 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:37.396555 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:37.396581 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:37.396590 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:37.427609 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:37.427638 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:37.427647 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:38.300226 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:38.300918 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:38.300932 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:38.300939 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:38.336385 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:38.336410 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:38.336459 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:38.336469 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:38.336995 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:38.337009 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:38.337324 [ 5752]: Freeze priority 1
2012/10/05 15:36:38.338235 [ 5752]: Freeze priority 2
2012/10/05 15:36:38.338934 [ 5752]: Freeze priority 3
2012/10/05 15:36:38.339644 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:38.339967 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:38.339980 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:38.339998 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:38.388450 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:38.389430 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:38.389797 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:38.390032 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:38.390338 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:38.390357 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:38.390381 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:38.390628 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:38.391098 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:38.391353 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:38.391419 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:38.391699 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:38.391947 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:38.391974 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:38.391983 [ 5752]: Thawing priority 1
2012/10/05 15:36:38.391989 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:38.392004 [ 5752]: Thawing priority 2
2012/10/05 15:36:38.392011 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:38.392020 [ 5752]: Thawing priority 3
2012/10/05 15:36:38.392027 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:38.432962 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:38.432994 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:38.433003 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:38.433010 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:38.433017 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:38.433023 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:39.301718 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:39.302420 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:39.302756 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:39.302769 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:39.336595 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:39.336629 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:39.336694 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:39.336704 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:39.337295 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:39.337310 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:39.337593 [ 5752]: Freeze priority 1
2012/10/05 15:36:39.338389 [ 5752]: Freeze priority 2
2012/10/05 15:36:39.339055 [ 5752]: Freeze priority 3
2012/10/05 15:36:39.339737 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:39.340070 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:39.340083 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:39.340091 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:39.388376 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:39.406884 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:39.407267 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:39.407576 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:39.407889 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:39.407903 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:39.407924 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:39.408193 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:39.408700 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:39.408956 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:39.409021 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:39.409377 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:39.409651 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:39.409678 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:39.409687 [ 5752]: Thawing priority 1
2012/10/05 15:36:39.409693 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:39.409711 [ 5752]: Thawing priority 2
2012/10/05 15:36:39.409717 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:39.409728 [ 5752]: Thawing priority 3
2012/10/05 15:36:39.409734 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:39.440578 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:39.440614 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:39.440623 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:39.461715 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:39.461740 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:39.461748 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:40.303273 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:40.304037 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:40.304051 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:40.304057 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:40.323747 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:40.323767 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:40.323814 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:40.323823 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:40.324399 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:40.324411 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:40.324697 [ 5752]: Freeze priority 1
2012/10/05 15:36:40.325527 [ 5752]: Freeze priority 2
2012/10/05 15:36:40.326214 [ 5752]: Freeze priority 3
2012/10/05 15:36:40.326923 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:40.327253 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:40.327266 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:40.327274 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:40.391211 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:40.391378 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:40.391901 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:40.392204 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:40.392524 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:40.392539 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:40.392560 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:40.392855 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:40.393426 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:40.393709 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:40.393776 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:40.394110 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:40.394422 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:40.394460 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:40.394470 [ 5752]: Thawing priority 1
2012/10/05 15:36:40.394476 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:40.394490 [ 5752]: Thawing priority 2
2012/10/05 15:36:40.394497 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:40.394507 [ 5752]: Thawing priority 3
2012/10/05 15:36:40.394514 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:40.413900 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:40.413924 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:40.413933 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:40.459183 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:40.459205 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:40.459214 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:41.304806 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:41.305518 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:41.305536 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:41.305543 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:41.325356 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:41.325379 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:41.325442 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:41.325454 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:41.325986 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:41.326000 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:41.326307 [ 5752]: Freeze priority 1
2012/10/05 15:36:41.327123 [ 5752]: Freeze priority 2
2012/10/05 15:36:41.327849 [ 5752]: Freeze priority 3
2012/10/05 15:36:41.328634 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:41.328958 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:41.328972 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:41.328980 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:41.377773 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:41.379162 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:41.379516 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:41.379829 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:41.380149 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:41.380163 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:41.380185 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:41.380470 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:41.380986 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:41.381247 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:41.381310 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:41.381635 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:41.381901 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:41.381940 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:41.381958 [ 5752]: Thawing priority 1
2012/10/05 15:36:41.381964 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:41.381978 [ 5752]: Thawing priority 2
2012/10/05 15:36:41.381984 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:41.381993 [ 5752]: Thawing priority 3
2012/10/05 15:36:41.381999 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:41.422361 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:41.422404 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:41.422415 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:41.428751 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:41.428763 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:41.428771 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:42.306441 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:42.307111 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:42.307125 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:42.307132 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:42.348422 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:42.348454 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:42.348527 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:42.348538 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:42.349108 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:42.349122 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:42.349429 [ 5752]: Freeze priority 1
2012/10/05 15:36:42.350259 [ 5752]: Freeze priority 2
2012/10/05 15:36:42.350917 [ 5752]: Freeze priority 3
2012/10/05 15:36:42.351594 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:42.351906 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:42.351919 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:42.351928 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:42.399777 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:42.421419 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:42.421825 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:42.422103 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:42.422387 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:42.422401 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:42.422422 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:42.422689 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:42.423215 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:42.423483 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:42.423551 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:42.423891 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:42.424175 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:42.424202 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:42.424211 [ 5752]: Thawing priority 1
2012/10/05 15:36:42.424217 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:42.424234 [ 5752]: Thawing priority 2
2012/10/05 15:36:42.424254 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:42.424266 [ 5752]: Thawing priority 3
2012/10/05 15:36:42.424272 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:42.468188 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:42.468218 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:42.468227 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:42.468234 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:42.468240 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:42.468247 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:43.307596 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:43.308285 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:43.308302 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:43.308309 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:43.335341 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:43.335363 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:43.335408 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:43.335417 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:43.335933 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:43.335946 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:43.336247 [ 5752]: Freeze priority 1
2012/10/05 15:36:43.337065 [ 5752]: Freeze priority 2
2012/10/05 15:36:43.337819 [ 5752]: Freeze priority 3
2012/10/05 15:36:43.338559 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:43.338869 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:43.338883 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:43.338891 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:43.387183 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:43.387802 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:43.388178 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:43.388458 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:43.388777 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:43.388793 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:43.388816 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:43.389096 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:43.389636 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:43.389907 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:43.389974 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:43.390300 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:43.390575 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:43.390602 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:43.390611 [ 5752]: Thawing priority 1
2012/10/05 15:36:43.390617 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:43.390631 [ 5752]: Thawing priority 2
2012/10/05 15:36:43.390638 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:43.390648 [ 5752]: Thawing priority 3
2012/10/05 15:36:43.390655 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:43.404339 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:43.404376 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:43.404385 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:43.452987 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:43.453021 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:43.453030 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:44.309548 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:44.310374 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:44.310978 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:44.310991 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:44.330081 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:44.330117 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:44.330171 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:44.330180 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:44.330718 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:44.330731 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:44.331020 [ 5752]: Freeze priority 1
2012/10/05 15:36:44.331811 [ 5752]: Freeze priority 2
2012/10/05 15:36:44.332460 [ 5752]: Freeze priority 3
2012/10/05 15:36:44.333158 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:44.333476 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:44.333489 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:44.333497 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:44.398953 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:44.399083 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:44.399481 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:44.399806 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:44.400118 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:44.400134 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:44.400157 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:44.400448 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:44.401027 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:44.401294 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:44.401359 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:44.401682 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:44.401957 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:44.401982 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:44.401991 [ 5752]: Thawing priority 1
2012/10/05 15:36:44.401996 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:44.402009 [ 5752]: Thawing priority 2
2012/10/05 15:36:44.402015 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:44.402024 [ 5752]: Thawing priority 3
2012/10/05 15:36:44.402030 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:44.440830 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:44.440862 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:44.440883 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:44.448978 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:44.448991 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:44.448998 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:45.311647 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:45.312322 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:45.312338 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:45.312346 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:45.338329 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:45.338349 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:45.338394 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:45.338405 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:45.338937 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:45.338952 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:45.339246 [ 5752]: Freeze priority 1
2012/10/05 15:36:45.340045 [ 5752]: Freeze priority 2
2012/10/05 15:36:45.340719 [ 5752]: Freeze priority 3
2012/10/05 15:36:45.341386 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:45.341720 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:45.341734 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:45.341742 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:45.390319 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:45.390450 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:45.390800 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:45.391070 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:45.391347 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:45.391364 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:45.391385 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:45.391677 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:45.392222 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:45.392496 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:45.392558 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:45.392911 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:45.393209 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:45.393237 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:45.393246 [ 5752]: Thawing priority 1
2012/10/05 15:36:45.393252 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:45.393266 [ 5752]: Thawing priority 2
2012/10/05 15:36:45.393273 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:45.393282 [ 5752]: Thawing priority 3
2012/10/05 15:36:45.393289 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:45.406595 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:45.406623 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:45.406632 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:45.431270 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:45.431295 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:45.431305 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:46.312690 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:46.313359 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:46.313373 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:46.313379 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:46.348299 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:46.348342 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:46.348398 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:46.348408 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:46.348987 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:46.349002 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:46.349320 [ 5752]: Freeze priority 1
2012/10/05 15:36:46.350112 [ 5752]: Freeze priority 2
2012/10/05 15:36:46.350792 [ 5752]: Freeze priority 3
2012/10/05 15:36:46.351453 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:46.351777 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:46.351789 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:46.351797 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:46.398125 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:46.415650 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:46.416015 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:46.416279 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:46.416543 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:46.416556 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:46.416575 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:46.416831 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:46.417324 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:46.417580 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:46.417645 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:46.417968 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:46.418291 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:46.418325 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:46.418334 [ 5752]: Thawing priority 1
2012/10/05 15:36:46.418340 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:46.418357 [ 5752]: Thawing priority 2
2012/10/05 15:36:46.418363 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:46.418372 [ 5752]: Thawing priority 3
2012/10/05 15:36:46.418378 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:46.464271 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:46.464304 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:46.464313 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:46.464319 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:46.464325 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:46.464341 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:47.314720 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:47.315432 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:47.315446 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:47.315453 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:47.335100 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:47.335117 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:47.335158 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:47.335167 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:47.335697 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:47.335709 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:47.335990 [ 5752]: Freeze priority 1
2012/10/05 15:36:47.336764 [ 5752]: Freeze priority 2
2012/10/05 15:36:47.337401 [ 5752]: Freeze priority 3
2012/10/05 15:36:47.338032 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:47.338327 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:47.338339 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:47.338355 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:47.385114 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:47.387041 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:47.387399 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:47.387703 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:47.388015 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:47.388032 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:47.388053 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:47.388362 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:47.388913 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:47.389230 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:47.389322 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:47.389653 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:47.389971 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:47.389998 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:47.390008 [ 5752]: Thawing priority 1
2012/10/05 15:36:47.390014 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:47.390028 [ 5752]: Thawing priority 2
2012/10/05 15:36:47.390035 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:47.390045 [ 5752]: Thawing priority 3
2012/10/05 15:36:47.390052 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:47.403733 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:47.403770 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:47.403793 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:47.447169 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:47.447201 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:47.447210 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:48.316828 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:48.317532 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:48.317547 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:48.317554 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:48.343137 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:48.343162 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:48.343211 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:48.343220 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:48.343768 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:48.343782 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:48.344056 [ 5752]: Freeze priority 1
2012/10/05 15:36:48.344891 [ 5752]: Freeze priority 2
2012/10/05 15:36:48.345664 [ 5752]: Freeze priority 3
2012/10/05 15:36:48.346443 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:48.346749 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:48.346774 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:48.346784 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:48.406456 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:48.406612 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:48.407068 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:48.407327 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:48.407607 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:48.407620 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:48.407640 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:48.407899 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:48.408388 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:48.408633 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:48.408694 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:48.408988 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:48.409246 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:48.409271 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:48.409280 [ 5752]: Thawing priority 1
2012/10/05 15:36:48.409286 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:48.409299 [ 5752]: Thawing priority 2
2012/10/05 15:36:48.409305 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:48.409314 [ 5752]: Thawing priority 3
2012/10/05 15:36:48.409320 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:48.452295 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:48.452331 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:48.452340 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:48.461908 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:48.461926 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:48.461934 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:49.318367 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:49.319085 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:49.319441 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:49.319453 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:49.338647 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:49.338665 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:49.338710 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:49.338720 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:49.339268 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:49.339283 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:49.339617 [ 5752]: Freeze priority 1
2012/10/05 15:36:49.340464 [ 5752]: Freeze priority 2
2012/10/05 15:36:49.341165 [ 5752]: Freeze priority 3
2012/10/05 15:36:49.341894 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:49.342343 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:49.342359 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:49.342379 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:49.390637 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:49.400907 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:49.401295 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:49.401572 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:49.401823 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:49.401838 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:49.401859 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:49.402112 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:49.402590 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:49.402818 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:49.402884 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:49.403172 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:49.403420 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:49.403460 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:49.403470 [ 5752]: Thawing priority 1
2012/10/05 15:36:49.403476 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:49.403493 [ 5752]: Thawing priority 2
2012/10/05 15:36:49.403499 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:49.403509 [ 5752]: Thawing priority 3
2012/10/05 15:36:49.403516 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:49.417370 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:49.417397 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:49.417406 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:49.478731 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:49.478761 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:49.478769 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:50.320425 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:50.321113 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:50.321127 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:50.321133 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:50.340815 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:50.340845 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:50.340905 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:50.340914 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:50.341496 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:50.341510 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:50.341780 [ 5752]: Freeze priority 1
2012/10/05 15:36:50.342557 [ 5752]: Freeze priority 2
2012/10/05 15:36:50.343198 [ 5752]: Freeze priority 3
2012/10/05 15:36:50.343825 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:50.344135 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:50.344148 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:50.344156 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:50.392756 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:50.412580 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:50.413003 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:50.413298 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:50.413601 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:50.413616 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:50.413637 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:50.413918 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:50.414463 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:50.414754 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:50.414821 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:50.415162 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:50.415441 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:50.415467 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:50.415489 [ 5752]: Thawing priority 1
2012/10/05 15:36:50.415497 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:50.415516 [ 5752]: Thawing priority 2
2012/10/05 15:36:50.415523 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:50.415534 [ 5752]: Thawing priority 3
2012/10/05 15:36:50.415540 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:50.460445 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:50.460482 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:50.460490 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:50.478868 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:50.478890 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:50.478898 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:51.322451 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:51.323232 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:51.323247 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:51.323255 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:51.344722 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:51.344778 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:51.344926 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:51.344941 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:51.345808 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:51.345831 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:51.346135 [ 5752]: Freeze priority 1
2012/10/05 15:36:51.346935 [ 5752]: Freeze priority 2
2012/10/05 15:36:51.347606 [ 5752]: Freeze priority 3
2012/10/05 15:36:51.348356 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:51.348754 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:51.348770 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:51.348779 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:51.409948 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:51.410121 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:51.410540 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:51.410844 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:51.411169 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:51.411186 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:51.411207 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:51.411465 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:51.412025 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:51.412313 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:51.412381 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:51.412709 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:51.412969 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:51.412994 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:51.413003 [ 5752]: Thawing priority 1
2012/10/05 15:36:51.413009 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:51.413022 [ 5752]: Thawing priority 2
2012/10/05 15:36:51.413028 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:51.413037 [ 5752]: Thawing priority 3
2012/10/05 15:36:51.413043 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:51.427225 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:51.427262 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:51.427271 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:51.451023 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:51.451072 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:51.451083 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:52.324616 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:52.325320 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:52.325335 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:52.325342 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:52.365675 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:52.365714 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:52.365785 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:52.365810 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:52.366468 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:52.366482 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:52.366784 [ 5752]: Freeze priority 1
2012/10/05 15:36:52.367583 [ 5752]: Freeze priority 2
2012/10/05 15:36:52.368345 [ 5752]: Freeze priority 3
2012/10/05 15:36:52.369154 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:52.369464 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:52.369478 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:52.369486 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:52.421020 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:52.430420 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:52.430813 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:52.431097 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:52.431392 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:52.431407 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:52.431427 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:52.431715 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:52.432262 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:52.432541 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:52.432607 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:52.432951 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:52.433276 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:52.433307 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:52.433317 [ 5752]: Thawing priority 1
2012/10/05 15:36:52.433323 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:52.433340 [ 5752]: Thawing priority 2
2012/10/05 15:36:52.433346 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:52.433355 [ 5752]: Thawing priority 3
2012/10/05 15:36:52.433362 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:52.535052 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:52.535092 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:52.535101 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:52.535109 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:52.535115 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:52.535122 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:53.326711 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:53.327407 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:53.327421 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:53.327428 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:53.347226 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:53.347256 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:53.347315 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:53.347325 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:53.347896 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:53.347921 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:53.348214 [ 5752]: Freeze priority 1
2012/10/05 15:36:53.349002 [ 5752]: Freeze priority 2
2012/10/05 15:36:53.349686 [ 5752]: Freeze priority 3
2012/10/05 15:36:53.350368 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:53.350654 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:53.350677 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:53.350688 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:53.400834 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:53.401257 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:53.401660 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:53.401984 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:53.402278 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:53.402291 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:53.402311 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:53.402573 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:53.403097 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:53.403485 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:53.403592 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:53.403944 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:53.404212 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:53.404237 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:53.404246 [ 5752]: Thawing priority 1
2012/10/05 15:36:53.404252 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:53.404266 [ 5752]: Thawing priority 2
2012/10/05 15:36:53.404272 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:53.404282 [ 5752]: Thawing priority 3
2012/10/05 15:36:53.404288 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:53.417528 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:53.417557 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:53.417566 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:53.462836 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:53.462867 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:53.462875 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:54.328136 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:54.328834 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:54.329209 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:54.329222 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:54.350085 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:54.350116 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:54.350177 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:54.350186 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:54.350800 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:54.350813 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:54.351104 [ 5752]: Freeze priority 1
2012/10/05 15:36:54.351878 [ 5752]: Freeze priority 2
2012/10/05 15:36:54.352506 [ 5752]: Freeze priority 3
2012/10/05 15:36:54.353145 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:54.353451 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:54.353464 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:54.353472 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:54.402353 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:54.413944 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:54.414304 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:54.414568 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:54.414810 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:54.414824 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:54.414845 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:54.415089 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:54.415562 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:54.415796 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:54.415861 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:54.416152 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:54.416410 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:54.416436 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:54.416445 [ 5752]: Thawing priority 1
2012/10/05 15:36:54.416451 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:54.416469 [ 5752]: Thawing priority 2
2012/10/05 15:36:54.416476 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:54.416487 [ 5752]: Thawing priority 3
2012/10/05 15:36:54.416493 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:54.458307 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:54.458341 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:54.458350 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:54.471538 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:54.471557 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:54.471565 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:55.330190 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:55.330982 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:55.330997 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:55.331004 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:55.366547 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:55.366737 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:55.367160 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:55.367221 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:55.369631 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:55.369683 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:55.370074 [ 5752]: Freeze priority 1
2012/10/05 15:36:55.371262 [ 5752]: Freeze priority 2
2012/10/05 15:36:55.372388 [ 5752]: Freeze priority 3
2012/10/05 15:36:55.373430 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:55.373761 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:55.373775 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:55.373783 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:55.433090 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:55.433258 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:55.433772 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:55.434087 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:55.434339 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:55.434352 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:55.434371 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:55.434594 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:55.435034 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:55.435260 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:55.435345 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:55.435617 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:55.435836 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:55.435861 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:55.435870 [ 5752]: Thawing priority 1
2012/10/05 15:36:55.435875 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:55.435888 [ 5752]: Thawing priority 2
2012/10/05 15:36:55.435895 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:55.435904 [ 5752]: Thawing priority 3
2012/10/05 15:36:55.435910 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:55.466144 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:55.466184 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:55.466193 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:55.489267 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:55.489302 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:55.489309 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:56.331823 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:56.332549 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:56.332563 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:56.332570 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:56.351658 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:56.351676 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:56.351720 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:56.351730 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:56.352267 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:56.352281 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:56.352593 [ 5752]: Freeze priority 1
2012/10/05 15:36:56.353410 [ 5752]: Freeze priority 2
2012/10/05 15:36:56.354118 [ 5752]: Freeze priority 3
2012/10/05 15:36:56.354820 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:56.355149 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:56.355174 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:56.355192 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:56.403703 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:56.405307 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:56.405700 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:56.405974 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:56.406260 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:56.406274 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:56.406294 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:56.406601 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:56.407150 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:56.407425 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:56.407491 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:56.407814 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:56.408110 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:56.408138 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:56.408148 [ 5752]: Thawing priority 1
2012/10/05 15:36:56.408154 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:56.408168 [ 5752]: Thawing priority 2
2012/10/05 15:36:56.408175 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:56.408185 [ 5752]: Thawing priority 3
2012/10/05 15:36:56.408192 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:56.427665 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:56.427692 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:56.427700 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:56.452941 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:56.452962 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:56.452970 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:57.333409 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:57.334133 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:57.334147 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:57.334153 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:57.412603 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:57.412641 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:57.412708 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:57.412718 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:57.413295 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:57.413310 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:57.413605 [ 5752]: Freeze priority 1
2012/10/05 15:36:57.414372 [ 5752]: Freeze priority 2
2012/10/05 15:36:57.415041 [ 5752]: Freeze priority 3
2012/10/05 15:36:57.415725 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:57.416023 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:57.416037 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:57.416045 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:57.463946 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:57.465491 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:57.465847 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:57.466099 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:57.466334 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:57.466347 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:57.466366 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:57.466596 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:57.467042 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:57.467261 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:57.467321 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:57.467590 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:57.467835 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:57.467859 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:57.467868 [ 5752]: Thawing priority 1
2012/10/05 15:36:57.467874 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:57.467925 [ 5752]: Thawing priority 2
2012/10/05 15:36:57.467933 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:57.467957 [ 5752]: Thawing priority 3
2012/10/05 15:36:57.467963 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:57.513676 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:57.513712 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:57.513721 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:57.513729 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:57.513735 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:57.513742 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:58.335481 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:58.336227 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:58.336240 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:58.336247 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:58.356024 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:58.356057 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:58.356120 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:58.356131 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:58.356729 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:58.356745 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:58.357039 [ 5752]: Freeze priority 1
2012/10/05 15:36:58.357933 [ 5752]: Freeze priority 2
2012/10/05 15:36:58.358667 [ 5752]: Freeze priority 3
2012/10/05 15:36:58.359408 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:58.359730 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:58.359745 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:58.359754 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:58.408714 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:58.410440 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:58.410808 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:58.411074 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:58.411358 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:58.411372 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:58.411392 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:58.411707 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:58.412241 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:58.412497 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:58.412578 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:58.412892 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:58.413163 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:58.413190 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:58.413200 [ 5752]: Thawing priority 1
2012/10/05 15:36:58.413206 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:58.413220 [ 5752]: Thawing priority 2
2012/10/05 15:36:58.413227 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:58.413237 [ 5752]: Thawing priority 3
2012/10/05 15:36:58.413244 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:58.427357 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:58.427381 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:58.427389 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:58.467683 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:58.467714 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:58.467722 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:36:59.337044 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:36:59.337776 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:36:59.337999 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:36:59.338011 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:36:59.363438 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:36:59.363462 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:36:59.363514 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:36:59.363524 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:36:59.364095 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:36:59.364109 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:36:59.364404 [ 5752]: Freeze priority 1
2012/10/05 15:36:59.365209 [ 5752]: Freeze priority 2
2012/10/05 15:36:59.365943 [ 5752]: Freeze priority 3
2012/10/05 15:36:59.366700 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:36:59.367031 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:36:59.367046 [ 5752]: Monitoring has been disabled
2012/10/05 15:36:59.367055 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:36:59.415643 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:36:59.427736 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:59.428141 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:59.428426 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:36:59.428720 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:36:59.428734 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:36:59.428755 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:36:59.429029 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:36:59.429580 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:36:59.429874 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:36:59.429940 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:36:59.430282 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:36:59.430565 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:36:59.430619 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:36:59.430630 [ 5752]: Thawing priority 1
2012/10/05 15:36:59.430636 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:36:59.430655 [ 5752]: Thawing priority 2
2012/10/05 15:36:59.430662 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:36:59.430673 [ 5752]: Thawing priority 3
2012/10/05 15:36:59.430679 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:36:59.472972 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:59.473002 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:36:59.473011 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:36:59.483858 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:36:59.483872 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:36:59.483880 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:00.338611 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:00.339244 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:00.339258 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:00.339265 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:00.358479 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:00.358496 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:00.358539 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:00.358549 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:00.359081 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:00.359094 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:00.359379 [ 5752]: Freeze priority 1
2012/10/05 15:37:00.360177 [ 5752]: Freeze priority 2
2012/10/05 15:37:00.360852 [ 5752]: Freeze priority 3
2012/10/05 15:37:00.361539 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:00.361870 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:00.361884 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:00.361902 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:00.410456 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:00.410988 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:00.411342 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:00.411621 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:00.411947 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:00.411962 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:00.411993 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:00.412293 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:00.412871 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:00.413243 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:00.413315 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:00.413626 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:00.413934 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:00.413965 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:00.413975 [ 5752]: Thawing priority 1
2012/10/05 15:37:00.413981 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:00.413994 [ 5752]: Thawing priority 2
2012/10/05 15:37:00.414000 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:00.414010 [ 5752]: Thawing priority 3
2012/10/05 15:37:00.414016 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:00.427283 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:00.427317 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:00.427325 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:00.458765 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:00.458797 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:00.458805 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:01.340198 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:01.340923 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:01.340937 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:01.340944 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:01.375676 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:01.375709 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:01.375791 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:01.375801 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:01.376419 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:01.376432 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:01.376729 [ 5752]: Freeze priority 1
2012/10/05 15:37:01.377513 [ 5752]: Freeze priority 2
2012/10/05 15:37:01.378145 [ 5752]: Freeze priority 3
2012/10/05 15:37:01.378815 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:01.379102 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:01.379114 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:01.379122 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:01.426706 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:01.427661 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:01.428055 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:01.428353 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:01.428655 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:01.428671 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:01.428693 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:01.428989 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:01.429576 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:01.429852 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:01.429916 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:01.430243 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:01.430512 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:01.430537 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:01.430545 [ 5752]: Thawing priority 1
2012/10/05 15:37:01.430551 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:01.430566 [ 5752]: Thawing priority 2
2012/10/05 15:37:01.430572 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:01.430582 [ 5752]: Thawing priority 3
2012/10/05 15:37:01.430588 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:01.487149 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:01.487184 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:01.487193 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:01.487200 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:01.487206 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:01.487212 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:02.341744 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:02.342451 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:02.342465 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:02.342472 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:02.361815 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:02.361841 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:02.361895 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:02.361905 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:02.362492 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:02.362508 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:02.362820 [ 5752]: Freeze priority 1
2012/10/05 15:37:02.363637 [ 5752]: Freeze priority 2
2012/10/05 15:37:02.364338 [ 5752]: Freeze priority 3
2012/10/05 15:37:02.365049 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:02.365363 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:02.365378 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:02.365386 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:02.412645 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:02.441042 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:02.441445 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:02.441735 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:02.442048 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:02.442063 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:02.442083 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:02.442377 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:02.442942 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:02.443243 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:02.443324 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:02.443658 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:02.443951 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:02.443978 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:02.443987 [ 5752]: Thawing priority 1
2012/10/05 15:37:02.443993 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:02.444011 [ 5752]: Thawing priority 2
2012/10/05 15:37:02.444018 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:02.444028 [ 5752]: Thawing priority 3
2012/10/05 15:37:02.444035 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:02.471617 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:02.471652 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:02.471660 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:02.551583 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:02.551618 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:02.551627 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:03.343258 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:03.343829 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:03.343853 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:03.343862 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:03.363526 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:03.363548 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:03.363599 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:03.363608 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:03.364126 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:03.364140 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:03.364432 [ 5752]: Freeze priority 1
2012/10/05 15:37:03.365330 [ 5752]: Freeze priority 2
2012/10/05 15:37:03.366003 [ 5752]: Freeze priority 3
2012/10/05 15:37:03.366647 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:03.366938 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:03.366952 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:03.366960 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:03.425249 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:03.425440 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:03.425842 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:03.426135 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:03.426446 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:03.426461 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:03.426482 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:03.426743 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:03.427272 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:03.427532 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:03.427598 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:03.427928 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:03.428190 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:03.428217 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:03.428226 [ 5752]: Thawing priority 1
2012/10/05 15:37:03.428232 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:03.428246 [ 5752]: Thawing priority 2
2012/10/05 15:37:03.428253 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:03.428263 [ 5752]: Thawing priority 3
2012/10/05 15:37:03.428270 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:03.452207 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:03.452244 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:03.452253 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:03.465087 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:03.465102 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:03.465110 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:04.344766 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:04.345435 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:04.345682 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:04.345696 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:04.394025 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:04.394064 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:04.394134 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:04.394143 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:04.394746 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:04.394759 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:04.395047 [ 5752]: Freeze priority 1
2012/10/05 15:37:04.395828 [ 5752]: Freeze priority 2
2012/10/05 15:37:04.396466 [ 5752]: Freeze priority 3
2012/10/05 15:37:04.397100 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:04.397419 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:04.397434 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:04.397442 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:04.447953 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:04.456268 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:04.456616 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:04.456871 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:04.457154 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:04.457167 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:04.457186 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:04.457469 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:04.458029 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:04.458279 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:04.458340 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:04.458706 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:04.459010 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:04.459059 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:04.459069 [ 5752]: Thawing priority 1
2012/10/05 15:37:04.459074 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:04.459101 [ 5752]: Thawing priority 2
2012/10/05 15:37:04.459107 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:04.459118 [ 5752]: Thawing priority 3
2012/10/05 15:37:04.459124 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:04.504354 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:04.504392 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:04.504400 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:04.504407 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:04.504413 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:04.504433 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:05.346939 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:05.347618 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:05.347634 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:05.347641 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:05.367262 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:05.367284 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:05.367334 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:05.367344 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:05.367924 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:05.367938 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:05.368229 [ 5752]: Freeze priority 1
2012/10/05 15:37:05.369038 [ 5752]: Freeze priority 2
2012/10/05 15:37:05.369702 [ 5752]: Freeze priority 3
2012/10/05 15:37:05.370390 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:05.370701 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:05.370714 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:05.370722 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:05.430199 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:05.430341 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:05.430767 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:05.431039 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:05.431310 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:05.431323 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:05.431342 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:05.431611 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:05.432121 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:05.432377 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:05.432439 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:05.432753 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:05.433037 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:05.433062 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:05.433071 [ 5752]: Thawing priority 1
2012/10/05 15:37:05.433084 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:05.433097 [ 5752]: Thawing priority 2
2012/10/05 15:37:05.433103 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:05.433113 [ 5752]: Thawing priority 3
2012/10/05 15:37:05.433119 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:05.446524 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:05.446546 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:05.446554 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:05.495183 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:05.495214 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:05.495223 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:06.348490 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:06.349154 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:06.349169 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:06.349175 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:06.368425 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:06.368442 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:06.368481 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:06.368490 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:06.369001 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:06.369014 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:06.369324 [ 5752]: Freeze priority 1
2012/10/05 15:37:06.370169 [ 5752]: Freeze priority 2
2012/10/05 15:37:06.370871 [ 5752]: Freeze priority 3
2012/10/05 15:37:06.371576 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:06.371872 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:06.371886 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:06.371904 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:06.420508 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:06.420660 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:06.421044 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:06.421363 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:06.421680 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:06.421695 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:06.421716 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:06.421990 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:06.422573 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:06.422846 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:06.422912 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:06.423251 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:06.423526 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:06.423564 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:06.423576 [ 5752]: Thawing priority 1
2012/10/05 15:37:06.423582 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:06.423596 [ 5752]: Thawing priority 2
2012/10/05 15:37:06.423614 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:06.423626 [ 5752]: Thawing priority 3
2012/10/05 15:37:06.423632 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:06.467346 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:06.467374 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:06.467383 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:06.467390 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:06.467397 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:06.467403 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:07.350202 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:07.351204 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:07.351272 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:07.351280 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:07.390891 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:07.390930 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:07.391001 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:07.391011 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:07.391651 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:07.391664 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:07.391951 [ 5752]: Freeze priority 1
2012/10/05 15:37:07.392759 [ 5752]: Freeze priority 2
2012/10/05 15:37:07.393424 [ 5752]: Freeze priority 3
2012/10/05 15:37:07.394075 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:07.394372 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:07.394388 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:07.394397 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:07.442267 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:07.443146 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:07.443507 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:07.443828 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:07.444140 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:07.444153 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:07.444172 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:07.444468 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:07.444973 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:07.445273 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:07.445340 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:07.445696 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:07.445978 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:07.446006 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:07.446015 [ 5752]: Thawing priority 1
2012/10/05 15:37:07.446022 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:07.446035 [ 5752]: Thawing priority 2
2012/10/05 15:37:07.446042 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:07.446052 [ 5752]: Thawing priority 3
2012/10/05 15:37:07.446059 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:07.491259 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:07.491300 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:07.491309 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:07.491317 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:07.491323 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:07.491330 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:08.351606 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:08.352181 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:08.352198 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:08.352205 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:08.371610 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:08.371645 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:08.371720 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:08.371731 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:08.372335 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:08.372348 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:08.372621 [ 5752]: Freeze priority 1
2012/10/05 15:37:08.373392 [ 5752]: Freeze priority 2
2012/10/05 15:37:08.374013 [ 5752]: Freeze priority 3
2012/10/05 15:37:08.374637 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:08.374940 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:08.374953 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:08.374961 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:08.423994 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:08.424121 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:08.424486 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:08.424763 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:08.425081 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:08.425097 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:08.425119 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:08.425443 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:08.426044 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:08.426361 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:08.426435 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:08.426767 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:08.427047 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:08.427072 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:08.427081 [ 5752]: Thawing priority 1
2012/10/05 15:37:08.427087 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:08.427100 [ 5752]: Thawing priority 2
2012/10/05 15:37:08.427107 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:08.427116 [ 5752]: Thawing priority 3
2012/10/05 15:37:08.427122 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:08.440634 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:08.440673 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:08.440694 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:08.488706 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:08.488741 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:08.488750 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:09.353201 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:09.354009 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:09.354743 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:09.354758 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:09.373941 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:09.373959 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:09.374000 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:09.374009 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:09.374553 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:09.374568 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:09.374874 [ 5752]: Freeze priority 1
2012/10/05 15:37:09.375691 [ 5752]: Freeze priority 2
2012/10/05 15:37:09.376383 [ 5752]: Freeze priority 3
2012/10/05 15:37:09.377046 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:09.377349 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:09.377363 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:09.377378 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:09.424992 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:09.433875 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:09.434225 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:09.434503 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:09.434773 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:09.434788 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:09.434809 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:09.435077 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:09.435549 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:09.435777 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:09.435843 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:09.436155 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:09.436466 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:09.436493 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:09.436503 [ 5752]: Thawing priority 1
2012/10/05 15:37:09.436509 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:09.436525 [ 5752]: Thawing priority 2
2012/10/05 15:37:09.436532 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:09.436543 [ 5752]: Thawing priority 3
2012/10/05 15:37:09.436550 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:09.478735 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:09.478769 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:09.478778 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:09.485882 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:09.485897 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:09.485905 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:10.354180 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:10.355056 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:10.355071 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:10.355078 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:10.379929 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:10.379969 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:10.380040 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:10.380050 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:10.380701 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:10.380714 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:10.381033 [ 5752]: Freeze priority 1
2012/10/05 15:37:10.381914 [ 5752]: Freeze priority 2
2012/10/05 15:37:10.382597 [ 5752]: Freeze priority 3
2012/10/05 15:37:10.383296 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:10.383602 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:10.383616 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:10.383624 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:10.432229 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:10.462032 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:10.462385 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:10.462632 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:10.462885 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:10.462899 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:10.462920 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:10.463166 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:10.463613 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:10.463845 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:10.463910 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:10.464204 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:10.464460 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:10.464486 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:10.464496 [ 5752]: Thawing priority 1
2012/10/05 15:37:10.464502 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:10.464519 [ 5752]: Thawing priority 2
2012/10/05 15:37:10.464526 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:10.464537 [ 5752]: Thawing priority 3
2012/10/05 15:37:10.464543 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:10.508351 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:10.508388 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:10.508397 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:10.518695 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:10.518711 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:10.518727 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:11.356300 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:11.356947 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:11.356960 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:11.356966 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:11.376488 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:11.376519 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:11.376581 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:11.376592 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:11.377183 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:11.377199 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:11.377471 [ 5752]: Freeze priority 1
2012/10/05 15:37:11.378234 [ 5752]: Freeze priority 2
2012/10/05 15:37:11.378884 [ 5752]: Freeze priority 3
2012/10/05 15:37:11.379525 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:11.379834 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:11.379847 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:11.379855 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:11.438699 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:11.438860 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:11.439339 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:11.439617 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:11.439895 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:11.439908 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:11.439927 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:11.440188 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:11.440690 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:11.440952 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:11.441014 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:11.441344 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:11.441627 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:11.441651 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:11.441660 [ 5752]: Thawing priority 1
2012/10/05 15:37:11.441666 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:11.441679 [ 5752]: Thawing priority 2
2012/10/05 15:37:11.441685 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:11.441694 [ 5752]: Thawing priority 3
2012/10/05 15:37:11.441700 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:11.455870 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:11.455917 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:11.455926 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:11.485111 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:11.485144 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:11.485152 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:12.357866 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:12.358572 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:12.358586 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:12.358593 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:12.393194 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:12.393215 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:12.393262 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:12.393273 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:12.393846 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:12.393860 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:12.394198 [ 5752]: Freeze priority 1
2012/10/05 15:37:12.395032 [ 5752]: Freeze priority 2
2012/10/05 15:37:12.395717 [ 5752]: Freeze priority 3
2012/10/05 15:37:12.396399 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:12.396715 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:12.396728 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:12.396737 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:12.452979 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:12.457095 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:12.457465 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:12.457732 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:12.458022 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:12.458036 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:12.458057 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:12.458325 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:12.458854 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:12.459123 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:12.459189 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:12.459515 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:12.459802 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:12.459829 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:12.459838 [ 5752]: Thawing priority 1
2012/10/05 15:37:12.459844 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:12.459860 [ 5752]: Thawing priority 2
2012/10/05 15:37:12.459867 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:12.459877 [ 5752]: Thawing priority 3
2012/10/05 15:37:12.459884 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:12.503345 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:12.503383 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:12.503392 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:12.504019 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:12.504031 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:12.504038 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:13.359432 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:13.359992 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:13.360031 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:13.360039 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:13.378975 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:13.378991 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:13.379045 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:13.379056 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:13.379551 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:13.379565 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:13.379868 [ 5752]: Freeze priority 1
2012/10/05 15:37:13.380695 [ 5752]: Freeze priority 2
2012/10/05 15:37:13.381376 [ 5752]: Freeze priority 3
2012/10/05 15:37:13.382064 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:13.382344 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:13.382358 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:13.382366 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:13.430650 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:13.430790 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:13.431145 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:13.431382 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:13.431645 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:13.431659 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:13.431680 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:13.431911 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:13.432365 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:13.432599 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:13.432665 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:13.432941 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:13.433217 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:13.433245 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:13.433254 [ 5752]: Thawing priority 1
2012/10/05 15:37:13.433260 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:13.433274 [ 5752]: Thawing priority 2
2012/10/05 15:37:13.433281 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:13.433290 [ 5752]: Thawing priority 3
2012/10/05 15:37:13.433297 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:13.447628 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:13.447651 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:13.447660 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:13.491403 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:13.491436 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:13.491444 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:14.361013 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:14.361715 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:14.362089 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:14.362100 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:14.386859 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:14.386878 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:14.386926 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:14.386935 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:14.387503 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:14.387524 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:14.387842 [ 5752]: Freeze priority 1
2012/10/05 15:37:14.388611 [ 5752]: Freeze priority 2
2012/10/05 15:37:14.389279 [ 5752]: Freeze priority 3
2012/10/05 15:37:14.389940 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:14.390251 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:14.390264 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:14.390273 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:14.438114 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:14.438241 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:14.438584 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:14.438838 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:14.439088 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:14.439103 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:14.439125 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:14.439364 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:14.439832 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:14.440048 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:14.440122 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:14.440396 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:14.440639 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:14.440664 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:14.440672 [ 5752]: Thawing priority 1
2012/10/05 15:37:14.440678 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:14.440690 [ 5752]: Thawing priority 2
2012/10/05 15:37:14.440696 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:14.440706 [ 5752]: Thawing priority 3
2012/10/05 15:37:14.440712 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:14.530667 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:14.530704 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:14.530714 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:14.530720 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:14.530726 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:14.530733 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:15.362047 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:15.362715 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:15.362729 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:15.362735 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:15.382435 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:15.382452 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:15.382506 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:15.382515 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:15.383040 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:15.383053 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:15.383349 [ 5752]: Freeze priority 1
2012/10/05 15:37:15.384102 [ 5752]: Freeze priority 2
2012/10/05 15:37:15.384744 [ 5752]: Freeze priority 3
2012/10/05 15:37:15.385386 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:15.385670 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:15.385682 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:15.385690 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:15.431313 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:15.444952 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:15.445316 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:15.445576 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:15.445864 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:15.445877 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:15.445897 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:15.446171 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:15.446711 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:15.446966 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:15.447027 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:15.447350 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:15.447621 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:15.447648 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:15.447657 [ 5752]: Thawing priority 1
2012/10/05 15:37:15.447662 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:15.447678 [ 5752]: Thawing priority 2
2012/10/05 15:37:15.447684 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:15.447694 [ 5752]: Thawing priority 3
2012/10/05 15:37:15.447700 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:15.460855 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:15.460876 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:15.460883 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:15.510942 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:15.510978 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:15.510986 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:16.363641 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:16.364308 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:16.364323 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:16.364330 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:16.383710 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:16.383726 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:16.383767 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:16.383789 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:16.384303 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:16.384316 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:16.384581 [ 5752]: Freeze priority 1
2012/10/05 15:37:16.385334 [ 5752]: Freeze priority 2
2012/10/05 15:37:16.385977 [ 5752]: Freeze priority 3
2012/10/05 15:37:16.386606 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:16.386915 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:16.386928 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:16.386936 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:16.450341 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:16.450489 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:16.450925 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:16.451191 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:16.451482 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:16.451497 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:16.451517 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:16.451797 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:16.452387 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:16.452651 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:16.452732 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:16.453053 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:16.453365 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:16.453392 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:16.453402 [ 5752]: Thawing priority 1
2012/10/05 15:37:16.453408 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:16.453422 [ 5752]: Thawing priority 2
2012/10/05 15:37:16.453429 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:16.453439 [ 5752]: Thawing priority 3
2012/10/05 15:37:16.453446 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:16.497872 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:16.497907 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:16.497916 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:16.497922 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:16.497928 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:16.497934 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:17.365246 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:17.365983 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:17.365996 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:17.366003 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:17.406839 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:17.406865 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:17.406918 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:17.406928 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:17.407474 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:17.407499 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:17.407820 [ 5752]: Freeze priority 1
2012/10/05 15:37:17.408612 [ 5752]: Freeze priority 2
2012/10/05 15:37:17.409288 [ 5752]: Freeze priority 3
2012/10/05 15:37:17.409958 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:17.410259 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:17.410285 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:17.410293 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:17.457988 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:17.458117 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:17.458480 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:17.458732 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:17.459005 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:17.459018 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:17.459037 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:17.459297 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:17.459805 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:17.460048 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:17.460110 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:17.460452 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:17.460710 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:17.460735 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:17.460754 [ 5752]: Thawing priority 1
2012/10/05 15:37:17.460763 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:17.460776 [ 5752]: Thawing priority 2
2012/10/05 15:37:17.460782 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:17.460807 [ 5752]: Thawing priority 3
2012/10/05 15:37:17.460816 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:17.479735 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:17.479773 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:17.479783 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:17.498640 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:17.498665 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:17.498673 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:18.367229 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:18.367816 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:18.367839 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:18.367847 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:18.387244 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:18.387268 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:18.387339 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:18.387350 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:18.387922 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:18.387936 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:18.388240 [ 5752]: Freeze priority 1
2012/10/05 15:37:18.389106 [ 5752]: Freeze priority 2
2012/10/05 15:37:18.389829 [ 5752]: Freeze priority 3
2012/10/05 15:37:18.390528 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:18.390867 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:18.390881 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:18.390890 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:18.441044 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:18.442445 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:18.442766 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:18.443013 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:18.443281 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:18.443295 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:18.443315 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:18.443619 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:18.444080 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:18.444308 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:18.444388 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:18.444681 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:18.444918 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:18.444944 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:18.444954 [ 5752]: Thawing priority 1
2012/10/05 15:37:18.444960 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:18.444975 [ 5752]: Thawing priority 2
2012/10/05 15:37:18.444982 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:18.444992 [ 5752]: Thawing priority 3
2012/10/05 15:37:18.444998 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:18.475862 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:18.475897 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:18.475906 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:18.509048 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:18.509075 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:18.509083 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:19.368379 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:19.369085 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:19.369486 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:19.369500 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:19.388619 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:19.388638 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:19.388684 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:19.388694 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:19.389210 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:19.389224 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:19.389544 [ 5752]: Freeze priority 1
2012/10/05 15:37:19.390338 [ 5752]: Freeze priority 2
2012/10/05 15:37:19.391037 [ 5752]: Freeze priority 3
2012/10/05 15:37:19.391735 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:19.392045 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:19.392058 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:19.392067 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:19.440093 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:19.452026 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:19.452430 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:19.452741 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:19.453061 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:19.453075 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:19.453094 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:19.453323 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:19.453797 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:19.454026 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:19.454091 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:19.454387 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:19.454650 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:19.454676 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:19.454686 [ 5752]: Thawing priority 1
2012/10/05 15:37:19.454692 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:19.454709 [ 5752]: Thawing priority 2
2012/10/05 15:37:19.454716 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:19.454726 [ 5752]: Thawing priority 3
2012/10/05 15:37:19.454733 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:19.489126 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:19.489161 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:19.489170 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:19.510323 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:19.510344 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:19.510352 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:20.369930 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:20.370595 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:20.370611 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:20.370618 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:20.396166 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:20.396186 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:20.396232 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:20.396241 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:20.396768 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:20.396781 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:20.397086 [ 5752]: Freeze priority 1
2012/10/05 15:37:20.397917 [ 5752]: Freeze priority 2
2012/10/05 15:37:20.398607 [ 5752]: Freeze priority 3
2012/10/05 15:37:20.399299 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:20.399598 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:20.399621 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:20.399629 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:20.447972 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:20.448761 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:20.449116 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:20.449404 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:20.449695 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:20.449709 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:20.449729 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:20.450004 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:20.450500 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:20.450764 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:20.450830 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:20.451164 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:20.451437 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:20.451474 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:20.451486 [ 5752]: Thawing priority 1
2012/10/05 15:37:20.451492 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:20.451508 [ 5752]: Thawing priority 2
2012/10/05 15:37:20.451515 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:20.451525 [ 5752]: Thawing priority 3
2012/10/05 15:37:20.451531 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:20.470707 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:20.470729 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:20.470738 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:20.489321 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:20.489343 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:20.489352 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:21.371938 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:21.372657 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:21.372671 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:21.372677 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:21.407750 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:21.407786 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:21.407857 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:21.407867 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:21.408513 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:21.408541 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:21.408859 [ 5752]: Freeze priority 1
2012/10/05 15:37:21.409696 [ 5752]: Freeze priority 2
2012/10/05 15:37:21.410385 [ 5752]: Freeze priority 3
2012/10/05 15:37:21.411095 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:21.411396 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:21.411409 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:21.411418 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:21.460256 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:21.460399 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:21.460811 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:21.461081 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:21.461364 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:21.461379 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:21.461401 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:21.461697 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:21.462274 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:21.462552 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:21.462621 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:21.462959 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:21.463242 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:21.463268 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:21.463278 [ 5752]: Thawing priority 1
2012/10/05 15:37:21.463284 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:21.463298 [ 5752]: Thawing priority 2
2012/10/05 15:37:21.463305 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:21.463315 [ 5752]: Thawing priority 3
2012/10/05 15:37:21.463322 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:21.506678 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:21.506717 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:21.506726 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:21.507318 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:21.507330 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:21.507338 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:22.374049 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:22.374735 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:22.374750 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:22.374757 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:22.394655 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:22.394672 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:22.394713 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:22.394722 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:22.395235 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:22.395252 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:22.395564 [ 5752]: Freeze priority 1
2012/10/05 15:37:22.396339 [ 5752]: Freeze priority 2
2012/10/05 15:37:22.396979 [ 5752]: Freeze priority 3
2012/10/05 15:37:22.397650 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:22.397923 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:22.397935 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:22.397943 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:22.443923 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:22.444283 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:22.444648 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:22.444927 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:22.445248 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:22.445263 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:22.445283 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:22.445585 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:22.446074 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:22.446408 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:22.446474 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:22.446815 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:22.447086 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:22.447130 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:22.447140 [ 5752]: Thawing priority 1
2012/10/05 15:37:22.447147 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:22.447160 [ 5752]: Thawing priority 2
2012/10/05 15:37:22.447167 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:22.447177 [ 5752]: Thawing priority 3
2012/10/05 15:37:22.447184 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:22.460067 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:22.460090 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:22.460113 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:22.501066 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:22.501093 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:22.501134 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:23.375588 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:23.376205 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:23.376218 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:23.376225 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:23.401712 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:23.401735 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:23.401786 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:23.401795 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:23.402370 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:23.402383 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:23.402667 [ 5752]: Freeze priority 1
2012/10/05 15:37:23.403456 [ 5752]: Freeze priority 2
2012/10/05 15:37:23.404122 [ 5752]: Freeze priority 3
2012/10/05 15:37:23.404787 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:23.405088 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:23.405100 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:23.405108 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:23.451229 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:23.454100 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:23.454421 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:23.454718 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:23.454992 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:23.455005 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:23.455024 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:23.455320 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:23.455786 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:23.456032 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:23.456093 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:23.456431 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:23.456728 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:23.456752 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:23.456761 [ 5752]: Thawing priority 1
2012/10/05 15:37:23.456766 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:23.456782 [ 5752]: Thawing priority 2
2012/10/05 15:37:23.456788 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:23.456798 [ 5752]: Thawing priority 3
2012/10/05 15:37:23.456804 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:23.499540 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:23.499576 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:23.499585 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:23.511523 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:23.511541 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:23.511550 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:24.377181 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:24.377912 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:24.378385 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:24.378396 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:24.397955 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:24.397978 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:24.398029 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:24.398039 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:24.398606 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:24.398620 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:24.398937 [ 5752]: Freeze priority 1
2012/10/05 15:37:24.399736 [ 5752]: Freeze priority 2
2012/10/05 15:37:24.400422 [ 5752]: Freeze priority 3
2012/10/05 15:37:24.401094 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:24.401412 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:24.401426 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:24.401435 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:24.449117 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:24.463321 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:24.463710 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:24.464038 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:24.464377 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:24.464393 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:24.464427 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:24.464723 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:24.465275 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:24.465544 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:24.465607 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:24.465927 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:24.466195 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:24.466220 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:24.466240 [ 5752]: Thawing priority 1
2012/10/05 15:37:24.466247 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:24.466263 [ 5752]: Thawing priority 2
2012/10/05 15:37:24.466270 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:24.466280 [ 5752]: Thawing priority 3
2012/10/05 15:37:24.466286 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:24.549525 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:24.549563 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:24.549571 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:24.739083 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:24.739116 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:24.739125 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:25.378227 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:25.378877 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:25.378890 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:25.378896 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:25.397629 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:25.397645 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:25.397686 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:25.397695 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:25.398209 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:25.398222 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:25.398512 [ 5752]: Freeze priority 1
2012/10/05 15:37:25.399261 [ 5752]: Freeze priority 2
2012/10/05 15:37:25.399898 [ 5752]: Freeze priority 3
2012/10/05 15:37:25.400556 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:25.400870 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:25.400883 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:25.400891 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:25.449232 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:25.449432 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:25.449835 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:25.450148 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:25.450441 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:25.450454 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:25.450474 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:25.450777 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:25.451361 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:25.451654 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:25.451724 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:25.452069 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:25.452378 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:25.452409 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:25.452419 [ 5752]: Thawing priority 1
2012/10/05 15:37:25.452425 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:25.452437 [ 5752]: Thawing priority 2
2012/10/05 15:37:25.452444 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:25.452452 [ 5752]: Thawing priority 3
2012/10/05 15:37:25.452458 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:25.472067 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:25.472093 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:25.472102 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:25.489652 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:25.489669 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:25.489677 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:26.380294 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:26.381064 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:26.381081 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:26.381089 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:26.422639 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:26.422673 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:26.422737 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:26.422748 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:26.423319 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:26.423333 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:26.423633 [ 5752]: Freeze priority 1
2012/10/05 15:37:26.424413 [ 5752]: Freeze priority 2
2012/10/05 15:37:26.425089 [ 5752]: Freeze priority 3
2012/10/05 15:37:26.425747 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:26.426057 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:26.426073 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:26.426082 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:26.473689 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:26.474047 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:26.474444 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:26.474711 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:26.474990 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:26.475004 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:26.475026 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:26.475299 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:26.475822 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:26.476137 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:26.476203 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:26.476535 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:26.476808 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:26.476834 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:26.476844 [ 5752]: Thawing priority 1
2012/10/05 15:37:26.476873 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:26.476889 [ 5752]: Thawing priority 2
2012/10/05 15:37:26.476896 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:26.476906 [ 5752]: Thawing priority 3
2012/10/05 15:37:26.476913 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:26.521743 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:26.521779 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:26.521788 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:26.521795 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:26.521802 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:26.521808 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:27.382365 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:27.383032 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:27.383046 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:27.383053 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:27.402577 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:27.402594 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:27.402636 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:27.402646 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:27.403192 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:27.403206 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:27.403506 [ 5752]: Freeze priority 1
2012/10/05 15:37:27.404334 [ 5752]: Freeze priority 2
2012/10/05 15:37:27.405036 [ 5752]: Freeze priority 3
2012/10/05 15:37:27.405715 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:27.406016 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:27.406029 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:27.406037 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:27.454012 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:27.455687 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:27.456053 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:27.456346 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:27.456627 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:27.456640 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:27.456660 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:27.456963 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:27.457549 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:27.457807 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:27.457871 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:27.458203 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:27.458510 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:27.458542 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:27.458551 [ 5752]: Thawing priority 1
2012/10/05 15:37:27.458557 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:27.458570 [ 5752]: Thawing priority 2
2012/10/05 15:37:27.458576 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:27.458586 [ 5752]: Thawing priority 3
2012/10/05 15:37:27.458592 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:27.471965 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:27.471992 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:27.472000 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:27.521177 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:27.521211 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:27.521220 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:28.383926 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:28.384567 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:28.384581 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:28.384588 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:28.403756 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:28.403776 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:28.403822 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:28.403833 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:28.404376 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:28.404389 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:28.404688 [ 5752]: Freeze priority 1
2012/10/05 15:37:28.405527 [ 5752]: Freeze priority 2
2012/10/05 15:37:28.406209 [ 5752]: Freeze priority 3
2012/10/05 15:37:28.406884 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:28.407225 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:28.407238 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:28.407247 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:28.455195 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:28.468924 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:28.469279 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:28.469522 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:28.469796 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:28.469810 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:28.469831 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:28.470075 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:28.470546 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:28.470782 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:28.470848 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:28.471150 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:28.471391 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:28.471428 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:28.471438 [ 5752]: Thawing priority 1
2012/10/05 15:37:28.471445 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:28.471461 [ 5752]: Thawing priority 2
2012/10/05 15:37:28.471468 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:28.471478 [ 5752]: Thawing priority 3
2012/10/05 15:37:28.471485 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:28.514206 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:28.514243 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:28.514251 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:28.521261 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:28.521274 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:28.521282 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:29.385999 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:29.386703 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:29.387067 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:29.387080 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:29.412003 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:29.412023 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:29.412085 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:29.412096 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:29.412630 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:29.412643 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:29.412925 [ 5752]: Freeze priority 1
2012/10/05 15:37:29.413729 [ 5752]: Freeze priority 2
2012/10/05 15:37:29.414415 [ 5752]: Freeze priority 3
2012/10/05 15:37:29.415114 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:29.415411 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:29.415424 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:29.415442 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:29.463076 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:29.463208 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:29.463573 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:29.463830 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:29.464099 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:29.464111 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:29.464130 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:29.464387 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:29.464883 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:29.465153 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:29.465219 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:29.465555 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:29.465811 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:29.465837 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:29.465847 [ 5752]: Thawing priority 1
2012/10/05 15:37:29.465861 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:29.465875 [ 5752]: Thawing priority 2
2012/10/05 15:37:29.465882 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:29.465892 [ 5752]: Thawing priority 3
2012/10/05 15:37:29.465899 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:29.479779 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:29.479812 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:29.479822 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:29.502998 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:29.503016 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:29.503025 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:30.387568 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:30.388249 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:30.388263 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:30.388270 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:30.422828 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:30.422849 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:30.422895 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:30.422904 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:30.423419 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:30.423432 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:30.423743 [ 5752]: Freeze priority 1
2012/10/05 15:37:30.424517 [ 5752]: Freeze priority 2
2012/10/05 15:37:30.425163 [ 5752]: Freeze priority 3
2012/10/05 15:37:30.425839 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:30.426121 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:30.426135 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:30.426142 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:30.472074 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:30.472673 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:30.472989 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:30.473249 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:30.473493 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:30.473507 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:30.473528 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:30.473803 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:30.474272 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:30.474510 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:30.474577 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:30.474864 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:30.475105 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:30.475145 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:30.475155 [ 5752]: Thawing priority 1
2012/10/05 15:37:30.475162 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:30.475176 [ 5752]: Thawing priority 2
2012/10/05 15:37:30.475191 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:30.475202 [ 5752]: Thawing priority 3
2012/10/05 15:37:30.475209 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:30.520144 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:30.520178 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:30.520187 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:30.520194 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:30.520200 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:30.520206 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:31.389730 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:31.390442 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:31.390456 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:31.390462 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:31.411530 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:31.411547 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:31.411587 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:31.411596 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:31.412116 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:31.412129 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:31.412402 [ 5752]: Freeze priority 1
2012/10/05 15:37:31.413168 [ 5752]: Freeze priority 2
2012/10/05 15:37:31.413806 [ 5752]: Freeze priority 3
2012/10/05 15:37:31.414469 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:31.414792 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:31.414805 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:31.414813 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:31.461918 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:31.462887 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:31.463284 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:31.463598 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:31.463905 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:31.463919 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:31.463938 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:31.464242 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:31.464762 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:31.465022 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:31.465083 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:31.465437 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:31.465770 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:31.465797 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:31.465807 [ 5752]: Thawing priority 1
2012/10/05 15:37:31.465813 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:31.465828 [ 5752]: Thawing priority 2
2012/10/05 15:37:31.465835 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:31.465845 [ 5752]: Thawing priority 3
2012/10/05 15:37:31.465852 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:31.479374 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:31.479409 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:31.479418 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:31.519008 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:31.519033 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:31.519042 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:32.391703 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:32.392387 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:32.392401 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:32.392407 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:32.416828 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:32.416857 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:32.416913 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:32.416922 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:32.417498 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:32.417511 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:32.417809 [ 5752]: Freeze priority 1
2012/10/05 15:37:32.418581 [ 5752]: Freeze priority 2
2012/10/05 15:37:32.419224 [ 5752]: Freeze priority 3
2012/10/05 15:37:32.419894 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:32.420170 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:32.420183 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:32.420191 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:32.466640 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:32.468515 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:32.468901 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:32.469176 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:32.469457 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:32.469471 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:32.469490 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:32.469770 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:32.470267 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:32.470521 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:32.470583 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:32.470908 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:32.471174 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:32.471198 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:32.471207 [ 5752]: Thawing priority 1
2012/10/05 15:37:32.471213 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:32.471229 [ 5752]: Thawing priority 2
2012/10/05 15:37:32.471235 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:32.471244 [ 5752]: Thawing priority 3
2012/10/05 15:37:32.471264 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:32.514738 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:32.514784 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:32.514805 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:32.523157 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:32.523170 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:32.523178 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:33.393725 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:33.394391 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:33.394407 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:33.394414 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:33.413650 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:33.413670 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:33.413717 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:33.413726 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:33.414266 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:33.414279 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:33.414709 [ 5752]: Freeze priority 1
2012/10/05 15:37:33.415548 [ 5752]: Freeze priority 2
2012/10/05 15:37:33.416178 [ 5752]: Freeze priority 3
2012/10/05 15:37:33.416818 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:33.417095 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:33.417108 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:33.417125 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:33.465125 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:33.467800 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:33.468216 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:33.468632 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:33.468943 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:33.468959 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:33.468981 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:33.469264 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:33.469768 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:33.470047 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:33.470114 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:33.470408 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:33.470659 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:33.470686 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:33.470695 [ 5752]: Thawing priority 1
2012/10/05 15:37:33.470701 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:33.470718 [ 5752]: Thawing priority 2
2012/10/05 15:37:33.470724 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:33.470734 [ 5752]: Thawing priority 3
2012/10/05 15:37:33.470741 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:33.488133 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:33.488170 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:33.488179 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:33.540587 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:33.540616 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:33.540624 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:34.395334 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:34.396030 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:34.396292 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:34.396305 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:34.415774 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:34.415793 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:34.415842 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:34.415852 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:34.416400 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:34.416414 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:34.416695 [ 5752]: Freeze priority 1
2012/10/05 15:37:34.417502 [ 5752]: Freeze priority 2
2012/10/05 15:37:34.418175 [ 5752]: Freeze priority 3
2012/10/05 15:37:34.418868 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:34.419179 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:34.419192 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:34.419201 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:34.467467 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:34.467597 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:34.467993 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:34.468300 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:34.468584 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:34.468598 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:34.468617 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:34.468878 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:34.469413 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:34.469689 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:34.469756 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:34.470138 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:34.470443 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:34.470470 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:34.470480 [ 5752]: Thawing priority 1
2012/10/05 15:37:34.470485 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:34.470499 [ 5752]: Thawing priority 2
2012/10/05 15:37:34.470506 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:34.470515 [ 5752]: Thawing priority 3
2012/10/05 15:37:34.470521 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:34.510440 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:34.510477 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:34.510486 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:34.510494 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:34.510500 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:34.510516 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:35.396897 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:35.397607 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:35.397620 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:35.397626 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:35.438046 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:35.438066 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:35.438111 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:35.438121 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:35.438661 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:35.438675 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:35.438974 [ 5752]: Freeze priority 1
2012/10/05 15:37:35.439774 [ 5752]: Freeze priority 2
2012/10/05 15:37:35.440490 [ 5752]: Freeze priority 3
2012/10/05 15:37:35.441195 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:35.441496 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:35.441509 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:35.441518 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:35.499848 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:35.500021 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:35.500516 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:35.500792 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:35.501079 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:35.501094 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:35.501115 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:35.501390 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:35.501944 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:35.502209 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:35.502275 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:35.502589 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:35.502854 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:35.502881 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:35.502891 [ 5752]: Thawing priority 1
2012/10/05 15:37:35.502897 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:35.502910 [ 5752]: Thawing priority 2
2012/10/05 15:37:35.502917 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:35.502926 [ 5752]: Thawing priority 3
2012/10/05 15:37:35.502933 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:35.547780 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:35.547813 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:35.547822 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:35.547829 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:35.547835 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:35.547840 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:36.398485 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:36.399216 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:36.399229 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:36.399236 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:36.418400 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:36.418418 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:36.418462 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:36.418472 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:36.419003 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:36.419028 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:36.419324 [ 5752]: Freeze priority 1
2012/10/05 15:37:36.420150 [ 5752]: Freeze priority 2
2012/10/05 15:37:36.420854 [ 5752]: Freeze priority 3
2012/10/05 15:37:36.421564 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:36.421856 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:36.421870 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:36.421878 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:36.470727 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:36.471142 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:36.471521 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:36.471829 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:36.472133 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:36.472147 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:36.472168 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:36.472444 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:36.472975 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:36.473248 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:36.473316 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:36.474133 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:36.474424 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:36.474451 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:36.474460 [ 5752]: Thawing priority 1
2012/10/05 15:37:36.474466 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:36.474505 [ 5752]: Thawing priority 2
2012/10/05 15:37:36.474526 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:36.474539 [ 5752]: Thawing priority 3
2012/10/05 15:37:36.474546 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:36.487896 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:36.487934 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:36.487943 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:36.536396 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:36.536429 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:36.536437 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:37.400034 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:37.400699 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:37.400724 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:37.400732 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:37.419966 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:37.419983 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:37.420023 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:37.420033 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:37.420527 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:37.420540 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:37.420838 [ 5752]: Freeze priority 1
2012/10/05 15:37:37.421652 [ 5752]: Freeze priority 2
2012/10/05 15:37:37.422313 [ 5752]: Freeze priority 3
2012/10/05 15:37:37.422972 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:37.423292 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:37.423306 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:37.423314 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:37.469860 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:37.516910 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:37.517307 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:37.517607 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:37.517895 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:37.517910 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:37.517930 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:37.518220 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:37.518764 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:37.519026 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:37.519108 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:37.519427 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:37.519721 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:37.519749 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:37.519758 [ 5752]: Thawing priority 1
2012/10/05 15:37:37.519764 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:37.519782 [ 5752]: Thawing priority 2
2012/10/05 15:37:37.519788 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:37.519798 [ 5752]: Thawing priority 3
2012/10/05 15:37:37.519804 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:37.561680 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:37.561711 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:37.561720 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:37.582266 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:37.582281 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:37.582289 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:38.401562 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:38.402169 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:38.402183 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:38.402189 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:38.421425 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:38.421441 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:38.421483 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:38.421493 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:38.422006 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:38.422020 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:38.422321 [ 5752]: Freeze priority 1
2012/10/05 15:37:38.423107 [ 5752]: Freeze priority 2
2012/10/05 15:37:38.423791 [ 5752]: Freeze priority 3
2012/10/05 15:37:38.424438 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:38.424763 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:38.424776 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:38.424785 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:38.472586 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:38.480200 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:38.480570 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:38.480837 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:38.481137 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:38.481164 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:38.481188 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:38.481454 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:38.481978 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:38.482246 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:38.482312 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:38.482626 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:38.482900 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:38.482927 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:38.482936 [ 5752]: Thawing priority 1
2012/10/05 15:37:38.482942 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:38.482959 [ 5752]: Thawing priority 2
2012/10/05 15:37:38.482966 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:38.482976 [ 5752]: Thawing priority 3
2012/10/05 15:37:38.482982 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:38.496136 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:38.496176 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:38.496185 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:38.520268 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:38.520287 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:38.520295 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:39.402573 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:39.403233 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:39.403498 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:39.403510 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:39.444655 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:39.444679 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:39.444754 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:39.444765 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:39.445310 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:39.445325 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:39.445568 [ 5752]: Freeze priority 1
2012/10/05 15:37:39.446341 [ 5752]: Freeze priority 2
2012/10/05 15:37:39.447012 [ 5752]: Freeze priority 3
2012/10/05 15:37:39.447680 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:39.448002 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:39.448015 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:39.448024 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:39.495977 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:39.496766 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:39.497126 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:39.497439 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:39.497763 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:39.497777 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:39.497798 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:39.498083 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:39.498656 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:39.498944 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:39.499010 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:39.499348 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:39.499630 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:39.499657 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:39.499667 [ 5752]: Thawing priority 1
2012/10/05 15:37:39.499673 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:39.499697 [ 5752]: Thawing priority 2
2012/10/05 15:37:39.499708 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:39.499719 [ 5752]: Thawing priority 3
2012/10/05 15:37:39.499726 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:39.542672 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:39.542712 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:39.542723 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:39.542730 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:39.542736 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:39.542743 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:40.404287 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:40.405041 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:40.405059 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:40.405066 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:40.424580 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:40.424601 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:40.424649 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:40.424659 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:40.425245 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:40.425272 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:40.425565 [ 5752]: Freeze priority 1
2012/10/05 15:37:40.426392 [ 5752]: Freeze priority 2
2012/10/05 15:37:40.427083 [ 5752]: Freeze priority 3
2012/10/05 15:37:40.427791 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:40.428083 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:40.428096 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:40.428113 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:40.477167 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:40.488062 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:40.488488 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:40.488752 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:40.489043 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:40.489057 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:40.489076 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:40.489357 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:40.489916 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:40.490193 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:40.490260 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:40.490601 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:40.490920 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:40.490962 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:40.490974 [ 5752]: Thawing priority 1
2012/10/05 15:37:40.490981 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:40.491002 [ 5752]: Thawing priority 2
2012/10/05 15:37:40.491009 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:40.491021 [ 5752]: Thawing priority 3
2012/10/05 15:37:40.491028 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:40.505337 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:40.505376 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:40.505385 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:40.553944 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:40.553979 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:40.553988 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:41.405308 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:41.406039 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:41.406060 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:41.406069 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:41.434463 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:41.434485 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:41.434535 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:41.434545 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:41.435113 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:41.435139 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:41.435426 [ 5752]: Freeze priority 1
2012/10/05 15:37:41.436183 [ 5752]: Freeze priority 2
2012/10/05 15:37:41.436847 [ 5752]: Freeze priority 3
2012/10/05 15:37:41.437523 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:41.437848 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:41.437863 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:41.437871 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:41.486106 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:41.486237 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:41.486603 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:41.486847 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:41.487091 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:41.487104 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:41.487123 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:41.487360 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:41.487793 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:41.488094 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:41.488165 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:41.488472 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:41.488695 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:41.488720 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:41.488729 [ 5752]: Thawing priority 1
2012/10/05 15:37:41.488734 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:41.488747 [ 5752]: Thawing priority 2
2012/10/05 15:37:41.488753 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:41.488763 [ 5752]: Thawing priority 3
2012/10/05 15:37:41.488769 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:41.533302 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:41.533337 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:41.533359 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:41.541504 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:41.541517 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:41.541524 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:42.406872 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:42.407654 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:42.407670 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:42.407677 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:42.437084 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:42.437110 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:42.437167 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:42.437177 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:42.437729 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:42.437744 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:42.438058 [ 5752]: Freeze priority 1
2012/10/05 15:37:42.438856 [ 5752]: Freeze priority 2
2012/10/05 15:37:42.439718 [ 5752]: Freeze priority 3
2012/10/05 15:37:42.440453 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:42.440755 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:42.440769 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:42.440777 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:42.491344 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:42.491777 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:42.492116 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:42.492339 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:42.492588 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:42.492600 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:42.492619 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:42.492883 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:42.493380 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:42.493680 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:42.493749 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:42.494077 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:42.494317 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:42.494345 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:42.494355 [ 5752]: Thawing priority 1
2012/10/05 15:37:42.494361 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:42.494375 [ 5752]: Thawing priority 2
2012/10/05 15:37:42.494382 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:42.494413 [ 5752]: Thawing priority 3
2012/10/05 15:37:42.494420 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:42.507454 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:42.507490 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:42.507499 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:42.531752 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:42.531782 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:42.531819 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:43.408434 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:43.409155 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:43.409170 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:43.409177 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:43.447087 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:43.447124 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:43.447192 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:43.447202 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:43.447798 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:43.447817 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:43.448132 [ 5752]: Freeze priority 1
2012/10/05 15:37:43.448974 [ 5752]: Freeze priority 2
2012/10/05 15:37:43.449723 [ 5752]: Freeze priority 3
2012/10/05 15:37:43.450453 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:43.450797 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:43.450811 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:43.450819 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:43.500496 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:43.500628 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:43.501014 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:43.501295 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:43.501609 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:43.501623 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:43.501644 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:43.501924 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:43.502482 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:43.502755 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:43.502821 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:43.503199 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:43.503487 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:43.503532 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:43.503543 [ 5752]: Thawing priority 1
2012/10/05 15:37:43.503549 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:43.503562 [ 5752]: Thawing priority 2
2012/10/05 15:37:43.503569 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:43.503579 [ 5752]: Thawing priority 3
2012/10/05 15:37:43.503587 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:43.546247 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:43.546276 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:43.546285 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:43.546907 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:43.546920 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:43.546927 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:44.410504 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:44.411335 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:44.412184 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:44.412197 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:44.432048 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:44.432068 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:44.432111 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:44.432121 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:44.432655 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:44.432669 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:44.432976 [ 5752]: Freeze priority 1
2012/10/05 15:37:44.433836 [ 5752]: Freeze priority 2
2012/10/05 15:37:44.434571 [ 5752]: Freeze priority 3
2012/10/05 15:37:44.435261 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:44.435587 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:44.435609 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:44.435626 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:44.483941 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:44.484936 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:44.485300 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:44.485595 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:44.485878 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:44.485892 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:44.485912 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:44.486226 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:44.486776 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:44.487097 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:44.487166 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:44.487494 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:44.487771 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:44.487798 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:44.487807 [ 5752]: Thawing priority 1
2012/10/05 15:37:44.487814 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:44.487828 [ 5752]: Thawing priority 2
2012/10/05 15:37:44.487835 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:44.487845 [ 5752]: Thawing priority 3
2012/10/05 15:37:44.487852 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:44.501585 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:44.501613 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:44.501621 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:44.592609 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:44.592640 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:44.592648 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:45.412065 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:45.412780 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:45.412797 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:45.412804 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:45.438851 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:45.438869 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:45.438911 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:45.438920 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:45.439428 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:45.439440 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:45.439740 [ 5752]: Freeze priority 1
2012/10/05 15:37:45.440503 [ 5752]: Freeze priority 2
2012/10/05 15:37:45.441184 [ 5752]: Freeze priority 3
2012/10/05 15:37:45.441872 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:45.442164 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:45.442177 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:45.442185 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:45.488021 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:45.491340 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:45.491660 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:45.491885 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:45.492138 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:45.492152 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:45.492172 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:45.492402 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:45.492841 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:45.493065 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:45.493163 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:45.493447 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:45.493713 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:45.493739 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:45.493748 [ 5752]: Thawing priority 1
2012/10/05 15:37:45.493755 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:45.493770 [ 5752]: Thawing priority 2
2012/10/05 15:37:45.493777 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:45.493787 [ 5752]: Thawing priority 3
2012/10/05 15:37:45.493794 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:45.538579 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:45.538637 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:45.538648 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:45.538655 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:45.538661 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:45.538668 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:46.414142 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:46.414838 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:46.414852 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:46.414858 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:46.451252 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:46.451280 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:46.451336 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:46.451346 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:46.451869 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:46.451882 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:46.452176 [ 5752]: Freeze priority 1
2012/10/05 15:37:46.452949 [ 5752]: Freeze priority 2
2012/10/05 15:37:46.453617 [ 5752]: Freeze priority 3
2012/10/05 15:37:46.454338 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:46.454641 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:46.454654 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:46.454675 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:46.502495 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:46.503734 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:46.504081 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:46.504371 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:46.504648 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:46.504673 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:46.504698 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:46.504960 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:46.505468 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:46.505766 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:46.505831 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:46.506147 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:46.506431 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:46.506457 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:46.506467 [ 5752]: Thawing priority 1
2012/10/05 15:37:46.506473 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:46.506487 [ 5752]: Thawing priority 2
2012/10/05 15:37:46.506494 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:46.506504 [ 5752]: Thawing priority 3
2012/10/05 15:37:46.506511 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:46.539675 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:46.539712 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:46.539721 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:46.550283 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:46.550300 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:46.550308 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:47.415720 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:47.416433 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:47.416447 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:47.416454 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:47.435897 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:47.435914 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:47.435955 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:47.435965 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:47.436490 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:47.436504 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:47.436821 [ 5752]: Freeze priority 1
2012/10/05 15:37:47.437645 [ 5752]: Freeze priority 2
2012/10/05 15:37:47.438347 [ 5752]: Freeze priority 3
2012/10/05 15:37:47.439088 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:47.439392 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:47.439406 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:47.439424 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:47.487184 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:47.500564 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:47.500936 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:47.501169 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:47.501447 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:47.501461 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:47.501483 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:47.501748 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:47.502201 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:47.502443 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:47.502509 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:47.502803 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:47.503041 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:47.503068 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:47.503077 [ 5752]: Thawing priority 1
2012/10/05 15:37:47.503084 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:47.503101 [ 5752]: Thawing priority 2
2012/10/05 15:37:47.503108 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:47.503118 [ 5752]: Thawing priority 3
2012/10/05 15:37:47.503125 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:47.522248 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:47.522274 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:47.522283 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:47.553642 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:47.553668 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:47.553675 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:48.417227 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:48.417871 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:48.417885 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:48.417891 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:48.443314 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:48.443330 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:48.443369 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:48.443378 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:48.443893 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:48.443906 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:48.444182 [ 5752]: Freeze priority 1
2012/10/05 15:37:48.444957 [ 5752]: Freeze priority 2
2012/10/05 15:37:48.445637 [ 5752]: Freeze priority 3
2012/10/05 15:37:48.446323 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:48.446624 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:48.446637 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:48.446645 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:48.492856 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:48.494106 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:48.494475 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:48.494777 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:48.495092 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:48.495106 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:48.495135 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:48.495441 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:48.495905 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:48.496136 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:48.496197 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:48.496519 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:48.496775 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:48.496809 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:48.496820 [ 5752]: Thawing priority 1
2012/10/05 15:37:48.496825 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:48.496839 [ 5752]: Thawing priority 2
2012/10/05 15:37:48.496845 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:48.496855 [ 5752]: Thawing priority 3
2012/10/05 15:37:48.496861 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:48.529263 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:48.529293 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:48.529317 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:48.535212 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:48.535225 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:48.535232 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:49.418787 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:49.419472 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:49.419744 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:49.419759 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:49.457080 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:49.457120 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:49.457191 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:49.457200 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:49.457825 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:49.457840 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:49.458181 [ 5752]: Freeze priority 1
2012/10/05 15:37:49.458970 [ 5752]: Freeze priority 2
2012/10/05 15:37:49.459616 [ 5752]: Freeze priority 3
2012/10/05 15:37:49.460238 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:49.460550 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:49.460562 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:49.460570 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:49.509836 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:49.509992 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:49.510385 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:49.510684 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:49.511005 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:49.511019 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:49.511040 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:49.511375 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:49.511952 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:49.512245 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:49.512313 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:49.512644 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:49.512931 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:49.512958 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:49.512967 [ 5752]: Thawing priority 1
2012/10/05 15:37:49.512973 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:49.512988 [ 5752]: Thawing priority 2
2012/10/05 15:37:49.512995 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:49.513005 [ 5752]: Thawing priority 3
2012/10/05 15:37:49.513012 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:49.557331 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:49.557384 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:49.557396 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:49.557403 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:49.557409 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:49.557416 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:50.420893 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:50.421610 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:50.421626 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:50.421633 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:50.441618 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:50.441640 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:50.441689 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:50.441699 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:50.442263 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:50.442278 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:50.442570 [ 5752]: Freeze priority 1
2012/10/05 15:37:50.443325 [ 5752]: Freeze priority 2
2012/10/05 15:37:50.444000 [ 5752]: Freeze priority 3
2012/10/05 15:37:50.444656 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:50.444969 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:50.444981 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:50.444989 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:50.493711 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:50.493847 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:50.494252 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:50.494531 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:50.494823 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:50.494838 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:50.494860 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:50.495151 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:50.495604 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:50.495844 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:50.495911 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:50.496209 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:50.496458 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:50.496485 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:50.496494 [ 5752]: Thawing priority 1
2012/10/05 15:37:50.496500 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:50.496514 [ 5752]: Thawing priority 2
2012/10/05 15:37:50.496521 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:50.496531 [ 5752]: Thawing priority 3
2012/10/05 15:37:50.496538 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:50.509921 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:50.509961 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:50.509971 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:50.548554 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:50.548579 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:50.548587 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:51.421962 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:51.422726 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:51.422741 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:51.422748 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:51.448722 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:51.448744 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:51.448793 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:51.448802 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:51.449352 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:51.449366 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:51.449655 [ 5752]: Freeze priority 1
2012/10/05 15:37:51.450441 [ 5752]: Freeze priority 2
2012/10/05 15:37:51.451177 [ 5752]: Freeze priority 3
2012/10/05 15:37:51.451907 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:51.452211 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:51.452224 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:51.452242 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:51.500268 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:51.500694 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:51.501063 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:51.501300 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:51.501637 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:51.501654 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:51.501676 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:51.501931 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:51.502384 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:51.502614 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:51.502676 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:51.502956 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:51.503183 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:51.503207 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:51.503216 [ 5752]: Thawing priority 1
2012/10/05 15:37:51.503222 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:51.503235 [ 5752]: Thawing priority 2
2012/10/05 15:37:51.503241 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:51.503251 [ 5752]: Thawing priority 3
2012/10/05 15:37:51.503257 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:51.596133 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:51.596169 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:51.596178 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:51.596185 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:51.596191 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:51.596197 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:52.423506 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:52.424187 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:52.424201 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:52.424208 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:52.443486 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:52.443507 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:52.443554 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:52.443578 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:52.444109 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:52.444123 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:52.444412 [ 5752]: Freeze priority 1
2012/10/05 15:37:52.445205 [ 5752]: Freeze priority 2
2012/10/05 15:37:52.445892 [ 5752]: Freeze priority 3
2012/10/05 15:37:52.446600 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:52.446894 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:52.446908 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:52.446916 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:52.494392 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:52.494757 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:52.495162 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:52.495484 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:52.495801 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:52.495818 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:52.495839 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:52.496151 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:52.496721 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:52.497016 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:52.497100 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:52.497427 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:52.497733 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:52.497774 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:52.497784 [ 5752]: Thawing priority 1
2012/10/05 15:37:52.497790 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:52.497803 [ 5752]: Thawing priority 2
2012/10/05 15:37:52.497809 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:52.497819 [ 5752]: Thawing priority 3
2012/10/05 15:37:52.497825 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:52.511594 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:52.511627 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:52.511636 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:52.541430 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:52.541456 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:52.541464 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:53.425069 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:53.425772 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:53.425786 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:53.425792 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:53.461067 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:53.461109 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:53.461277 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:53.461289 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:53.461914 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:53.461927 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:53.462210 [ 5752]: Freeze priority 1
2012/10/05 15:37:53.462997 [ 5752]: Freeze priority 2
2012/10/05 15:37:53.463671 [ 5752]: Freeze priority 3
2012/10/05 15:37:53.464659 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:53.465101 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:53.465155 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:53.465166 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:53.518464 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:53.518860 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:53.519413 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:53.519720 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:53.519984 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:53.519999 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:53.520020 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:53.520339 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:53.520854 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:53.521084 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:53.521191 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:53.521513 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:53.521797 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:53.521826 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:53.521836 [ 5752]: Thawing priority 1
2012/10/05 15:37:53.521854 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:53.521872 [ 5752]: Thawing priority 2
2012/10/05 15:37:53.521879 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:53.521889 [ 5752]: Thawing priority 3
2012/10/05 15:37:53.521896 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:53.569555 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:53.569595 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:53.569604 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:53.577185 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:53.577211 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:53.577219 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:54.426716 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:54.427421 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:54.427931 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:54.427944 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:54.447992 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:54.448033 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:54.448107 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:54.448117 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:54.448698 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:54.448713 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:54.448995 [ 5752]: Freeze priority 1
2012/10/05 15:37:54.449777 [ 5752]: Freeze priority 2
2012/10/05 15:37:54.450421 [ 5752]: Freeze priority 3
2012/10/05 15:37:54.451052 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:54.451327 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:54.451341 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:54.451349 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:54.503302 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:54.503753 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:54.504306 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:54.504774 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:54.505022 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:54.505036 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:54.505056 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:54.505383 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:54.505959 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:54.506403 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:54.506524 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:54.507011 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:54.507345 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:54.507380 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:54.507391 [ 5752]: Thawing priority 1
2012/10/05 15:37:54.507397 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:54.507416 [ 5752]: Thawing priority 2
2012/10/05 15:37:54.507423 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:54.507448 [ 5752]: Thawing priority 3
2012/10/05 15:37:54.507455 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:54.521794 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:54.521833 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:54.521842 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:54.563935 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:54.563968 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:54.563976 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:55.428689 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:55.429410 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:55.429426 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:55.429432 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:55.455434 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:55.455454 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:55.455499 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:55.455508 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:55.456006 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:55.456019 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:55.456309 [ 5752]: Freeze priority 1
2012/10/05 15:37:55.457079 [ 5752]: Freeze priority 2
2012/10/05 15:37:55.457766 [ 5752]: Freeze priority 3
2012/10/05 15:37:55.458437 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:55.458738 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:55.458761 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:55.458772 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:55.505523 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:55.516565 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:55.516984 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:55.517296 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:55.517589 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:55.517602 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:55.517622 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:55.517908 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:55.518462 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:55.518736 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:55.518811 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:55.519140 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:55.519446 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:55.519471 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:55.519480 [ 5752]: Thawing priority 1
2012/10/05 15:37:55.519485 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:55.519501 [ 5752]: Thawing priority 2
2012/10/05 15:37:55.519508 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:55.519517 [ 5752]: Thawing priority 3
2012/10/05 15:37:55.519523 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:55.564176 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:55.564203 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:55.564211 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:55.573232 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:55.573246 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:55.573254 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:56.430727 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:56.431415 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:56.431429 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:56.431435 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:56.450613 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:56.450654 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:56.450725 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:56.450734 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:56.451358 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:56.451371 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:56.451674 [ 5752]: Freeze priority 1
2012/10/05 15:37:56.452431 [ 5752]: Freeze priority 2
2012/10/05 15:37:56.453102 [ 5752]: Freeze priority 3
2012/10/05 15:37:56.453732 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:56.454052 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:56.454065 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:56.454073 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:56.507280 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:56.507435 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:56.507886 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:56.508141 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:56.508409 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:56.508422 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:56.508442 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:56.508719 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:56.509258 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:56.509519 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:56.509586 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:56.509914 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:56.510178 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:56.510205 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:56.510215 [ 5752]: Thawing priority 1
2012/10/05 15:37:56.510221 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:56.510235 [ 5752]: Thawing priority 2
2012/10/05 15:37:56.510242 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:56.510252 [ 5752]: Thawing priority 3
2012/10/05 15:37:56.510259 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:56.524003 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:56.524041 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:56.524062 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:56.573541 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:56.573575 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:56.573584 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:57.432313 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:57.433014 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:57.433028 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:57.433035 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:57.453082 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:57.453101 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:57.453147 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:57.453158 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:57.453693 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:57.453709 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:57.454009 [ 5752]: Freeze priority 1
2012/10/05 15:37:57.454803 [ 5752]: Freeze priority 2
2012/10/05 15:37:57.455498 [ 5752]: Freeze priority 3
2012/10/05 15:37:57.456245 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:57.456582 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:57.456596 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:57.456605 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:57.504147 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:57.505349 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:57.505674 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:57.506351 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:57.506667 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:57.506680 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:57.506699 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:57.506974 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:57.507493 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:57.507728 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:57.507790 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:57.508161 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:57.508484 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:57.508516 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:57.508526 [ 5752]: Thawing priority 1
2012/10/05 15:37:57.508532 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:57.508547 [ 5752]: Thawing priority 2
2012/10/05 15:37:57.508554 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:57.508563 [ 5752]: Thawing priority 3
2012/10/05 15:37:57.508569 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:57.552998 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:57.553031 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:57.553039 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:57.553055 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:57.553061 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:57.553067 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:58.433486 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:58.434123 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:58.434140 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:58.434147 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:58.475384 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:58.475423 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:58.475506 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:58.475516 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:58.476091 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:58.476105 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:58.476334 [ 5752]: Freeze priority 1
2012/10/05 15:37:58.477133 [ 5752]: Freeze priority 2
2012/10/05 15:37:58.477763 [ 5752]: Freeze priority 3
2012/10/05 15:37:58.478467 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:58.478750 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:58.478764 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:58.478772 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:58.526814 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:58.528270 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:58.528689 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:58.528970 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:58.529271 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:58.529286 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:58.529307 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:58.529596 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:58.530127 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:58.530416 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:58.530498 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:58.530809 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:58.531048 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:58.531075 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:58.531085 [ 5752]: Thawing priority 1
2012/10/05 15:37:58.531091 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:58.531107 [ 5752]: Thawing priority 2
2012/10/05 15:37:58.531113 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:58.531124 [ 5752]: Thawing priority 3
2012/10/05 15:37:58.531130 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:58.575158 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:58.575194 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:58.575203 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:58.575210 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:58.575216 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:58.575232 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:37:59.435007 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:37:59.435692 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:37:59.436076 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:37:59.436089 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:37:59.455280 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:37:59.455308 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:37:59.455368 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:37:59.455377 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:37:59.455941 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:37:59.455955 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:37:59.456225 [ 5752]: Freeze priority 1
2012/10/05 15:37:59.456969 [ 5752]: Freeze priority 2
2012/10/05 15:37:59.457629 [ 5752]: Freeze priority 3
2012/10/05 15:37:59.458300 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:37:59.458655 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:37:59.458673 [ 5752]: Monitoring has been disabled
2012/10/05 15:37:59.458682 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:37:59.507067 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:37:59.508358 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:59.508689 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:59.508933 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:37:59.509197 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:37:59.509212 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:37:59.509233 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:37:59.509487 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:37:59.510068 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:37:59.510315 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:37:59.510382 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:37:59.510672 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:37:59.510914 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:37:59.510941 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:37:59.510950 [ 5752]: Thawing priority 1
2012/10/05 15:37:59.510956 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:37:59.510970 [ 5752]: Thawing priority 2
2012/10/05 15:37:59.510977 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:37:59.510988 [ 5752]: Thawing priority 3
2012/10/05 15:37:59.510994 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:37:59.524079 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:59.524101 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:37:59.524109 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:37:59.574912 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:37:59.574942 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:37:59.574951 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:00.436567 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:00.437287 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:00.437304 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:00.437311 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:00.456835 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:00.456855 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:00.456900 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:00.456910 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:00.457443 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:00.457457 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:00.457764 [ 5752]: Freeze priority 1
2012/10/05 15:38:00.458563 [ 5752]: Freeze priority 2
2012/10/05 15:38:00.459286 [ 5752]: Freeze priority 3
2012/10/05 15:38:00.459967 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:00.460280 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:00.460293 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:00.460302 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:00.525151 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:00.525309 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:00.525793 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:00.526075 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:00.526363 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:00.526377 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:00.526398 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:00.526678 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:00.527204 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:00.527466 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:00.527532 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:00.527857 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:00.528137 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:00.528164 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:00.528173 [ 5752]: Thawing priority 1
2012/10/05 15:38:00.528180 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:00.528193 [ 5752]: Thawing priority 2
2012/10/05 15:38:00.528200 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:00.528210 [ 5752]: Thawing priority 3
2012/10/05 15:38:00.528217 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:00.570693 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:00.570728 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:00.570737 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:00.633390 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:00.633416 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:00.633424 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:01.438109 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:01.438820 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:01.438847 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:01.438855 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:01.464000 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:01.464030 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:01.464091 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:01.464102 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:01.464683 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:01.464698 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:01.465679 [ 5752]: Freeze priority 1
2012/10/05 15:38:01.466488 [ 5752]: Freeze priority 2
2012/10/05 15:38:01.467185 [ 5752]: Freeze priority 3
2012/10/05 15:38:01.467911 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:01.468239 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:01.468253 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:01.468272 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:01.516702 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:01.517184 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:01.517492 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:01.517760 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:01.518015 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:01.518029 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:01.518050 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:01.518294 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:01.518761 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:01.518999 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:01.519065 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:01.519395 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:01.519639 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:01.519695 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:01.519706 [ 5752]: Thawing priority 1
2012/10/05 15:38:01.519711 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:01.519726 [ 5752]: Thawing priority 2
2012/10/05 15:38:01.519733 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:01.519742 [ 5752]: Thawing priority 3
2012/10/05 15:38:01.519749 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:01.533002 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:01.533034 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:01.533043 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:01.557307 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:01.557337 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:01.557347 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:02.439732 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:02.440416 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:02.440429 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:02.440435 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:02.474661 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:02.474697 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:02.474787 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:02.474798 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:02.475404 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:02.475417 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:02.475686 [ 5752]: Freeze priority 1
2012/10/05 15:38:02.476458 [ 5752]: Freeze priority 2
2012/10/05 15:38:02.477086 [ 5752]: Freeze priority 3
2012/10/05 15:38:02.477786 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:02.478087 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:02.478101 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:02.478109 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:02.526042 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:02.551074 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:02.551431 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:02.551691 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:02.551977 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:02.551990 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:02.552009 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:02.552278 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:02.552785 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:02.553038 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:02.553100 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:02.553418 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:02.553734 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:02.553763 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:02.553772 [ 5752]: Thawing priority 1
2012/10/05 15:38:02.553780 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:02.553798 [ 5752]: Thawing priority 2
2012/10/05 15:38:02.553804 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:02.553815 [ 5752]: Thawing priority 3
2012/10/05 15:38:02.553822 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:02.655471 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:02.655510 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:02.655519 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:02.655526 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:02.655532 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:02.655537 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:03.442611 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:03.444971 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:03.445026 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:03.445160 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:03.465481 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:03.465520 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:03.465606 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:03.465615 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:03.466212 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:03.466227 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:03.466491 [ 5752]: Freeze priority 1
2012/10/05 15:38:03.467282 [ 5752]: Freeze priority 2
2012/10/05 15:38:03.468638 [ 5752]: Freeze priority 3
2012/10/05 15:38:03.469611 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:03.469950 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:03.469967 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:03.469976 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:03.524841 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:03.525828 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:03.526117 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:03.526335 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:03.526571 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:03.526583 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:03.526602 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:03.526827 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:03.527237 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:03.527449 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:03.527510 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:03.527772 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:03.528001 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:03.528025 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:03.528033 [ 5752]: Thawing priority 1
2012/10/05 15:38:03.528039 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:03.528053 [ 5752]: Thawing priority 2
2012/10/05 15:38:03.528059 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:03.528069 [ 5752]: Thawing priority 3
2012/10/05 15:38:03.528075 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:03.542427 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:03.542569 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:03.542628 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:03.585089 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:03.585123 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:03.585131 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:04.442790 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:04.444332 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:04.444359 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:04.444367 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:04.470442 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:04.470465 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:04.470515 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:04.470525 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:04.471064 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:04.471077 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:04.471372 [ 5752]: Freeze priority 1
2012/10/05 15:38:04.472122 [ 5752]: Freeze priority 2
2012/10/05 15:38:04.472784 [ 5752]: Freeze priority 3
2012/10/05 15:38:04.473467 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:04.473767 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:04.473780 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:04.473789 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:04.532258 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:04.532428 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:04.532902 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:04.533194 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:04.533494 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:04.533508 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:04.533529 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:04.533810 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:04.534356 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:04.534623 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:04.534689 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:04.535049 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:04.535346 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:04.535374 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:04.535384 [ 5752]: Thawing priority 1
2012/10/05 15:38:04.535390 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:04.535403 [ 5752]: Thawing priority 2
2012/10/05 15:38:04.535410 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:04.535420 [ 5752]: Thawing priority 3
2012/10/05 15:38:04.535427 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:04.579696 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:04.579734 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:04.579744 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:04.589090 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:04.589106 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:04.589113 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:05.444849 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:05.445540 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:05.445554 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:05.445560 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:05.465596 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:05.465623 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:05.465680 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:05.465689 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:05.466259 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:05.466286 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:05.466578 [ 5752]: Freeze priority 1
2012/10/05 15:38:05.467345 [ 5752]: Freeze priority 2
2012/10/05 15:38:05.468026 [ 5752]: Freeze priority 3
2012/10/05 15:38:05.468707 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:05.469025 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:05.469038 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:05.469046 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:05.518422 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:05.518560 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:05.518942 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:05.519232 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:05.519553 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:05.519568 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:05.519589 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:05.519868 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:05.520443 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:05.520710 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:05.520776 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:05.521110 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:05.521391 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:05.521431 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:05.521442 [ 5752]: Thawing priority 1
2012/10/05 15:38:05.521448 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:05.521462 [ 5752]: Thawing priority 2
2012/10/05 15:38:05.521469 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:05.521479 [ 5752]: Thawing priority 3
2012/10/05 15:38:05.521486 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:05.534758 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:05.534787 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:05.534796 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:05.565029 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:05.565054 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:05.565062 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:06.446447 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:06.447171 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:06.447185 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:06.447192 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:06.482881 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:06.482915 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:06.482993 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:06.483005 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:06.483616 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:06.483631 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:06.483911 [ 5752]: Freeze priority 1
2012/10/05 15:38:06.484753 [ 5752]: Freeze priority 2
2012/10/05 15:38:06.485426 [ 5752]: Freeze priority 3
2012/10/05 15:38:06.486135 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:06.486452 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:06.486465 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:06.486488 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:06.535462 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:06.535613 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:06.535941 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:06.536188 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:06.536455 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:06.536467 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:06.536501 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:06.536770 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:06.537439 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:06.537794 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:06.537871 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:06.538212 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:06.538501 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:06.538529 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:06.538539 [ 5752]: Thawing priority 1
2012/10/05 15:38:06.538545 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:06.538560 [ 5752]: Thawing priority 2
2012/10/05 15:38:06.538567 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:06.538577 [ 5752]: Thawing priority 3
2012/10/05 15:38:06.538584 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:06.583992 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:06.584027 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:06.584036 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:06.584042 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:06.584048 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:06.584054 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:07.448475 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:07.449153 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:07.449167 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:07.449174 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:07.468727 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:07.468745 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:07.468786 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:07.468795 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:07.469334 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:07.469347 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:07.469654 [ 5752]: Freeze priority 1
2012/10/05 15:38:07.470428 [ 5752]: Freeze priority 2
2012/10/05 15:38:07.471101 [ 5752]: Freeze priority 3
2012/10/05 15:38:07.471754 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:07.472075 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:07.472087 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:07.472095 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:07.520541 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:07.520978 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:07.521317 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:07.521595 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:07.521856 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:07.521870 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:07.521891 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:07.522151 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:07.522633 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:07.522871 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:07.522937 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:07.523261 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:07.523520 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:07.523557 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:07.523569 [ 5752]: Thawing priority 1
2012/10/05 15:38:07.523575 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:07.523589 [ 5752]: Thawing priority 2
2012/10/05 15:38:07.523596 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:07.523606 [ 5752]: Thawing priority 3
2012/10/05 15:38:07.523613 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:07.537498 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:07.537532 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:07.537541 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:07.577951 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:07.577981 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:07.577989 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:08.450588 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:08.451287 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:08.451301 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:08.451307 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:08.476485 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:08.476514 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:08.476572 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:08.476581 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:08.477172 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:08.477191 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:08.477474 [ 5752]: Freeze priority 1
2012/10/05 15:38:08.478298 [ 5752]: Freeze priority 2
2012/10/05 15:38:08.478940 [ 5752]: Freeze priority 3
2012/10/05 15:38:08.479576 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:08.479852 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:08.479874 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:08.479882 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:08.525941 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:08.528219 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:08.528541 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:08.528834 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:08.529137 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:08.529152 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:08.529173 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:08.529439 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:08.529985 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:08.530277 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:08.530343 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:08.530679 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:08.530952 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:08.530978 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:08.530988 [ 5752]: Thawing priority 1
2012/10/05 15:38:08.530994 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:08.531009 [ 5752]: Thawing priority 2
2012/10/05 15:38:08.531016 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:08.531026 [ 5752]: Thawing priority 3
2012/10/05 15:38:08.531033 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:08.575195 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:08.575232 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:08.575241 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:08.585111 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:08.585167 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:08.585176 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:09.452630 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:09.453313 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:09.453543 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:09.453554 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:09.473112 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:09.473139 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:09.473214 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:09.473226 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:09.473781 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:09.473795 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:09.474079 [ 5752]: Freeze priority 1
2012/10/05 15:38:09.474853 [ 5752]: Freeze priority 2
2012/10/05 15:38:09.475553 [ 5752]: Freeze priority 3
2012/10/05 15:38:09.476245 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:09.476545 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:09.476559 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:09.476567 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:09.524438 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:09.524571 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:09.524960 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:09.525263 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:09.525568 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:09.525583 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:09.525605 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:09.525911 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:09.526502 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:09.526784 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:09.526851 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:09.527195 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:09.527493 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:09.527520 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:09.527530 [ 5752]: Thawing priority 1
2012/10/05 15:38:09.527536 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:09.527549 [ 5752]: Thawing priority 2
2012/10/05 15:38:09.527556 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:09.527566 [ 5752]: Thawing priority 3
2012/10/05 15:38:09.527573 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:09.541626 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:09.541673 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:09.541685 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:09.570424 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:09.570450 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:09.570458 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:10.454212 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:10.454935 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:10.454951 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:10.454958 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:10.490035 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:10.490061 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:10.490116 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:10.490126 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:10.490752 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:10.490766 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:10.491070 [ 5752]: Freeze priority 1
2012/10/05 15:38:10.491868 [ 5752]: Freeze priority 2
2012/10/05 15:38:10.492561 [ 5752]: Freeze priority 3
2012/10/05 15:38:10.493301 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:10.493639 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:10.493652 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:10.493660 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:10.541882 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:10.542585 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:10.542918 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:10.543195 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:10.543495 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:10.543509 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:10.543530 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:10.543809 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:10.544331 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:10.544590 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:10.544656 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:10.544982 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:10.545277 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:10.545304 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:10.545314 [ 5752]: Thawing priority 1
2012/10/05 15:38:10.545320 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:10.545334 [ 5752]: Thawing priority 2
2012/10/05 15:38:10.545341 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:10.545351 [ 5752]: Thawing priority 3
2012/10/05 15:38:10.545358 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:10.584217 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:10.584265 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:10.584274 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:10.589135 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:10.589151 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:10.589159 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:11.455789 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:11.456435 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:11.456450 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:11.456456 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:11.476478 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:11.476519 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:11.476606 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:11.476617 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:11.477248 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:11.477265 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:11.477558 [ 5752]: Freeze priority 1
2012/10/05 15:38:11.478333 [ 5752]: Freeze priority 2
2012/10/05 15:38:11.478942 [ 5752]: Freeze priority 3
2012/10/05 15:38:11.479599 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:11.479922 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:11.479937 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:11.479945 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:11.528017 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:11.558890 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:11.559316 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:11.559654 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:11.559975 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:11.559989 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:11.560008 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:11.560280 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:11.560861 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:11.561149 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:11.561216 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:11.561564 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:11.561855 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:11.561882 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:11.561892 [ 5752]: Thawing priority 1
2012/10/05 15:38:11.561898 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:11.561916 [ 5752]: Thawing priority 2
2012/10/05 15:38:11.561922 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:11.561932 [ 5752]: Thawing priority 3
2012/10/05 15:38:11.561939 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:11.604989 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:11.605019 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:11.605028 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:11.671119 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:11.671152 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:11.671160 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:12.457794 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:12.458468 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:12.458482 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:12.458488 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:12.479236 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:12.479271 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:12.479337 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:12.479347 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:12.480054 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:12.480068 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:12.480361 [ 5752]: Freeze priority 1
2012/10/05 15:38:12.481242 [ 5752]: Freeze priority 2
2012/10/05 15:38:12.481910 [ 5752]: Freeze priority 3
2012/10/05 15:38:12.482517 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:12.482778 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:12.482792 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:12.482800 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:12.532641 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:12.532777 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:12.533207 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:12.533455 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:12.533723 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:12.533737 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:12.533767 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:12.534026 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:12.534494 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:12.534727 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:12.534793 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:12.535090 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:12.535330 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:12.535356 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:12.535366 [ 5752]: Thawing priority 1
2012/10/05 15:38:12.535372 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:12.535407 [ 5752]: Thawing priority 2
2012/10/05 15:38:12.535414 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:12.535424 [ 5752]: Thawing priority 3
2012/10/05 15:38:12.535431 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:12.548936 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:12.548969 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:12.548980 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:12.573296 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:12.573318 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:12.573326 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:13.459950 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:13.461061 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:13.461077 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:13.461085 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:13.502491 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:13.502524 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:13.502587 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:13.502598 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:13.503206 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:13.503220 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:13.503507 [ 5752]: Freeze priority 1
2012/10/05 15:38:13.504312 [ 5752]: Freeze priority 2
2012/10/05 15:38:13.505002 [ 5752]: Freeze priority 3
2012/10/05 15:38:13.505704 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:13.506048 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:13.506062 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:13.506070 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:13.554671 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:13.555885 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:13.556234 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:13.556497 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:13.556784 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:13.556798 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:13.556819 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:13.557092 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:13.557629 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:13.557899 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:13.557980 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:13.558307 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:13.558581 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:13.558608 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:13.558618 [ 5752]: Thawing priority 1
2012/10/05 15:38:13.558624 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:13.558637 [ 5752]: Thawing priority 2
2012/10/05 15:38:13.558644 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:13.558654 [ 5752]: Thawing priority 3
2012/10/05 15:38:13.558661 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:13.662688 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:13.662728 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:13.662737 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:13.662744 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:13.662750 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:13.662757 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:14.460946 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:14.461640 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:14.461784 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:14.461795 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:14.481596 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:14.481625 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:14.481686 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:14.481696 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:14.482282 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:14.482296 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:14.482601 [ 5752]: Freeze priority 1
2012/10/05 15:38:14.483398 [ 5752]: Freeze priority 2
2012/10/05 15:38:14.484091 [ 5752]: Freeze priority 3
2012/10/05 15:38:14.484793 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:14.485129 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:14.485143 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:14.485151 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:14.548742 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:14.548901 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:14.549375 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:14.549676 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:14.549934 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:14.549948 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:14.549970 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:14.550222 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:14.550713 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:14.550956 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:14.551037 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:14.551325 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:14.551574 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:14.551600 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:14.551610 [ 5752]: Thawing priority 1
2012/10/05 15:38:14.551616 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:14.551629 [ 5752]: Thawing priority 2
2012/10/05 15:38:14.551636 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:14.551646 [ 5752]: Thawing priority 3
2012/10/05 15:38:14.551653 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:14.564759 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:14.564781 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:14.564789 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:14.614337 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:14.614370 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:14.614379 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:15.463053 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:15.463782 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:15.463798 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:15.463804 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:15.483194 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:15.483211 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:15.483253 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:15.483262 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:15.483760 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:15.483773 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:15.484066 [ 5752]: Freeze priority 1
2012/10/05 15:38:15.484824 [ 5752]: Freeze priority 2
2012/10/05 15:38:15.485480 [ 5752]: Freeze priority 3
2012/10/05 15:38:15.486164 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:15.486481 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:15.486494 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:15.486502 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:15.532432 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:15.535036 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:15.535357 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:15.535629 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:15.535904 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:15.535917 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:15.535936 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:15.536241 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:15.536764 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:15.537019 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:15.537093 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:15.537424 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:15.537731 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:15.537758 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:15.537768 [ 5752]: Thawing priority 1
2012/10/05 15:38:15.537774 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:15.537789 [ 5752]: Thawing priority 2
2012/10/05 15:38:15.537796 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:15.537806 [ 5752]: Thawing priority 3
2012/10/05 15:38:15.537812 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:15.572465 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:15.572494 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:15.572504 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:15.575619 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:15.575632 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:15.575640 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:16.465096 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:16.465851 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:16.465868 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:16.465875 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:16.507820 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:16.507853 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:16.507914 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:16.507924 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:16.508504 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:16.508520 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:16.508807 [ 5752]: Freeze priority 1
2012/10/05 15:38:16.509556 [ 5752]: Freeze priority 2
2012/10/05 15:38:16.510244 [ 5752]: Freeze priority 3
2012/10/05 15:38:16.510921 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:16.511245 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:16.511259 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:16.511268 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:16.559801 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:16.559939 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:16.560301 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:16.560562 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:16.560829 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:16.560843 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:16.560864 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:16.561153 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:16.561699 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:16.561968 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:16.562034 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:16.562367 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:16.562655 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:16.562694 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:16.562705 [ 5752]: Thawing priority 1
2012/10/05 15:38:16.562711 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:16.562724 [ 5752]: Thawing priority 2
2012/10/05 15:38:16.562731 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:16.562741 [ 5752]: Thawing priority 3
2012/10/05 15:38:16.562748 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:16.606546 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:16.606577 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:16.606586 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:16.606593 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:16.606599 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:16.606606 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:17.467155 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:17.467896 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:17.467913 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:17.467920 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:17.487848 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:17.487865 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:17.487905 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:17.487914 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:17.488445 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:17.488458 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:17.488763 [ 5752]: Freeze priority 1
2012/10/05 15:38:17.489526 [ 5752]: Freeze priority 2
2012/10/05 15:38:17.490179 [ 5752]: Freeze priority 3
2012/10/05 15:38:17.490863 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:17.491154 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:17.491167 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:17.491181 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:17.537259 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:17.539447 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:17.539787 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:17.540035 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:17.540307 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:17.540320 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:17.540339 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:17.540589 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:17.541091 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:17.541356 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:17.541424 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:17.541770 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:17.542048 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:17.542075 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:17.542084 [ 5752]: Thawing priority 1
2012/10/05 15:38:17.542100 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:17.542115 [ 5752]: Thawing priority 2
2012/10/05 15:38:17.542123 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:17.542132 [ 5752]: Thawing priority 3
2012/10/05 15:38:17.542139 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:17.555465 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:17.555492 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:17.555501 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:17.604034 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:17.604063 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:17.604071 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:18.468704 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:18.469360 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:18.469374 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:18.469380 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:18.489431 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:18.489463 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:18.489525 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:18.489534 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:18.490160 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:18.490177 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:18.490494 [ 5752]: Freeze priority 1
2012/10/05 15:38:18.491313 [ 5752]: Freeze priority 2
2012/10/05 15:38:18.492005 [ 5752]: Freeze priority 3
2012/10/05 15:38:18.492711 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:18.493035 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:18.493049 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:18.493058 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:18.539679 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:18.540615 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:18.541015 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:18.541344 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:18.541663 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:18.541676 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:18.541696 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:18.542005 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:18.542518 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:18.542794 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:18.542856 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:18.543173 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:18.543450 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:18.543475 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:18.543484 [ 5752]: Thawing priority 1
2012/10/05 15:38:18.543490 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:18.543503 [ 5752]: Thawing priority 2
2012/10/05 15:38:18.543510 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:18.543528 [ 5752]: Thawing priority 3
2012/10/05 15:38:18.543535 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:18.582761 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:18.582806 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:18.582816 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:18.596428 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:18.596442 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:18.596449 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:19.469780 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:19.470627 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:19.471203 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:19.471216 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:19.496666 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:19.496696 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:19.496757 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:19.496768 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:19.497369 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:19.497384 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:19.497678 [ 5752]: Freeze priority 1
2012/10/05 15:38:19.498506 [ 5752]: Freeze priority 2
2012/10/05 15:38:19.499172 [ 5752]: Freeze priority 3
2012/10/05 15:38:19.499868 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:19.500153 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:19.500166 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:19.500174 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:19.548067 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:19.557041 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:19.557411 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:19.557697 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:19.557991 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:19.558006 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:19.558026 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:19.558299 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:19.558857 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:19.559125 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:19.559190 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:19.559520 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:19.559793 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:19.559819 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:19.559841 [ 5752]: Thawing priority 1
2012/10/05 15:38:19.559850 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:19.559866 [ 5752]: Thawing priority 2
2012/10/05 15:38:19.559873 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:19.559884 [ 5752]: Thawing priority 3
2012/10/05 15:38:19.559890 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:19.594712 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:19.594744 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:19.594754 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:19.616121 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:19.616149 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:19.616157 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:20.471869 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:20.472514 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:20.472527 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:20.472533 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:20.492292 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:20.492322 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:20.492382 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:20.492404 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:20.492996 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:20.493014 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:20.493294 [ 5752]: Freeze priority 1
2012/10/05 15:38:20.494099 [ 5752]: Freeze priority 2
2012/10/05 15:38:20.494759 [ 5752]: Freeze priority 3
2012/10/05 15:38:20.495437 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:20.495727 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:20.495741 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:20.495749 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:20.545016 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:20.545148 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:20.545548 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:20.545852 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:20.546195 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:20.546212 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:20.546234 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:20.546534 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:20.547150 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:20.547434 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:20.547502 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:20.547826 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:20.548107 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:20.548138 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:20.548148 [ 5752]: Thawing priority 1
2012/10/05 15:38:20.548153 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:20.548166 [ 5752]: Thawing priority 2
2012/10/05 15:38:20.548172 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:20.548181 [ 5752]: Thawing priority 3
2012/10/05 15:38:20.548187 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:20.568636 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:20.568667 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:20.568687 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:20.591163 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:20.591188 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:20.591196 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:21.473909 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:21.474666 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:21.474697 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:21.474703 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:21.510711 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:21.510754 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:21.510831 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:21.510841 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:21.511502 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:21.511525 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:21.511803 [ 5752]: Freeze priority 1
2012/10/05 15:38:21.512590 [ 5752]: Freeze priority 2
2012/10/05 15:38:21.513277 [ 5752]: Freeze priority 3
2012/10/05 15:38:21.513990 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:21.514307 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:21.514321 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:21.514330 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:21.564294 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:21.564431 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:21.564801 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:21.565087 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:21.565376 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:21.565391 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:21.565412 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:21.565737 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:21.566260 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:21.566537 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:21.566603 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:21.566930 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:21.567228 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:21.567255 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:21.567264 [ 5752]: Thawing priority 1
2012/10/05 15:38:21.567270 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:21.567284 [ 5752]: Thawing priority 2
2012/10/05 15:38:21.567291 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:21.567301 [ 5752]: Thawing priority 3
2012/10/05 15:38:21.567307 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:21.611489 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:21.611524 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:21.611533 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:21.632212 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:21.632254 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:21.632263 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:22.475993 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:22.476725 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:22.476740 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:22.476746 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:22.496582 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:22.496610 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:22.496666 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:22.496676 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:22.497313 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:22.497348 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:22.497693 [ 5752]: Freeze priority 1
2012/10/05 15:38:22.498516 [ 5752]: Freeze priority 2
2012/10/05 15:38:22.499220 [ 5752]: Freeze priority 3
2012/10/05 15:38:22.499893 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:22.500203 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:22.500216 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:22.500225 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:22.548973 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:22.551110 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:22.551474 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:22.551777 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:22.552064 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:22.552078 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:22.552097 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:22.552362 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:22.552885 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:22.553161 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:22.553223 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:22.553543 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:22.553788 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:22.553813 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:22.553822 [ 5752]: Thawing priority 1
2012/10/05 15:38:22.553828 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:22.553841 [ 5752]: Thawing priority 2
2012/10/05 15:38:22.553848 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:22.553857 [ 5752]: Thawing priority 3
2012/10/05 15:38:22.553863 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:22.567794 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:22.567825 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:22.567834 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:22.611120 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:22.611149 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:22.611167 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:23.477540 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:23.478202 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:23.478216 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:23.478222 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:23.503637 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:23.503662 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:23.503715 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:23.503725 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:23.504308 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:23.504321 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:23.504603 [ 5752]: Freeze priority 1
2012/10/05 15:38:23.505368 [ 5752]: Freeze priority 2
2012/10/05 15:38:23.505995 [ 5752]: Freeze priority 3
2012/10/05 15:38:23.506660 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:23.506977 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:23.506990 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:23.506997 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:23.553535 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:23.553943 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:23.554287 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:23.554516 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:23.554757 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:23.554770 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:23.554790 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:23.555009 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:23.555441 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:23.555663 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:23.555725 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:23.555987 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:23.556283 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:23.556317 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:23.556328 [ 5752]: Thawing priority 1
2012/10/05 15:38:23.556335 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:23.556349 [ 5752]: Thawing priority 2
2012/10/05 15:38:23.556356 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:23.556366 [ 5752]: Thawing priority 3
2012/10/05 15:38:23.556373 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:23.598862 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:23.598897 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:23.598905 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:23.612416 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:23.612433 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:23.612441 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:24.479127 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:24.479954 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:24.479970 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:24.479977 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:24.499856 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:24.499887 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:24.499947 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:24.499958 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:24.500595 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:24.500614 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:24.500961 [ 5752]: Freeze priority 1
2012/10/05 15:38:24.501795 [ 5752]: Freeze priority 2
2012/10/05 15:38:24.502514 [ 5752]: Freeze priority 3
2012/10/05 15:38:24.503224 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:24.503517 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:24.503531 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:24.503539 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:24.552241 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:24.553003 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:24.553398 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:24.553752 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:24.554057 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:24.554070 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:24.554089 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:24.554416 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:24.555012 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:24.555300 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:24.555371 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:24.555732 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:24.556030 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:24.556061 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:24.556071 [ 5752]: Thawing priority 1
2012/10/05 15:38:24.556076 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:24.556089 [ 5752]: Thawing priority 2
2012/10/05 15:38:24.556096 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:24.556104 [ 5752]: Thawing priority 3
2012/10/05 15:38:24.556110 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:24.570212 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:24.570237 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:24.570246 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:24.599188 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:24.599206 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:24.599214 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:25.480678 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:25.481339 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:25.481364 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:25.481371 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:25.516362 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:25.516389 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:25.516444 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:25.516454 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:25.517016 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:25.517030 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:25.517318 [ 5752]: Freeze priority 1
2012/10/05 15:38:25.518110 [ 5752]: Freeze priority 2
2012/10/05 15:38:25.518787 [ 5752]: Freeze priority 3
2012/10/05 15:38:25.519469 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:25.519780 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:25.519793 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:25.519801 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:25.570019 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:25.570167 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:25.570730 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:25.570980 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:25.571298 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:25.571312 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:25.571332 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:25.571591 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:25.572086 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:25.572341 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:25.572402 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:25.572700 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:25.572934 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:25.572958 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:25.572966 [ 5752]: Thawing priority 1
2012/10/05 15:38:25.572972 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:25.572985 [ 5752]: Thawing priority 2
2012/10/05 15:38:25.572992 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:25.573001 [ 5752]: Thawing priority 3
2012/10/05 15:38:25.573007 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:25.613424 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:25.613461 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:25.613470 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:25.615520 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:25.615535 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:25.615543 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:26.482248 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:26.482878 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:26.482892 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:26.482899 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:26.502478 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:26.502515 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:26.502583 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:26.502593 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:26.503225 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:26.503240 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:26.503573 [ 5752]: Freeze priority 1
2012/10/05 15:38:26.504350 [ 5752]: Freeze priority 2
2012/10/05 15:38:26.505032 [ 5752]: Freeze priority 3
2012/10/05 15:38:26.505687 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:26.506004 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:26.506017 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:26.506025 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:26.552130 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:26.554476 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:26.554855 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:26.555128 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:26.555453 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:26.555470 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:26.555491 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:26.555758 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:26.556283 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:26.556558 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:26.556624 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:26.556945 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:26.557209 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:26.557236 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:26.557245 [ 5752]: Thawing priority 1
2012/10/05 15:38:26.557252 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:26.557266 [ 5752]: Thawing priority 2
2012/10/05 15:38:26.557273 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:26.557283 [ 5752]: Thawing priority 3
2012/10/05 15:38:26.557290 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:26.570557 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:26.570590 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:26.570599 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:26.613533 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:26.613563 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:26.613571 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:27.484336 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:27.484992 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:27.485006 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:27.485013 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:27.510010 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:27.510031 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:27.510093 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:27.510103 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:27.510622 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:27.510636 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:27.510943 [ 5752]: Freeze priority 1
2012/10/05 15:38:27.511733 [ 5752]: Freeze priority 2
2012/10/05 15:38:27.512404 [ 5752]: Freeze priority 3
2012/10/05 15:38:27.513059 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:27.513382 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:27.513396 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:27.513404 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:27.561967 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:27.562134 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:27.562633 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:27.562908 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:27.563185 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:27.563199 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:27.563219 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:27.563497 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:27.564055 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:27.564323 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:27.564406 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:27.564718 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:27.564992 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:27.565018 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:27.565028 [ 5752]: Thawing priority 1
2012/10/05 15:38:27.565034 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:27.565048 [ 5752]: Thawing priority 2
2012/10/05 15:38:27.565055 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:27.565065 [ 5752]: Thawing priority 3
2012/10/05 15:38:27.565072 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:27.605144 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:27.605169 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:27.605178 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:27.671630 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:27.671658 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:27.671667 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:28.486420 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:28.487063 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:28.487080 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:28.487086 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:28.507376 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:28.507399 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:28.507462 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:28.507474 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:28.508054 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:28.508068 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:28.508357 [ 5752]: Freeze priority 1
2012/10/05 15:38:28.509167 [ 5752]: Freeze priority 2
2012/10/05 15:38:28.509851 [ 5752]: Freeze priority 3
2012/10/05 15:38:28.510544 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:28.510862 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:28.510876 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:28.510884 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:28.559113 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:28.559250 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:28.559630 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:28.559956 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:28.560272 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:28.560288 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:28.560310 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:28.560596 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:28.561198 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:28.561522 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:28.561593 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:28.561955 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:28.562256 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:28.562287 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:28.562297 [ 5752]: Thawing priority 1
2012/10/05 15:38:28.562303 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:28.562315 [ 5752]: Thawing priority 2
2012/10/05 15:38:28.562322 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:28.562331 [ 5752]: Thawing priority 3
2012/10/05 15:38:28.562337 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:28.575733 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:28.575759 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:28.575768 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:28.605537 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:28.605555 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:28.605563 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:29.487951 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:29.488629 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:29.488866 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:29.488879 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:29.523222 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:29.523256 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:29.523321 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:29.523331 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:29.523949 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:29.523973 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:29.524267 [ 5752]: Freeze priority 1
2012/10/05 15:38:29.525038 [ 5752]: Freeze priority 2
2012/10/05 15:38:29.525696 [ 5752]: Freeze priority 3
2012/10/05 15:38:29.526410 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:29.526738 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:29.526752 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:29.526760 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:29.575200 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:29.575598 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:29.575890 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:29.576114 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:29.576341 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:29.576354 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:29.576373 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:29.576612 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:29.577041 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:29.577264 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:29.577329 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:29.577616 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:29.577851 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:29.577877 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:29.577887 [ 5752]: Thawing priority 1
2012/10/05 15:38:29.577893 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:29.577907 [ 5752]: Thawing priority 2
2012/10/05 15:38:29.577913 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:29.577924 [ 5752]: Thawing priority 3
2012/10/05 15:38:29.577930 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:29.622317 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:29.622355 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:29.622365 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:29.622933 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:29.622949 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:29.622957 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:30.489545 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:30.490212 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:30.490227 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:30.490233 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:30.509584 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:30.509624 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:30.509692 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:30.509702 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:30.510317 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:30.510330 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:30.510612 [ 5752]: Freeze priority 1
2012/10/05 15:38:30.511408 [ 5752]: Freeze priority 2
2012/10/05 15:38:30.512088 [ 5752]: Freeze priority 3
2012/10/05 15:38:30.512750 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:30.513071 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:30.513084 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:30.513092 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:30.560800 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:30.561263 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:30.561655 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:30.561951 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:30.562264 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:30.562278 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:30.562299 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:30.562581 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:30.563093 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:30.563351 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:30.563417 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:30.563740 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:30.564007 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:30.564034 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:30.564056 [ 5752]: Thawing priority 1
2012/10/05 15:38:30.564065 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:30.564079 [ 5752]: Thawing priority 2
2012/10/05 15:38:30.564086 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:30.564096 [ 5752]: Thawing priority 3
2012/10/05 15:38:30.564102 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:30.577297 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:30.577320 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:30.577329 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:30.617143 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:30.617177 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:30.617185 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:31.491570 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:31.492204 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:31.492218 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:31.492225 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:31.517734 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:31.517761 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:31.517818 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:31.517828 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:31.518402 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:31.518416 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:31.518712 [ 5752]: Freeze priority 1
2012/10/05 15:38:31.519518 [ 5752]: Freeze priority 2
2012/10/05 15:38:31.520231 [ 5752]: Freeze priority 3
2012/10/05 15:38:31.520923 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:31.521225 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:31.521239 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:31.521247 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:31.569292 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:31.569808 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:31.570155 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:31.570398 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:31.570655 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:31.570669 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:31.570690 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:31.570932 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:31.571387 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:31.571614 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:31.571704 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:31.571991 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:31.572237 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:31.572263 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:31.572273 [ 5752]: Thawing priority 1
2012/10/05 15:38:31.572279 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:31.572293 [ 5752]: Thawing priority 2
2012/10/05 15:38:31.572300 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:31.572310 [ 5752]: Thawing priority 3
2012/10/05 15:38:31.572316 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:31.610161 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:31.610195 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:31.610204 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:31.624538 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:31.624559 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:31.624567 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:32.493140 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:32.493815 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:32.493829 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:32.493835 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:32.514781 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:32.514800 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:32.514844 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:32.514854 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:32.515370 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:32.515384 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:32.515678 [ 5752]: Freeze priority 1
2012/10/05 15:38:32.516476 [ 5752]: Freeze priority 2
2012/10/05 15:38:32.517136 [ 5752]: Freeze priority 3
2012/10/05 15:38:32.517819 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:32.518207 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:32.518229 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:32.518238 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:32.565515 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:32.566800 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:32.567165 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:32.567506 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:32.567812 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:32.567826 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:32.567845 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:32.568113 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:32.568642 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:32.568909 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:32.568971 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:32.569328 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:32.569609 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:32.569636 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:32.569646 [ 5752]: Thawing priority 1
2012/10/05 15:38:32.569652 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:32.569666 [ 5752]: Thawing priority 2
2012/10/05 15:38:32.569673 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:32.569694 [ 5752]: Thawing priority 3
2012/10/05 15:38:32.569706 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:32.582916 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:32.582939 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:32.582947 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:32.621131 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:32.621156 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:32.621164 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:33.494709 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:33.495377 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:33.495391 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:33.495397 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:33.528504 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:33.528525 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:33.528573 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:33.528583 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:33.529183 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:33.529198 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:33.529524 [ 5752]: Freeze priority 1
2012/10/05 15:38:33.530347 [ 5752]: Freeze priority 2
2012/10/05 15:38:33.531074 [ 5752]: Freeze priority 3
2012/10/05 15:38:33.531808 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:33.532127 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:33.532140 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:33.532149 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:33.579739 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:33.592152 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:33.592545 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:33.592828 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:33.593140 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:33.593156 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:33.593177 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:33.593461 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:33.594035 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:33.594314 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:33.594380 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:33.594703 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:33.594985 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:33.595012 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:33.595021 [ 5752]: Thawing priority 1
2012/10/05 15:38:33.595027 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:33.595044 [ 5752]: Thawing priority 2
2012/10/05 15:38:33.595051 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:33.595061 [ 5752]: Thawing priority 3
2012/10/05 15:38:33.595068 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:33.638805 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:33.638842 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:33.638850 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:33.645607 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:33.645623 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:33.645631 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:34.496284 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:34.496992 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:34.497190 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:34.497205 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:34.516728 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:34.516747 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:34.516795 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:34.516805 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:34.517398 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:34.517412 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:34.517703 [ 5752]: Freeze priority 1
2012/10/05 15:38:34.518534 [ 5752]: Freeze priority 2
2012/10/05 15:38:34.519226 [ 5752]: Freeze priority 3
2012/10/05 15:38:34.519926 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:34.520241 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:34.520255 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:34.520263 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:34.567694 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:34.569163 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:34.569598 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:34.569890 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:34.570211 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:34.570228 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:34.570250 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:34.570554 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:34.571159 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:34.571465 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:34.571536 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:34.571903 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:34.572191 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:34.572218 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:34.572227 [ 5752]: Thawing priority 1
2012/10/05 15:38:34.572232 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:34.572246 [ 5752]: Thawing priority 2
2012/10/05 15:38:34.572252 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:34.572261 [ 5752]: Thawing priority 3
2012/10/05 15:38:34.572267 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:34.585705 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:34.585728 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:34.585736 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:34.621383 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:34.621405 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:34.621413 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:35.497828 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:35.498494 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:35.498510 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:35.498517 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:35.529506 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:35.529527 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:35.529576 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:35.529586 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:35.530138 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:35.530153 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:35.530467 [ 5752]: Freeze priority 1
2012/10/05 15:38:35.531267 [ 5752]: Freeze priority 2
2012/10/05 15:38:35.531978 [ 5752]: Freeze priority 3
2012/10/05 15:38:35.532683 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:35.532976 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:35.532989 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:35.532997 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:35.580503 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:35.581330 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:35.581692 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:35.581962 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:35.582224 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:35.582238 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:35.582258 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:35.582516 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:35.582979 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:35.583224 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:35.583289 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:35.583586 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:35.583836 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:35.583863 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:35.583872 [ 5752]: Thawing priority 1
2012/10/05 15:38:35.583878 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:35.583907 [ 5752]: Thawing priority 2
2012/10/05 15:38:35.583915 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:35.583925 [ 5752]: Thawing priority 3
2012/10/05 15:38:35.583931 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:35.628789 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:35.628823 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:35.628832 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:35.631280 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:35.631294 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:35.631301 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:36.499898 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:36.500547 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:36.500562 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:36.500568 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:36.520671 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:36.520691 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:36.520737 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:36.520747 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:36.521287 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:36.521302 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:36.521625 [ 5752]: Freeze priority 1
2012/10/05 15:38:36.522465 [ 5752]: Freeze priority 2
2012/10/05 15:38:36.523166 [ 5752]: Freeze priority 3
2012/10/05 15:38:36.523854 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:36.524181 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:36.524195 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:36.524203 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:36.572253 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:36.572397 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:36.572778 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:36.573071 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:36.573357 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:36.573371 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:36.573401 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:36.573693 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:36.574226 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:36.574501 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:36.574566 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:36.574887 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:36.575163 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:36.575189 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:36.575199 [ 5752]: Thawing priority 1
2012/10/05 15:38:36.575205 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:36.575218 [ 5752]: Thawing priority 2
2012/10/05 15:38:36.575225 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:36.575235 [ 5752]: Thawing priority 3
2012/10/05 15:38:36.575242 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:36.588790 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:36.588816 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:36.588824 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:36.636919 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:36.636949 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:36.636974 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:37.501487 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:37.502159 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:37.502173 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:37.502179 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:37.521699 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:37.521720 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:37.521767 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:37.521777 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:37.522293 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:37.522307 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:37.522603 [ 5752]: Freeze priority 1
2012/10/05 15:38:37.523392 [ 5752]: Freeze priority 2
2012/10/05 15:38:37.524068 [ 5752]: Freeze priority 3
2012/10/05 15:38:37.524736 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:37.525038 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:37.525052 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:37.525060 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:37.572326 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:37.587276 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:37.587679 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:37.587968 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:37.588270 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:37.588283 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:37.588303 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:37.588577 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:37.589125 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:37.589400 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:37.589466 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:37.589806 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:37.590096 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:37.590123 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:37.590132 [ 5752]: Thawing priority 1
2012/10/05 15:38:37.590139 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:37.590156 [ 5752]: Thawing priority 2
2012/10/05 15:38:37.590163 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:37.590173 [ 5752]: Thawing priority 3
2012/10/05 15:38:37.590180 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:37.630627 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:37.630662 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:37.630672 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:37.652467 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:37.652483 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:37.652491 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:38.503044 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:38.503700 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:38.503713 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:38.503719 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:38.522800 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:38.522819 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:38.522861 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:38.522870 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:38.523397 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:38.523409 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:38.523729 [ 5752]: Freeze priority 1
2012/10/05 15:38:38.524501 [ 5752]: Freeze priority 2
2012/10/05 15:38:38.525179 [ 5752]: Freeze priority 3
2012/10/05 15:38:38.525860 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:38.526173 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:38.526186 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:38.526193 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:38.571990 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:38.574035 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:38.574371 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:38.574625 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:38.574898 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:38.574911 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:38.574931 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:38.575194 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:38.575691 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:38.575948 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:38.576009 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:38.576315 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:38.576567 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:38.576603 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:38.576613 [ 5752]: Thawing priority 1
2012/10/05 15:38:38.576619 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:38.576633 [ 5752]: Thawing priority 2
2012/10/05 15:38:38.576639 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:38.576648 [ 5752]: Thawing priority 3
2012/10/05 15:38:38.576655 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:38.590733 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:38.590754 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:38.590762 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:38.613264 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:38.613279 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:38.613287 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:39.504604 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:39.505288 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:39.505425 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:39.505436 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:39.545973 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:39.545999 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:39.546056 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:39.546065 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:39.546628 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:39.546655 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:39.546950 [ 5752]: Freeze priority 1
2012/10/05 15:38:39.547763 [ 5752]: Freeze priority 2
2012/10/05 15:38:39.548430 [ 5752]: Freeze priority 3
2012/10/05 15:38:39.549109 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:39.549426 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:39.549440 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:39.549449 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:39.597321 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:39.606417 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:39.606800 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:39.607066 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:39.607363 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:39.607377 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:39.607398 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:39.607673 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:39.608182 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:39.608431 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:39.608497 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:39.608827 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:39.609089 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:39.609116 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:39.609125 [ 5752]: Thawing priority 1
2012/10/05 15:38:39.609131 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:39.609179 [ 5752]: Thawing priority 2
2012/10/05 15:38:39.609189 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:39.609200 [ 5752]: Thawing priority 3
2012/10/05 15:38:39.609207 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:39.653995 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:39.654028 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:39.654037 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:39.654044 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:39.654049 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:39.654056 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:40.506654 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:40.507347 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:40.507363 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:40.507369 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:40.526481 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:40.526500 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:40.526545 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:40.526555 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:40.527124 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:40.527138 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:40.527462 [ 5752]: Freeze priority 1
2012/10/05 15:38:40.528279 [ 5752]: Freeze priority 2
2012/10/05 15:38:40.528975 [ 5752]: Freeze priority 3
2012/10/05 15:38:40.529649 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:40.529954 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:40.529967 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:40.529976 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:40.579156 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:40.579309 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:40.579679 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:40.579997 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:40.580311 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:40.580328 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:40.580349 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:40.580646 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:40.581288 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:40.581594 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:40.581665 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:40.582001 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:40.582294 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:40.582328 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:40.582337 [ 5752]: Thawing priority 1
2012/10/05 15:38:40.582343 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:40.582356 [ 5752]: Thawing priority 2
2012/10/05 15:38:40.582362 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:40.582371 [ 5752]: Thawing priority 3
2012/10/05 15:38:40.582377 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:40.595635 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:40.595658 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:40.595666 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:40.638594 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:40.638620 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:40.638628 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:41.508231 [recoverd: 5823]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/05 15:38:41.508918 [recoverd: 5823]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/05 15:38:41.508932 [recoverd: 5823]: Taking out recovery lock from recovery daemon
2012/10/05 15:38:41.508938 [recoverd: 5823]: Take the recovery lock
2012/10/05 15:38:41.528818 [recoverd: 5823]: Recovery lock taken successfully
2012/10/05 15:38:41.528847 [recoverd: 5823]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/05 15:38:41.528907 [recoverd: 5823]: Recovery lock taken successfully by recovery daemon
2012/10/05 15:38:41.528917 [recoverd: 5823]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/05 15:38:41.529578 [recoverd: 5823]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/05 15:38:41.529593 [recoverd: 5823]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/05 15:38:41.529855 [ 5752]: Freeze priority 1
2012/10/05 15:38:41.530672 [ 5752]: Freeze priority 2
2012/10/05 15:38:41.531376 [ 5752]: Freeze priority 3
2012/10/05 15:38:41.532084 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/05 15:38:41.532413 [ 5752]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/05 15:38:41.532427 [ 5752]: Monitoring has been disabled
2012/10/05 15:38:41.532435 [ 5752]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/05 15:38:41.583829 [ 5752]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/05 15:38:41.589325 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:41.589740 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:41.590027 [recoverd: 5823]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/05 15:38:41.590302 [recoverd: 5823]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/05 15:38:41.590317 [recoverd: 5823]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/05 15:38:41.590338 [ 5752]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/05 15:38:41.590598 [recoverd: 5823]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/05 15:38:41.591077 [recoverd: 5823]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/05 15:38:41.591341 [recoverd: 5823]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/05 15:38:41.591409 [ 5752]: Control modflags on node 0 - Unchanged - flags 0x2
2012/10/05 15:38:41.591716 [ 5752]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/05 15:38:41.592001 [recoverd: 5823]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/05 15:38:41.592028 [ 5752]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/05 15:38:41.592037 [ 5752]: Thawing priority 1
2012/10/05 15:38:41.592056 [ 5752]: Release freeze handler for prio 1
2012/10/05 15:38:41.592075 [ 5752]: Thawing priority 2
2012/10/05 15:38:41.592082 [ 5752]: Release freeze handler for prio 2
2012/10/05 15:38:41.592093 [ 5752]: Thawing priority 3
2012/10/05 15:38:41.592099 [ 5752]: Release freeze handler for prio 3
2012/10/05 15:38:41.634302 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:41.634340 [recoverd: 5823]: ctdb_control error: 'managed to lock reclock file from inside daemon'
2012/10/05 15:38:41.634350 [recoverd: 5823]: Async operation failed with ret=-1 res=-1 opcode=16
2012/10/05 15:38:41.644782 [recoverd: 5823]: Async wait failed - fail_count=1
2012/10/05 15:38:41.644802 [recoverd: 5823]: server/ctdb_recoverd.c:309 Unable to set recovery mode. Recovery failed.
2012/10/05 15:38:41.644809 [recoverd: 5823]: server/ctdb_recoverd.c:1620 Unable to set recovery mode to normal on cluster
2012/10/05 15:38:42.220597 [ 5752]: Shutting down recovery daemon
2012/10/05 15:38:42.220633 [ 5752]: Monitoring has been stopped
2012/10/05 15:38:42.220659 [ 5752]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/05 15:38:42.220736 [ 5752]: server/eventscript.c:754 Starting eventscript shutdown
2012/10/05 15:38:42.226214 [ 5752]: server/eventscript.c:485 Eventscript shutdown finished with state -5
2012/10/05 15:38:42.226278 [ 5752]: Received SHUTDOWN command. Stopping CTDB daemon.
2012/10/05 15:38:42.226294 [ 5752]: CTDB daemon shutting down
More information about the samba-technical
mailing list