ctdb relock file issues with glusterfs
patrick medina
pgmedinajr at gmail.com
Tue Oct 9 10:11:20 MDT 2012
Please ignore last email logs, those were from the previous client.
Here are the new log files, it was a late work night yesterday...
On Tue, Oct 9, 2012 at 10:06 AM, patrick medina <pgmedinajr at gmail.com>wrote:
> Our test lab just went semi production so I built a replica on another
> stack of servers we have.
>
> Results are the same unfortunately, but here are the log files from both
> clients running ctdb.
>
> Client 1 is running ERR=4, client 2 is running ERR=2
>
> Cheers,
> Gilbert
>
>
> On Tue, Oct 9, 2012 at 5:20 AM, Martin Schwenke <martin at meltin.net> wrote:
>
>> On Tue, 9 Oct 2012 16:32:12 +1100, Amitay Isaacs <amitay at gmail.com>
>> wrote:
>>
>> > On Tue, Oct 9, 2012 at 1:55 PM, patrick medina <pgmedinajr at gmail.com>
>> wrote:
>> > > Howdy samba folks,
>> > >
>> > > I've been running into a lot of issues lately with ctdb's re-lock
>> file and
>> > > glusterfs as the shared storage. When I started, I could get one or
>> the
>> > > other node to become healthy, but at least one would complain it
>> could not
>> > > lock the re-lock file. Nowi'm at the point where neither node will
>> become
>> > > healthy and stay in a recovery loop. Just to be sure it was the
>> re-lock
>> > > file, I commented it out in the config and both nodes became healthy.
>>
>> > What version of CTDB are you using? Can you attach the log file where
>> > you notice CTDB is continuously going in recovery? It would be useful
>> > to get log files from all the nodes.
>>
>> Michael Adam and I took a look at this on the weekend. Gilbert sent me
>> some logs and this was happening:
>>
>> ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
>>
>> That seems to indicate that locking isn't working as expected...
>>
>> peace & happiness,
>> martin
>>
>
>
-------------- next part --------------
2012/10/08 20:25:51.932028 [11382]: CTDB starting on node
2012/10/08 20:25:51.935752 [11383]: Starting CTDBD as pid : 11383
2012/10/08 20:25:51.935892 [11383]: Set scheduler to SCHED_FIFO
2012/10/08 20:25:51.935999 [11383]: server/ctdb_logging.c:551 Created PIPE FD:7 for logging
2012/10/08 20:25:51.936090 [11383]: Added timed event "ctdb_statistics_update": 0x25aa0c0
2012/10/08 20:25:51.936188 [11383]: tcp/tcp_connect.c:362 Failed to bind() to socket. Cannot assign requested address(99)
2012/10/08 20:25:51.936247 [11383]: ctdb chose network address 10.100.55.229:4379 pnn 1
2012/10/08 20:25:51.936362 [11383]: Added timed event "ctdb_check_interfaces_event": 0x25aa8e0
2012/10/08 20:25:51.936619 [11383]: server/eventscript.c:754 Starting eventscript init
2012/10/08 20:25:51.936746 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:25:51.936781 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:25:51.936805 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:25:51.936823 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:25:51.937539 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:51.937587 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ade80
2012/10/08 20:25:51.937999 [eventscript-00.ctdb-init:11384]: Executing event script /etc/ctdb/events.d/00.ctdb init
2012/10/08 20:25:51.958395 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:51.959105 [eventscript-01.reclock-init:11392]: Executing event script /etc/ctdb/events.d/01.reclock init
2012/10/08 20:25:51.970757 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:51.971414 [eventscript-10.interface-init:11397]: Executing event script /etc/ctdb/events.d/10.interface init
2012/10/08 20:25:51.983044 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:51.983715 [eventscript-11.natgw-init:11402]: Executing event script /etc/ctdb/events.d/11.natgw init
2012/10/08 20:25:51.992395 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:51.993044 [eventscript-11.routing-init:11406]: Executing event script /etc/ctdb/events.d/11.routing init
2012/10/08 20:25:52.001833 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.002599 [eventscript-13.per_ip_routing-init:11410]: Executing event script /etc/ctdb/events.d/13.per_ip_routing init
2012/10/08 20:25:52.014009 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.014703 [eventscript-20.multipathd-init:11415]: Executing event script /etc/ctdb/events.d/20.multipathd init
2012/10/08 20:25:52.015172 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.015858 [eventscript-31.clamd-init:11416]: Executing event script /etc/ctdb/events.d/31.clamd init
2012/10/08 20:25:52.016343 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.016837 [eventscript-40.vsftpd-init:11417]: Executing event script /etc/ctdb/events.d/40.vsftpd init
2012/10/08 20:25:52.025910 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.026598 [eventscript-40.fs_use-init:11421]: Executing event script /etc/ctdb/events.d/40.fs_use init
2012/10/08 20:25:52.027060 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.027696 [eventscript-41.httpd-init:11422]: Executing event script /etc/ctdb/events.d/41.httpd init
2012/10/08 20:25:52.036887 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.037561 [eventscript-50.samba-init:11426]: Executing event script /etc/ctdb/events.d/50.samba init
2012/10/08 20:25:52.049681 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.050369 [eventscript-60.nfs-init:11431]: Executing event script /etc/ctdb/events.d/60.nfs init
2012/10/08 20:25:52.064454 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.065162 [eventscript-60.ganesha-init:11437]: Executing event script /etc/ctdb/events.d/60.ganesha init
2012/10/08 20:25:52.065619 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.066303 [eventscript-62.cnfs-init:11438]: Executing event script /etc/ctdb/events.d/62.cnfs init
2012/10/08 20:25:52.076713 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.077185 [eventscript-70.iscsi-init:11443]: Executing event script /etc/ctdb/events.d/70.iscsi init
2012/10/08 20:25:52.083743 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:52.084219 [eventscript-91.lvs-init:11447]: Executing event script /etc/ctdb/events.d/91.lvs init
2012/10/08 20:25:52.090478 [11383]: server/eventscript.c:485 Eventscript init finished with state 0
2012/10/08 20:25:52.090522 [11383]: Destroying timer event 0x25ade80 "ctdb_event_script_timeout"
2012/10/08 20:25:52.091996 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface 'eth0' refs[0])
2012/10/08 20:25:52.092034 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25ada10
2012/10/08 20:25:52.092271 [11383]: Added timed event "ctdb_check_recd": 0x25adac0
2012/10/08 20:25:52.092309 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25adb70
2012/10/08 20:25:52.092328 [11383]: Keepalive monitoring has been started
2012/10/08 20:25:52.092336 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25ade00
2012/10/08 20:25:52.092342 [11383]: Monitoring has been started
2012/10/08 20:25:52.092353 [11383]: Added timed event "ctdb_update_tcp_tickles": 0x25adff0
2012/10/08 20:25:52.092369 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:25:52.092378 [11383]: Added timed event "ctdb_time_tick": 0x25ae300
2012/10/08 20:25:52.092418 [11383]: server/eventscript.c:754 Starting eventscript setup
2012/10/08 20:25:52.092481 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:25:52.092513 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:25:52.092525 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:25:52.092539 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:25:52.092840 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.092882 [11383]: Added timed event "ctdb_event_script_timeout": 0x25b20f0
2012/10/08 20:25:52.095020 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:18
2012/10/08 20:25:52.095077 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2250
2012/10/08 20:25:52.095086 [11383]: Ending timer event 0x25ada10 "ctdb_tcp_node_connect"
2012/10/08 20:25:52.095232 [11383]: SIGCHLD from 11384
2012/10/08 20:25:52.095250 [11383]: SIGCHLD from 11392
2012/10/08 20:25:52.095261 [11383]: SIGCHLD from 11397
2012/10/08 20:25:52.095272 [11383]: SIGCHLD from 11402
2012/10/08 20:25:52.095282 [11383]: SIGCHLD from 11406
2012/10/08 20:25:52.095291 [11383]: SIGCHLD from 11410
2012/10/08 20:25:52.095301 [11383]: SIGCHLD from 11415
2012/10/08 20:25:52.095311 [11383]: SIGCHLD from 11416
2012/10/08 20:25:52.095320 [11383]: SIGCHLD from 11417
2012/10/08 20:25:52.095330 [11383]: SIGCHLD from 11421
2012/10/08 20:25:52.095340 [11383]: SIGCHLD from 11422
2012/10/08 20:25:52.095350 [11383]: SIGCHLD from 11426
2012/10/08 20:25:52.095360 [11383]: SIGCHLD from 11431
2012/10/08 20:25:52.095370 [11383]: SIGCHLD from 11437
2012/10/08 20:25:52.095379 [11383]: SIGCHLD from 11438
2012/10/08 20:25:52.095406 [11383]: SIGCHLD from 11443
2012/10/08 20:25:52.095420 [11383]: SIGCHLD from 11447
2012/10/08 20:25:52.095504 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:20 to connected child
2012/10/08 20:25:52.095702 [eventscript-00.ctdb-setup:11455]: Executing event script /etc/ctdb/events.d/00.ctdb setup
2012/10/08 20:25:52.096747 [11383]: Destroying timer event 0x25b2250 "ctdb_tcp_node_connect"
2012/10/08 20:25:52.096765 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b21a0
2012/10/08 20:25:52.096814 [recoverd:11454]: server/ctdb_recoverd.c:3600 Created PIPE FD:14 to recovery daemon
2012/10/08 20:25:52.096920 [recoverd:11454]: monitor_cluster starting
2012/10/08 20:25:52.096968 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.096982 [11383]: Added timed event "queue_next_trigger": 0x25b5080
2012/10/08 20:25:52.096989 [11383]: Added timed event "ctdb_control_timeout": 0x25b51a0
2012/10/08 20:25:52.096998 [11383]: server/ctdb_server.c:331 ctdb request 2147483449 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097007 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=17870283321406128128
2012/10/08 20:25:52.097016 [11383]: Added timed event "queue_next_trigger": 0x25b5580
2012/10/08 20:25:52.097023 [11383]: Ending timer event 0x25b5080 "queue_next_trigger"
2012/10/08 20:25:52.097030 [11383]: server/ctdb_server.c:331 ctdb request 2147483449 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097041 [11383]: Destroying timer event 0x25b51a0 "ctdb_control_timeout"
2012/10/08 20:25:52.097049 [11383]: Ending timer event 0x25b5580 "queue_next_trigger"
2012/10/08 20:25:52.097094 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097105 [11383]: Added timed event "queue_next_trigger": 0x25b4fb0
2012/10/08 20:25:52.097112 [11383]: Added timed event "ctdb_control_timeout": 0x25b56a0
2012/10/08 20:25:52.097120 [11383]: server/ctdb_server.c:331 ctdb request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097127 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=17365880163140632576
2012/10/08 20:25:52.097135 [11383]: Added timed event "queue_next_trigger": 0x25b5970
2012/10/08 20:25:52.097142 [11383]: Ending timer event 0x25b4fb0 "queue_next_trigger"
2012/10/08 20:25:52.097149 [11383]: server/ctdb_server.c:331 ctdb request 2147483450 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097159 [11383]: Destroying timer event 0x25b56a0 "ctdb_control_timeout"
2012/10/08 20:25:52.097166 [11383]: Ending timer event 0x25b5970 "queue_next_trigger"
2012/10/08 20:25:52.097195 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097205 [11383]: Added timed event "queue_next_trigger": 0x25b5850
2012/10/08 20:25:52.097213 [11383]: Added timed event "ctdb_control_timeout": 0x25b5970
2012/10/08 20:25:52.097221 [11383]: server/ctdb_server.c:331 ctdb request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097229 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=17582052945254416384
2012/10/08 20:25:52.097236 [11383]: Added timed event "queue_next_trigger": 0x25b5d20
2012/10/08 20:25:52.097243 [11383]: Ending timer event 0x25b5850 "queue_next_trigger"
2012/10/08 20:25:52.097250 [11383]: server/ctdb_server.c:331 ctdb request 2147483451 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097259 [11383]: Destroying timer event 0x25b5970 "ctdb_control_timeout"
2012/10/08 20:25:52.097267 [11383]: Ending timer event 0x25b5d20 "queue_next_trigger"
2012/10/08 20:25:52.097295 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097306 [11383]: Added timed event "queue_next_trigger": 0x25b5c00
2012/10/08 20:25:52.097313 [11383]: Added timed event "ctdb_control_timeout": 0x25b5d20
2012/10/08 20:25:52.097325 [11383]: server/ctdb_server.c:331 ctdb request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097333 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=17942340915444056064
2012/10/08 20:25:52.097341 [11383]: Added timed event "queue_next_trigger": 0x25b60d0
2012/10/08 20:25:52.097348 [11383]: Ending timer event 0x25b5c00 "queue_next_trigger"
2012/10/08 20:25:52.097355 [11383]: server/ctdb_server.c:331 ctdb request 2147483452 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097364 [11383]: Destroying timer event 0x25b5d20 "ctdb_control_timeout"
2012/10/08 20:25:52.097371 [11383]: Ending timer event 0x25b60d0 "queue_next_trigger"
2012/10/08 20:25:52.097400 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097410 [11383]: Added timed event "queue_next_trigger": 0x25b5fb0
2012/10/08 20:25:52.097418 [11383]: Added timed event "ctdb_control_timeout": 0x25b60d0
2012/10/08 20:25:52.097425 [11383]: server/ctdb_server.c:331 ctdb request 2147483453 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097432 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=17798225727368200192
2012/10/08 20:25:52.097440 [11383]: Added timed event "queue_next_trigger": 0x25b6480
2012/10/08 20:25:52.097447 [11383]: Ending timer event 0x25b5fb0 "queue_next_trigger"
2012/10/08 20:25:52.097454 [11383]: server/ctdb_server.c:331 ctdb request 2147483453 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097463 [11383]: Destroying timer event 0x25b60d0 "ctdb_control_timeout"
2012/10/08 20:25:52.097470 [11383]: Ending timer event 0x25b6480 "queue_next_trigger"
2012/10/08 20:25:52.097499 [11383]: server/ctdb_daemon.c:628 client request 2147483453 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097509 [11383]: Added timed event "queue_next_trigger": 0x25b6360
2012/10/08 20:25:52.097517 [11383]: Added timed event "ctdb_control_timeout": 0x25b6480
2012/10/08 20:25:52.097524 [11383]: server/ctdb_server.c:331 ctdb request 2147483454 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097531 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=18014398509481984000
2012/10/08 20:25:52.097539 [11383]: Added timed event "queue_next_trigger": 0x25b6830
2012/10/08 20:25:52.097545 [11383]: Ending timer event 0x25b6360 "queue_next_trigger"
2012/10/08 20:25:52.097552 [11383]: server/ctdb_server.c:331 ctdb request 2147483454 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097562 [11383]: Destroying timer event 0x25b6480 "ctdb_control_timeout"
2012/10/08 20:25:52.097569 [11383]: Ending timer event 0x25b6830 "queue_next_trigger"
2012/10/08 20:25:52.097598 [11383]: server/ctdb_daemon.c:628 client request 2147483454 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097608 [11383]: Added timed event "queue_next_trigger": 0x25b6710
2012/10/08 20:25:52.097615 [11383]: Added timed event "ctdb_control_timeout": 0x25b6830
2012/10/08 20:25:52.097623 [11383]: server/ctdb_server.c:331 ctdb request 2147483455 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097630 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=18086456103519911936
2012/10/08 20:25:52.097637 [11383]: Added timed event "queue_next_trigger": 0x25b6be0
2012/10/08 20:25:52.097644 [11383]: Ending timer event 0x25b6710 "queue_next_trigger"
2012/10/08 20:25:52.097651 [11383]: server/ctdb_server.c:331 ctdb request 2147483455 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097660 [11383]: Destroying timer event 0x25b6830 "ctdb_control_timeout"
2012/10/08 20:25:52.097668 [11383]: Ending timer event 0x25b6be0 "queue_next_trigger"
2012/10/08 20:25:52.097696 [11383]: server/ctdb_daemon.c:628 client request 2147483455 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097707 [11383]: Added timed event "queue_next_trigger": 0x25b6ac0
2012/10/08 20:25:52.097714 [11383]: Added timed event "ctdb_control_timeout": 0x25b6be0
2012/10/08 20:25:52.097721 [11383]: server/ctdb_server.c:331 ctdb request 2147483456 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097733 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=18158513697557839872
2012/10/08 20:25:52.097741 [11383]: Added timed event "queue_next_trigger": 0x25b6f90
2012/10/08 20:25:52.097748 [11383]: Ending timer event 0x25b6ac0 "queue_next_trigger"
2012/10/08 20:25:52.097755 [11383]: server/ctdb_server.c:331 ctdb request 2147483456 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097765 [11383]: Destroying timer event 0x25b6be0 "ctdb_control_timeout"
2012/10/08 20:25:52.097772 [11383]: Ending timer event 0x25b6f90 "queue_next_trigger"
2012/10/08 20:25:52.097801 [11383]: server/ctdb_daemon.c:628 client request 2147483456 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097811 [11383]: Added timed event "queue_next_trigger": 0x25b6e70
2012/10/08 20:25:52.097819 [11383]: Added timed event "ctdb_control_timeout": 0x25b6f90
2012/10/08 20:25:52.097826 [11383]: server/ctdb_server.c:331 ctdb request 2147483457 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097834 [11383]: server/ctdb_daemon.c:187 Registered message handler for srvid=17654110539292344320
2012/10/08 20:25:52.097841 [11383]: Added timed event "queue_next_trigger": 0x25b7340
2012/10/08 20:25:52.097848 [11383]: Ending timer event 0x25b6e70 "queue_next_trigger"
2012/10/08 20:25:52.097855 [11383]: server/ctdb_server.c:331 ctdb request 2147483457 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097864 [11383]: Destroying timer event 0x25b6f90 "ctdb_control_timeout"
2012/10/08 20:25:52.097872 [11383]: Ending timer event 0x25b7340 "queue_next_trigger"
2012/10/08 20:25:52.097908 [11383]: server/ctdb_daemon.c:628 client request 2147483457 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.097918 [11383]: Added timed event "queue_next_trigger": 0x25b7220
2012/10/08 20:25:52.097926 [11383]: Added timed event "ctdb_control_timeout": 0x25b7340
2012/10/08 20:25:52.097933 [11383]: server/ctdb_server.c:331 ctdb request 2147483458 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.097941 [11383]: Destroying timer event 0x25ae1e0 "ctdb_recd_ping_timeout"
2012/10/08 20:25:52.097948 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:25:52.097955 [11383]: Added timed event "queue_next_trigger": 0x25b76d0
2012/10/08 20:25:52.097962 [11383]: Ending timer event 0x25b7220 "queue_next_trigger"
2012/10/08 20:25:52.097969 [11383]: server/ctdb_server.c:331 ctdb request 2147483458 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.097978 [11383]: Destroying timer event 0x25b7340 "ctdb_control_timeout"
2012/10/08 20:25:52.097985 [11383]: Ending timer event 0x25b76d0 "queue_next_trigger"
2012/10/08 20:25:52.098019 [11383]: server/ctdb_daemon.c:628 client request 2147483458 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.098030 [11383]: Added timed event "queue_next_trigger": 0x25b75b0
2012/10/08 20:25:52.098037 [11383]: Added timed event "ctdb_control_timeout": 0x25b76d0
2012/10/08 20:25:52.098044 [11383]: server/ctdb_server.c:331 ctdb request 2147483459 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098052 [11383]: Added timed event "queue_next_trigger": 0x25b79f0
2012/10/08 20:25:52.098059 [11383]: Ending timer event 0x25b75b0 "queue_next_trigger"
2012/10/08 20:25:52.098066 [11383]: server/ctdb_server.c:331 ctdb request 2147483459 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.098076 [11383]: Destroying timer event 0x25b76d0 "ctdb_control_timeout"
2012/10/08 20:25:52.098083 [11383]: Ending timer event 0x25b79f0 "queue_next_trigger"
2012/10/08 20:25:52.098117 [11383]: server/ctdb_daemon.c:628 client request 2147483459 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.098128 [11383]: Added timed event "queue_next_trigger": 0x25b78d0
2012/10/08 20:25:52.098135 [11383]: Added timed event "ctdb_control_timeout": 0x25b79f0
2012/10/08 20:25:52.098143 [11383]: server/ctdb_server.c:331 ctdb request 2147483460 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098156 [11383]: Added timed event "queue_next_trigger": 0x25b7e70
2012/10/08 20:25:52.098163 [11383]: Ending timer event 0x25b78d0 "queue_next_trigger"
2012/10/08 20:25:52.098170 [11383]: server/ctdb_server.c:331 ctdb request 2147483460 of type 8 length 228 from node 1 to 1
2012/10/08 20:25:52.098181 [11383]: Destroying timer event 0x25b79f0 "ctdb_control_timeout"
2012/10/08 20:25:52.098188 [11383]: Ending timer event 0x25b7e70 "queue_next_trigger"
2012/10/08 20:25:52.098251 [11383]: server/ctdb_daemon.c:628 client request 2147483460 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.098263 [11383]: Added timed event "queue_next_trigger": 0x25b7bf0
2012/10/08 20:25:52.098271 [11383]: Added timed event "ctdb_control_timeout": 0x25b7d10
2012/10/08 20:25:52.098278 [11383]: server/ctdb_server.c:331 ctdb request 2147483461 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098287 [11383]: Added timed event "queue_next_trigger": 0x25b80a0
2012/10/08 20:25:52.098294 [11383]: Ending timer event 0x25b7bf0 "queue_next_trigger"
2012/10/08 20:25:52.098301 [11383]: server/ctdb_server.c:331 ctdb request 2147483461 of type 8 length 67 from node 1 to 1
2012/10/08 20:25:52.098311 [11383]: Destroying timer event 0x25b7d10 "ctdb_control_timeout"
2012/10/08 20:25:52.098318 [11383]: Ending timer event 0x25b80a0 "queue_next_trigger"
2012/10/08 20:25:52.098355 [11383]: server/ctdb_daemon.c:628 client request 2147483461 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.098365 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.098373 [11383]: Added timed event "ctdb_control_timeout": 0x25b8070
2012/10/08 20:25:52.098381 [11383]: server/ctdb_server.c:331 ctdb request 2147483462 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098389 [11383]: Added timed event "queue_next_trigger": 0x25b83c0
2012/10/08 20:25:52.098396 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.098403 [11383]: server/ctdb_server.c:331 ctdb request 2147483462 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.098412 [11383]: Destroying timer event 0x25b8070 "ctdb_control_timeout"
2012/10/08 20:25:52.098419 [11383]: Ending timer event 0x25b83c0 "queue_next_trigger"
2012/10/08 20:25:52.098449 [11383]: server/ctdb_daemon.c:628 client request 2147483462 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098459 [11383]: Added timed event "queue_next_trigger": 0x25b82a0
2012/10/08 20:25:52.098467 [11383]: Added timed event "ctdb_control_timeout": 0x25b83c0
2012/10/08 20:25:52.098474 [11383]: server/ctdb_server.c:331 ctdb request 2147483463 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098483 [11383]: Added timed event "queue_next_trigger": 0x25b8700
2012/10/08 20:25:52.098490 [11383]: Ending timer event 0x25b82a0 "queue_next_trigger"
2012/10/08 20:25:52.098497 [11383]: server/ctdb_server.c:331 ctdb request 2147483463 of type 8 length 60 from node 1 to 1
2012/10/08 20:25:52.098507 [11383]: Destroying timer event 0x25b83c0 "ctdb_control_timeout"
2012/10/08 20:25:52.098514 [11383]: Ending timer event 0x25b8700 "queue_next_trigger"
2012/10/08 20:25:52.098545 [11383]: server/ctdb_daemon.c:628 client request 2147483463 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098555 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.098563 [11383]: Added timed event "ctdb_control_timeout": 0x25b8630
2012/10/08 20:25:52.098570 [11383]: server/ctdb_server.c:331 ctdb request 2147483464 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098579 [11383]: Added timed event "queue_next_trigger": 0x25b8ae0
2012/10/08 20:25:52.098586 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.098593 [11383]: server/ctdb_server.c:331 ctdb request 2147483464 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:52.098603 [11383]: Destroying timer event 0x25b8630 "ctdb_control_timeout"
2012/10/08 20:25:52.098611 [11383]: Ending timer event 0x25b8ae0 "queue_next_trigger"
2012/10/08 20:25:52.098646 [11383]: server/ctdb_daemon.c:628 client request 2147483464 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098656 [11383]: Added timed event "queue_next_trigger": 0x25b8890
2012/10/08 20:25:52.098664 [11383]: Added timed event "ctdb_control_timeout": 0x25b89b0
2012/10/08 20:25:52.098671 [11383]: server/ctdb_server.c:331 ctdb request 2147483465 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098679 [11383]: Added timed event "queue_next_trigger": 0x25b8d00
2012/10/08 20:25:52.098686 [11383]: Ending timer event 0x25b8890 "queue_next_trigger"
2012/10/08 20:25:52.098693 [11383]: server/ctdb_server.c:331 ctdb request 2147483465 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.098703 [11383]: Destroying timer event 0x25b89b0 "ctdb_control_timeout"
2012/10/08 20:25:52.098710 [11383]: Ending timer event 0x25b8d00 "queue_next_trigger"
2012/10/08 20:25:52.098736 [recoverd:11454]: server/ctdb_recoverd.c:3029 Initial recovery master set - forcing election
2012/10/08 20:25:52.098752 [recoverd:11454]: server/ctdb_recoverd.c:2160 Force an election
2012/10/08 20:25:52.098780 [11383]: server/ctdb_daemon.c:628 client request 2147483465 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098791 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.098798 [11383]: Added timed event "ctdb_control_timeout": 0x25b8ce0
2012/10/08 20:25:52.098805 [11383]: server/ctdb_server.c:331 ctdb request 2147483466 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098813 [11383]: Freeze priority 1
2012/10/08 20:25:52.098820 [11383]: Added timed event "queue_next_trigger": 0x25b9090
2012/10/08 20:25:52.098827 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.098834 [11383]: server/ctdb_server.c:331 ctdb request 2147483466 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.098844 [11383]: Destroying timer event 0x25b8ce0 "ctdb_control_timeout"
2012/10/08 20:25:52.098851 [11383]: Ending timer event 0x25b9090 "queue_next_trigger"
2012/10/08 20:25:52.098882 [11383]: server/ctdb_daemon.c:628 client request 2147483466 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098893 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.098900 [11383]: Added timed event "ctdb_control_timeout": 0x25b9070
2012/10/08 20:25:52.098908 [11383]: server/ctdb_server.c:331 ctdb request 2147483467 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098915 [11383]: Freeze priority 2
2012/10/08 20:25:52.098922 [11383]: Added timed event "queue_next_trigger": 0x25b9420
2012/10/08 20:25:52.098929 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.098936 [11383]: server/ctdb_server.c:331 ctdb request 2147483467 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.098945 [11383]: Destroying timer event 0x25b9070 "ctdb_control_timeout"
2012/10/08 20:25:52.098952 [11383]: Ending timer event 0x25b9420 "queue_next_trigger"
2012/10/08 20:25:52.098983 [11383]: server/ctdb_daemon.c:628 client request 2147483467 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.098993 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.099000 [11383]: Added timed event "ctdb_control_timeout": 0x25b9400
2012/10/08 20:25:52.099008 [11383]: server/ctdb_server.c:331 ctdb request 2147483468 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099015 [11383]: Freeze priority 3
2012/10/08 20:25:52.099022 [11383]: Added timed event "queue_next_trigger": 0x25b97b0
2012/10/08 20:25:52.099029 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.099036 [11383]: server/ctdb_server.c:331 ctdb request 2147483468 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.099045 [11383]: Destroying timer event 0x25b9400 "ctdb_control_timeout"
2012/10/08 20:25:52.099052 [11383]: Ending timer event 0x25b97b0 "queue_next_trigger"
2012/10/08 20:25:52.099083 [11383]: server/ctdb_daemon.c:628 client request 2147483468 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099093 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.099105 [11383]: Added timed event "ctdb_control_timeout": 0x25b9790
2012/10/08 20:25:52.099113 [11383]: server/ctdb_server.c:331 ctdb request 2147483469 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099121 [11383]: Added timed event "ctdb_drop_all_ips_event": 0x25b99b0
2012/10/08 20:25:52.099129 [11383]: Added timed event "queue_next_trigger": 0x25b9bf0
2012/10/08 20:25:52.099135 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.099143 [11383]: server/ctdb_server.c:331 ctdb request 2147483469 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.099152 [11383]: Destroying timer event 0x25b9790 "ctdb_control_timeout"
2012/10/08 20:25:52.099159 [11383]: Ending timer event 0x25b9bf0 "queue_next_trigger"
2012/10/08 20:25:52.099190 [11383]: server/ctdb_daemon.c:628 client request 2147483469 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.099201 [11383]: Added timed event "queue_next_trigger": 0x25b9ad0
2012/10/08 20:25:52.099208 [11383]: Added timed event "ctdb_control_timeout": 0x25b9bf0
2012/10/08 20:25:52.099216 [11383]: server/ctdb_server.c:331 ctdb request 2147483470 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099225 [11383]: Added timed event "queue_next_trigger": 0x25ba040
2012/10/08 20:25:52.099232 [11383]: Ending timer event 0x25b9ad0 "queue_next_trigger"
2012/10/08 20:25:52.099239 [11383]: server/ctdb_server.c:331 ctdb request 2147483470 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:52.099249 [11383]: Destroying timer event 0x25b9bf0 "ctdb_control_timeout"
2012/10/08 20:25:52.099256 [11383]: Ending timer event 0x25ba040 "queue_next_trigger"
2012/10/08 20:25:52.099275 [recoverd:11454]: server/ctdb_recoverd.c:1814 Send election request to all active nodes
2012/10/08 20:25:52.099295 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 80 from node 1 to 4026531842
2012/10/08 20:25:52.099305 [11383]: Added timed event "queue_next_trigger": 0x25b4dd0
2012/10/08 20:25:52.099313 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 5 length 80 from node 1 to 1
2012/10/08 20:25:52.099324 [11383]: Ending timer event 0x25b4dd0 "queue_next_trigger"
2012/10/08 20:25:52.099350 [11383]: server/ctdb_daemon.c:628 client request 2147483470 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099360 [11383]: Added timed event "queue_next_trigger": 0x25b7e30
2012/10/08 20:25:52.099367 [11383]: Added timed event "ctdb_control_timeout": 0x25ba240
2012/10/08 20:25:52.099375 [11383]: server/ctdb_server.c:331 ctdb request 2147483471 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099383 [11383]: Added timed event "queue_next_trigger": 0x25ba480
2012/10/08 20:25:52.099390 [11383]: Ending timer event 0x25b7e30 "queue_next_trigger"
2012/10/08 20:25:52.099397 [11383]: server/ctdb_server.c:331 ctdb request 2147483471 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.099406 [11383]: Destroying timer event 0x25ba240 "ctdb_control_timeout"
2012/10/08 20:25:52.099414 [11383]: Ending timer event 0x25ba480 "queue_next_trigger"
2012/10/08 20:25:52.099451 [11383]: server/ctdb_daemon.c:628 client request 2147483471 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.099461 [11383]: Added timed event "queue_next_trigger": 0x25ba360
2012/10/08 20:25:52.099469 [11383]: Added timed event "ctdb_control_timeout": 0x25ba480
2012/10/08 20:25:52.099476 [11383]: server/ctdb_server.c:331 ctdb request 2147483472 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099485 [11383]: Added timed event "queue_next_trigger": 0x25ba8d0
2012/10/08 20:25:52.099492 [11383]: Ending timer event 0x25ba360 "queue_next_trigger"
2012/10/08 20:25:52.099499 [11383]: server/ctdb_server.c:331 ctdb request 2147483472 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:52.099515 [11383]: Destroying timer event 0x25ba480 "ctdb_control_timeout"
2012/10/08 20:25:52.099524 [11383]: Ending timer event 0x25ba8d0 "queue_next_trigger"
2012/10/08 20:25:52.099557 [11383]: server/ctdb_daemon.c:628 client request 2147483472 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099572 [11383]: Added timed event "queue_next_trigger": 0x25ba610
2012/10/08 20:25:52.099580 [11383]: Added timed event "ctdb_control_timeout": 0x25ba800
2012/10/08 20:25:52.099587 [11383]: server/ctdb_server.c:331 ctdb request 2147483473 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.099595 [11383]: Added timed event "queue_next_trigger": 0x25bab80
2012/10/08 20:25:52.099602 [11383]: Ending timer event 0x25ba610 "queue_next_trigger"
2012/10/08 20:25:52.099609 [11383]: server/ctdb_server.c:331 ctdb request 2147483473 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.099619 [11383]: Destroying timer event 0x25ba800 "ctdb_control_timeout"
2012/10/08 20:25:52.099626 [11383]: Ending timer event 0x25bab80 "queue_next_trigger"
2012/10/08 20:25:52.114773 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.115166 [11383]: SIGCHLD from 11455
2012/10/08 20:25:52.115374 [eventscript-01.reclock-setup:11464]: Executing event script /etc/ctdb/events.d/01.reclock setup
2012/10/08 20:25:52.122507 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.122868 [11383]: SIGCHLD from 11464
2012/10/08 20:25:52.123071 [eventscript-10.interface-setup:11468]: Executing event script /etc/ctdb/events.d/10.interface setup
2012/10/08 20:25:52.130099 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.130479 [11383]: SIGCHLD from 11468
2012/10/08 20:25:52.130684 [eventscript-11.natgw-setup:11472]: Executing event script /etc/ctdb/events.d/11.natgw setup
2012/10/08 20:25:52.137603 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.137976 [11383]: SIGCHLD from 11472
2012/10/08 20:25:52.138178 [eventscript-11.routing-setup:11476]: Executing event script /etc/ctdb/events.d/11.routing setup
2012/10/08 20:25:52.145056 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.145416 [11383]: SIGCHLD from 11476
2012/10/08 20:25:52.145620 [eventscript-13.per_ip_routing-setup:11480]: Executing event script /etc/ctdb/events.d/13.per_ip_routing setup
2012/10/08 20:25:52.154599 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.154955 [11383]: SIGCHLD from 11480
2012/10/08 20:25:52.155159 [eventscript-20.multipathd-setup:11485]: Executing event script /etc/ctdb/events.d/20.multipathd setup
2012/10/08 20:25:52.155563 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.156082 [eventscript-31.clamd-setup:11486]: Executing event script /etc/ctdb/events.d/31.clamd setup
2012/10/08 20:25:52.156503 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.156744 [11383]: SIGCHLD from 11485
2012/10/08 20:25:52.156943 [eventscript-40.vsftpd-setup:11487]: Executing event script /etc/ctdb/events.d/40.vsftpd setup
2012/10/08 20:25:52.161133 [11383]: SIGCHLD from 11486
2012/10/08 20:25:52.164017 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.164383 [11383]: SIGCHLD from 11487
2012/10/08 20:25:52.164592 [eventscript-40.fs_use-setup:11491]: Executing event script /etc/ctdb/events.d/40.fs_use setup
2012/10/08 20:25:52.165029 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.165574 [eventscript-41.httpd-setup:11492]: Executing event script /etc/ctdb/events.d/41.httpd setup
2012/10/08 20:25:52.169626 [11383]: SIGCHLD from 11491
2012/10/08 20:25:52.172549 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.172921 [11383]: SIGCHLD from 11492
2012/10/08 20:25:52.173137 [eventscript-50.samba-setup:11496]: Executing event script /etc/ctdb/events.d/50.samba setup
2012/10/08 20:25:52.182459 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.182832 [11383]: SIGCHLD from 11496
2012/10/08 20:25:52.183037 [eventscript-60.nfs-setup:11501]: Executing event script /etc/ctdb/events.d/60.nfs setup
2012/10/08 20:25:52.193923 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.194329 [11383]: SIGCHLD from 11501
2012/10/08 20:25:52.194534 [eventscript-60.ganesha-setup:11507]: Executing event script /etc/ctdb/events.d/60.ganesha setup
2012/10/08 20:25:52.194960 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.195484 [eventscript-62.cnfs-setup:11508]: Executing event script /etc/ctdb/events.d/62.cnfs setup
2012/10/08 20:25:52.199608 [11383]: SIGCHLD from 11507
2012/10/08 20:25:52.204329 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.204683 [11383]: SIGCHLD from 11508
2012/10/08 20:25:52.204887 [eventscript-70.iscsi-setup:11513]: Executing event script /etc/ctdb/events.d/70.iscsi setup
2012/10/08 20:25:52.211973 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:52.212347 [11383]: SIGCHLD from 11513
2012/10/08 20:25:52.212551 [eventscript-91.lvs-setup:11517]: Executing event script /etc/ctdb/events.d/91.lvs setup
2012/10/08 20:25:52.218953 [11383]: server/eventscript.c:485 Eventscript setup finished with state 0
2012/10/08 20:25:52.219034 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:25:52.219049 [11383]: Destroying timer event 0x25b20f0 "ctdb_event_script_timeout"
2012/10/08 20:25:52.219058 [11383]: server/ctdb_server.c:331 ctdb request 2147483474 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.219068 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:25:52.219215 [11383]: SIGCHLD from 11517
2012/10/08 20:25:52.936329 [11383]: Added timed event "ctdb_statistics_update": 0x25b20f0
2012/10/08 20:25:52.936346 [11383]: Ending timer event 0x25aa0c0 "ctdb_statistics_update"
2012/10/08 20:25:52.938303 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:17 to connected child
2012/10/08 20:25:52.938442 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:25:52.938459 [11383]: Added timed event "queue_next_trigger": 0x25b6390
2012/10/08 20:25:52.938469 [11383]: Added timed event "ctdb_control_timeout": 0x25b64b0
2012/10/08 20:25:52.938480 [11383]: server/ctdb_server.c:331 ctdb request 2147483476 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.938491 [11383]: Added timed event "queue_next_trigger": 0x25b5c20
2012/10/08 20:25:52.938501 [11383]: Ending timer event 0x25b6390 "queue_next_trigger"
2012/10/08 20:25:52.938510 [11383]: server/ctdb_server.c:331 ctdb request 2147483476 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.938525 [11383]: Destroying timer event 0x25b64b0 "ctdb_control_timeout"
2012/10/08 20:25:52.938535 [11383]: Ending timer event 0x25b5c20 "queue_next_trigger"
2012/10/08 20:25:52.938596 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:18 to connected child
2012/10/08 20:25:52.938653 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:52.938668 [11383]: Added timed event "queue_next_trigger": 0x25b6b90
2012/10/08 20:25:52.938678 [11383]: Added timed event "ctdb_control_timeout": 0x25b6cb0
2012/10/08 20:25:52.938688 [11383]: server/ctdb_server.c:331 ctdb request 2147483478 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.938698 [11383]: Added timed event "queue_next_trigger": 0x25b60d0
2012/10/08 20:25:52.938708 [11383]: Ending timer event 0x25b6b90 "queue_next_trigger"
2012/10/08 20:25:52.938717 [11383]: server/ctdb_server.c:331 ctdb request 2147483478 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.938730 [11383]: Destroying timer event 0x25b6cb0 "ctdb_control_timeout"
2012/10/08 20:25:52.938739 [11383]: Ending timer event 0x25b60d0 "queue_next_trigger"
2012/10/08 20:25:52.938790 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.938814 [11383]: Added timed event "queue_next_trigger": 0x25b6850
2012/10/08 20:25:52.938824 [11383]: Added timed event "ctdb_control_timeout": 0x25b57e0
2012/10/08 20:25:52.938834 [11383]: server/ctdb_server.c:331 ctdb request 2147483479 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:52.938845 [11383]: Added timed event "queue_next_trigger": 0x25b5a20
2012/10/08 20:25:52.938855 [11383]: Ending timer event 0x25b6850 "queue_next_trigger"
2012/10/08 20:25:52.938864 [11383]: server/ctdb_server.c:331 ctdb request 2147483479 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:52.938877 [11383]: Destroying timer event 0x25b57e0 "ctdb_control_timeout"
2012/10/08 20:25:52.938887 [11383]: Ending timer event 0x25b5a20 "queue_next_trigger"
2012/10/08 20:25:53.093255 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:53.093286 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b57e0
2012/10/08 20:25:53.093296 [11383]: Ending timer event 0x25ade00 "ctdb_wait_until_recovered"
2012/10/08 20:25:53.093308 [11383]: Added timed event "ctdb_time_tick": 0x25b6f20
2012/10/08 20:25:53.093317 [11383]: Ending timer event 0x25ae300 "ctdb_time_tick"
2012/10/08 20:25:53.097379 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:17
2012/10/08 20:25:53.097425 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25ade00
2012/10/08 20:25:53.097438 [11383]: Ending timer event 0x25b21a0 "ctdb_tcp_node_connect"
2012/10/08 20:25:53.097734 [11383]: Destroying timer event 0x25ade00 "ctdb_tcp_node_connect"
2012/10/08 20:25:53.097759 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25ade00
2012/10/08 20:25:53.936825 [11383]: Added timed event "ctdb_statistics_update": 0x25b64b0
2012/10/08 20:25:53.936842 [11383]: Ending timer event 0x25b20f0 "ctdb_statistics_update"
2012/10/08 20:25:54.093894 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:54.093912 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b67b0
2012/10/08 20:25:54.093922 [11383]: Ending timer event 0x25b57e0 "ctdb_wait_until_recovered"
2012/10/08 20:25:54.093932 [11383]: Added timed event "ctdb_time_tick": 0x25b57e0
2012/10/08 20:25:54.093941 [11383]: Ending timer event 0x25b6f20 "ctdb_time_tick"
2012/10/08 20:25:54.097996 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:17
2012/10/08 20:25:54.098030 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b9840
2012/10/08 20:25:54.098041 [11383]: Ending timer event 0x25ade00 "ctdb_tcp_node_connect"
2012/10/08 20:25:54.098325 [11383]: Destroying timer event 0x25b9840 "ctdb_tcp_node_connect"
2012/10/08 20:25:54.098347 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b9840
2012/10/08 20:25:54.937407 [11383]: Added timed event "ctdb_statistics_update": 0x25b6020
2012/10/08 20:25:54.937423 [11383]: Ending timer event 0x25b64b0 "ctdb_statistics_update"
2012/10/08 20:25:55.094473 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:55.094489 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b64b0
2012/10/08 20:25:55.094498 [11383]: Ending timer event 0x25b67b0 "ctdb_wait_until_recovered"
2012/10/08 20:25:55.094508 [11383]: Added timed event "ctdb_time_tick": 0x25b67b0
2012/10/08 20:25:55.094516 [11383]: Ending timer event 0x25b57e0 "ctdb_time_tick"
2012/10/08 20:25:55.098570 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:17
2012/10/08 20:25:55.098607 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b53c0
2012/10/08 20:25:55.098618 [11383]: Ending timer event 0x25b9840 "ctdb_tcp_node_connect"
2012/10/08 20:25:55.098894 [11383]: Destroying timer event 0x25b53c0 "ctdb_tcp_node_connect"
2012/10/08 20:25:55.098916 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b53c0
2012/10/08 20:25:55.102695 [recoverd:11454]: server/ctdb_recoverd.c:913 Election timed out
2012/10/08 20:25:55.102758 [11383]: server/ctdb_daemon.c:628 client request 2147483473 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.102777 [11383]: Added timed event "queue_next_trigger": 0x25b1ce0
2012/10/08 20:25:55.102814 [11383]: Added timed event "ctdb_control_timeout": 0x25b1e00
2012/10/08 20:25:55.102827 [11383]: server/ctdb_server.c:331 ctdb request 2147483480 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.102839 [11383]: Destroying timer event 0x25ae1e0 "ctdb_recd_ping_timeout"
2012/10/08 20:25:55.102849 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:25:55.102859 [11383]: Added timed event "queue_next_trigger": 0x25b5040
2012/10/08 20:25:55.102868 [11383]: Ending timer event 0x25b1ce0 "queue_next_trigger"
2012/10/08 20:25:55.102878 [11383]: server/ctdb_server.c:331 ctdb request 2147483480 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.102893 [11383]: Destroying timer event 0x25b1e00 "ctdb_control_timeout"
2012/10/08 20:25:55.102902 [11383]: Ending timer event 0x25b5040 "queue_next_trigger"
2012/10/08 20:25:55.102944 [11383]: server/ctdb_daemon.c:628 client request 2147483474 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.102959 [11383]: Added timed event "queue_next_trigger": 0x25b1cb0
2012/10/08 20:25:55.102969 [11383]: Added timed event "ctdb_control_timeout": 0x25b4f90
2012/10/08 20:25:55.102979 [11383]: server/ctdb_server.c:331 ctdb request 2147483481 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.102990 [11383]: Added timed event "queue_next_trigger": 0x25b2210
2012/10/08 20:25:55.103000 [11383]: Ending timer event 0x25b1cb0 "queue_next_trigger"
2012/10/08 20:25:55.103010 [11383]: server/ctdb_server.c:331 ctdb request 2147483481 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.103022 [11383]: Destroying timer event 0x25b4f90 "ctdb_control_timeout"
2012/10/08 20:25:55.103032 [11383]: Ending timer event 0x25b2210 "queue_next_trigger"
2012/10/08 20:25:55.103072 [11383]: server/ctdb_daemon.c:628 client request 2147483475 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.103086 [11383]: Added timed event "queue_next_trigger": 0x25b1cb0
2012/10/08 20:25:55.103096 [11383]: Added timed event "ctdb_control_timeout": 0x25b21d0
2012/10/08 20:25:55.103106 [11383]: server/ctdb_server.c:331 ctdb request 2147483482 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103118 [11383]: Added timed event "queue_next_trigger": 0x25b2690
2012/10/08 20:25:55.103127 [11383]: Ending timer event 0x25b1cb0 "queue_next_trigger"
2012/10/08 20:25:55.103137 [11383]: server/ctdb_server.c:331 ctdb request 2147483482 of type 8 length 228 from node 1 to 1
2012/10/08 20:25:55.103150 [11383]: Destroying timer event 0x25b21d0 "ctdb_control_timeout"
2012/10/08 20:25:55.103159 [11383]: Ending timer event 0x25b2690 "queue_next_trigger"
2012/10/08 20:25:55.103200 [11383]: server/ctdb_daemon.c:628 client request 2147483476 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.103214 [11383]: Added timed event "queue_next_trigger": 0x25b1cb0
2012/10/08 20:25:55.103224 [11383]: Added timed event "ctdb_control_timeout": 0x25b2900
2012/10/08 20:25:55.103234 [11383]: server/ctdb_server.c:331 ctdb request 2147483483 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103245 [11383]: Added timed event "queue_next_trigger": 0x25b24b0
2012/10/08 20:25:55.103255 [11383]: Ending timer event 0x25b1cb0 "queue_next_trigger"
2012/10/08 20:25:55.103264 [11383]: server/ctdb_server.c:331 ctdb request 2147483483 of type 8 length 67 from node 1 to 1
2012/10/08 20:25:55.103277 [11383]: Destroying timer event 0x25b2900 "ctdb_control_timeout"
2012/10/08 20:25:55.103287 [11383]: Ending timer event 0x25b24b0 "queue_next_trigger"
2012/10/08 20:25:55.103327 [11383]: server/ctdb_daemon.c:628 client request 2147483477 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.103341 [11383]: Added timed event "queue_next_trigger": 0x25b2410
2012/10/08 20:25:55.103351 [11383]: Added timed event "ctdb_control_timeout": 0x25b1c10
2012/10/08 20:25:55.103361 [11383]: server/ctdb_server.c:331 ctdb request 2147483484 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103372 [11383]: Added timed event "queue_next_trigger": 0x25b72d0
2012/10/08 20:25:55.103400 [11383]: Ending timer event 0x25b2410 "queue_next_trigger"
2012/10/08 20:25:55.103411 [11383]: server/ctdb_server.c:331 ctdb request 2147483484 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.103425 [11383]: Destroying timer event 0x25b1c10 "ctdb_control_timeout"
2012/10/08 20:25:55.103434 [11383]: Ending timer event 0x25b72d0 "queue_next_trigger"
2012/10/08 20:25:55.103474 [11383]: server/ctdb_daemon.c:628 client request 2147483478 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103488 [11383]: Added timed event "queue_next_trigger": 0x25b2640
2012/10/08 20:25:55.103498 [11383]: Added timed event "ctdb_control_timeout": 0x25b7290
2012/10/08 20:25:55.103508 [11383]: server/ctdb_server.c:331 ctdb request 2147483485 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103519 [11383]: Added timed event "queue_next_trigger": 0x25b7680
2012/10/08 20:25:55.103529 [11383]: Ending timer event 0x25b2640 "queue_next_trigger"
2012/10/08 20:25:55.103538 [11383]: server/ctdb_server.c:331 ctdb request 2147483485 of type 8 length 60 from node 1 to 1
2012/10/08 20:25:55.103551 [11383]: Destroying timer event 0x25b7290 "ctdb_control_timeout"
2012/10/08 20:25:55.103561 [11383]: Ending timer event 0x25b7680 "queue_next_trigger"
2012/10/08 20:25:55.103601 [11383]: server/ctdb_daemon.c:628 client request 2147483479 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103615 [11383]: Added timed event "queue_next_trigger": 0x25b7290
2012/10/08 20:25:55.103625 [11383]: Added timed event "ctdb_control_timeout": 0x25b75e0
2012/10/08 20:25:55.103635 [11383]: server/ctdb_server.c:331 ctdb request 2147483486 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103647 [11383]: Added timed event "queue_next_trigger": 0x25b79b0
2012/10/08 20:25:55.103656 [11383]: Ending timer event 0x25b7290 "queue_next_trigger"
2012/10/08 20:25:55.103666 [11383]: server/ctdb_server.c:331 ctdb request 2147483486 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:55.103679 [11383]: Destroying timer event 0x25b75e0 "ctdb_control_timeout"
2012/10/08 20:25:55.103688 [11383]: Ending timer event 0x25b79b0 "queue_next_trigger"
2012/10/08 20:25:55.103729 [11383]: server/ctdb_daemon.c:628 client request 2147483480 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103743 [11383]: Added timed event "queue_next_trigger": 0x25b7290
2012/10/08 20:25:55.103753 [11383]: Added timed event "ctdb_control_timeout": 0x25b7880
2012/10/08 20:25:55.103762 [11383]: server/ctdb_server.c:331 ctdb request 2147483487 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103773 [11383]: Added timed event "queue_next_trigger": 0x25b7bd0
2012/10/08 20:25:55.103783 [11383]: Ending timer event 0x25b7290 "queue_next_trigger"
2012/10/08 20:25:55.103792 [11383]: server/ctdb_server.c:331 ctdb request 2147483487 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.103805 [11383]: Destroying timer event 0x25b7880 "ctdb_control_timeout"
2012/10/08 20:25:55.103815 [11383]: Ending timer event 0x25b7bd0 "queue_next_trigger"
2012/10/08 20:25:55.103854 [11383]: server/ctdb_daemon.c:628 client request 2147483481 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103868 [11383]: Added timed event "queue_next_trigger": 0x25b7290
2012/10/08 20:25:55.103877 [11383]: Added timed event "ctdb_control_timeout": 0x25b7bb0
2012/10/08 20:25:55.103887 [11383]: server/ctdb_server.c:331 ctdb request 2147483488 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.103912 [11383]: Added timed event "queue_next_trigger": 0x25b8020
2012/10/08 20:25:55.103924 [11383]: Ending timer event 0x25b7290 "queue_next_trigger"
2012/10/08 20:25:55.103934 [11383]: server/ctdb_server.c:331 ctdb request 2147483488 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:55.103947 [11383]: Destroying timer event 0x25b7bb0 "ctdb_control_timeout"
2012/10/08 20:25:55.103956 [11383]: Ending timer event 0x25b8020 "queue_next_trigger"
2012/10/08 20:25:55.103997 [11383]: server/ctdb_daemon.c:628 client request 2147483482 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.104029 [11383]: Added timed event "queue_next_trigger": 0x25b7dd0
2012/10/08 20:25:55.104040 [11383]: Added timed event "ctdb_control_timeout": 0x25b7ef0
2012/10/08 20:25:55.104050 [11383]: server/ctdb_server.c:331 ctdb request 2147483489 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104062 [11383]: Added timed event "queue_next_trigger": 0x25b8350
2012/10/08 20:25:55.104072 [11383]: Ending timer event 0x25b7dd0 "queue_next_trigger"
2012/10/08 20:25:55.104081 [11383]: server/ctdb_server.c:331 ctdb request 2147483489 of type 8 length 108 from node 1 to 1
2012/10/08 20:25:55.104095 [11383]: Destroying timer event 0x25b7ef0 "ctdb_control_timeout"
2012/10/08 20:25:55.104104 [11383]: Ending timer event 0x25b8350 "queue_next_trigger"
2012/10/08 20:25:55.104145 [11383]: server/ctdb_daemon.c:628 client request 2147483483 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.104159 [11383]: Added timed event "queue_next_trigger": 0x25b7290
2012/10/08 20:25:55.104169 [11383]: Added timed event "ctdb_control_timeout": 0x25b8220
2012/10/08 20:25:55.104178 [11383]: server/ctdb_server.c:331 ctdb request 2147483490 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104191 [11383]: Added timed event "queue_next_trigger": 0x25b8580
2012/10/08 20:25:55.104200 [11383]: Ending timer event 0x25b7290 "queue_next_trigger"
2012/10/08 20:25:55.104209 [11383]: server/ctdb_server.c:331 ctdb request 2147483490 of type 8 length 72 from node 1 to 1
2012/10/08 20:25:55.104222 [11383]: Destroying timer event 0x25b8220 "ctdb_control_timeout"
2012/10/08 20:25:55.104232 [11383]: Ending timer event 0x25b8580 "queue_next_trigger"
2012/10/08 20:25:55.104257 [recoverd:11454]: The interfaces status has changed on local node 1 - force takeover run
2012/10/08 20:25:55.104285 [11383]: server/ctdb_daemon.c:628 client request 2147483484 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.104299 [11383]: Added timed event "queue_next_trigger": 0x25b7290
2012/10/08 20:25:55.104308 [11383]: Added timed event "ctdb_control_timeout": 0x25b8550
2012/10/08 20:25:55.104318 [11383]: server/ctdb_server.c:331 ctdb request 2147483491 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104330 [11383]: Added timed event "queue_next_trigger": 0x25b88e0
2012/10/08 20:25:55.104339 [11383]: Ending timer event 0x25b7290 "queue_next_trigger"
2012/10/08 20:25:55.104349 [11383]: server/ctdb_server.c:331 ctdb request 2147483491 of type 8 length 80 from node 1 to 1
2012/10/08 20:25:55.104362 [11383]: Destroying timer event 0x25b8550 "ctdb_control_timeout"
2012/10/08 20:25:55.104371 [11383]: Ending timer event 0x25b88e0 "queue_next_trigger"
2012/10/08 20:25:55.104411 [11383]: server/ctdb_daemon.c:628 client request 2147483485 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.104425 [11383]: Added timed event "queue_next_trigger": 0x25b8780
2012/10/08 20:25:55.104435 [11383]: Added timed event "ctdb_control_timeout": 0x25b88a0
2012/10/08 20:25:55.104445 [11383]: server/ctdb_server.c:331 ctdb request 2147483492 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104456 [11383]: Added timed event "queue_next_trigger": 0x25b8d20
2012/10/08 20:25:55.104466 [11383]: Ending timer event 0x25b8780 "queue_next_trigger"
2012/10/08 20:25:55.104476 [11383]: server/ctdb_server.c:331 ctdb request 2147483492 of type 8 length 108 from node 1 to 1
2012/10/08 20:25:55.104488 [11383]: Destroying timer event 0x25b88a0 "ctdb_control_timeout"
2012/10/08 20:25:55.104498 [11383]: Ending timer event 0x25b8d20 "queue_next_trigger"
2012/10/08 20:25:55.104523 [recoverd:11454]: server/ctdb_recoverd.c:2581 in the middle of recovery or ip reallocation. skipping public ip address check
2012/10/08 20:25:55.104552 [11383]: server/ctdb_daemon.c:628 client request 2147483486 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104566 [11383]: Added timed event "queue_next_trigger": 0x25b8ae0
2012/10/08 20:25:55.104576 [11383]: Added timed event "ctdb_control_timeout": 0x25b8c00
2012/10/08 20:25:55.104586 [11383]: server/ctdb_server.c:331 ctdb request 2147483493 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104616 [11383]: Added timed event "queue_next_trigger": 0x25b8ed0
2012/10/08 20:25:55.104628 [11383]: Ending timer event 0x25b8ae0 "queue_next_trigger"
2012/10/08 20:25:55.104638 [11383]: server/ctdb_server.c:331 ctdb request 2147483493 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.104653 [11383]: Destroying timer event 0x25b8c00 "ctdb_control_timeout"
2012/10/08 20:25:55.104663 [11383]: Ending timer event 0x25b8ed0 "queue_next_trigger"
2012/10/08 20:25:55.104706 [11383]: server/ctdb_daemon.c:628 client request 2147483487 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104720 [11383]: Added timed event "queue_next_trigger": 0x25b88a0
2012/10/08 20:25:55.104729 [11383]: Added timed event "ctdb_control_timeout": 0x25b8eb0
2012/10/08 20:25:55.104739 [11383]: server/ctdb_server.c:331 ctdb request 2147483494 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.104750 [11383]: Added timed event "queue_next_trigger": 0x25b91f0
2012/10/08 20:25:55.104760 [11383]: Ending timer event 0x25b88a0 "queue_next_trigger"
2012/10/08 20:25:55.104769 [11383]: server/ctdb_server.c:331 ctdb request 2147483494 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.104782 [11383]: Destroying timer event 0x25b8eb0 "ctdb_control_timeout"
2012/10/08 20:25:55.104792 [11383]: Ending timer event 0x25b91f0 "queue_next_trigger"
2012/10/08 20:25:55.104822 [recoverd:11454]: server/ctdb_recoverd.c:2344 Node:1 was in recovery mode. Restart recovery process
2012/10/08 20:25:55.104843 [recoverd:11454]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:25:55.104854 [recoverd:11454]: Taking out recovery lock from recovery daemon
2012/10/08 20:25:55.104872 [recoverd:11454]: Take the recovery lock
2012/10/08 20:25:55.110531 [recoverd:11454]: Recovery lock taken successfully
2012/10/08 20:25:55.110546 [recoverd:11454]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:25:55.110575 [11383]: server/ctdb_daemon.c:628 client request 2147483488 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:25:55.110590 [11383]: Added timed event "queue_next_trigger": 0x25b90d0
2012/10/08 20:25:55.110600 [11383]: Added timed event "ctdb_control_timeout": 0x25b91f0
2012/10/08 20:25:55.110611 [11383]: server/ctdb_server.c:331 ctdb request 2147483495 of type 7 length 68 from node 1 to 1
2012/10/08 20:25:55.110622 [11383]: Added timed event "queue_next_trigger": 0x25b9510
2012/10/08 20:25:55.110632 [11383]: Ending timer event 0x25b90d0 "queue_next_trigger"
2012/10/08 20:25:55.110642 [11383]: server/ctdb_server.c:331 ctdb request 2147483495 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.110655 [11383]: Destroying timer event 0x25b91f0 "ctdb_control_timeout"
2012/10/08 20:25:55.110665 [11383]: Ending timer event 0x25b9510 "queue_next_trigger"
2012/10/08 20:25:55.110690 [recoverd:11454]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:25:55.110701 [recoverd:11454]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:25:55.110727 [11383]: server/ctdb_daemon.c:628 client request 2147483489 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.110741 [11383]: Added timed event "queue_next_trigger": 0x25b96a0
2012/10/08 20:25:55.110751 [11383]: Added timed event "ctdb_control_timeout": 0x25b93f0
2012/10/08 20:25:55.110761 [11383]: server/ctdb_server.c:331 ctdb request 2147483496 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.110772 [11383]: Added timed event "queue_next_trigger": 0x25b9b80
2012/10/08 20:25:55.110781 [11383]: Ending timer event 0x25b96a0 "queue_next_trigger"
2012/10/08 20:25:55.110791 [11383]: server/ctdb_server.c:331 ctdb request 2147483496 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.110804 [11383]: Destroying timer event 0x25b93f0 "ctdb_control_timeout"
2012/10/08 20:25:55.110813 [11383]: Ending timer event 0x25b9b80 "queue_next_trigger"
2012/10/08 20:25:55.110846 [recoverd:11454]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:25:55.110871 [recoverd:11454]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:25:55.110901 [11383]: server/ctdb_daemon.c:628 client request 2147483490 of type 7 length 88 from node 1 to 1
2012/10/08 20:25:55.110915 [11383]: Added timed event "queue_next_trigger": 0x25b9a60
2012/10/08 20:25:55.110924 [11383]: Added timed event "ctdb_control_timeout": 0x25b9b80
2012/10/08 20:25:55.110934 [11383]: server/ctdb_server.c:331 ctdb request 2147483497 of type 7 length 83 from node 1 to 1
2012/10/08 20:25:55.110945 [11383]: Added timed event "queue_next_trigger": 0x25b9ea0
2012/10/08 20:25:55.110955 [11383]: Ending timer event 0x25b9a60 "queue_next_trigger"
2012/10/08 20:25:55.110964 [11383]: server/ctdb_server.c:331 ctdb request 2147483497 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.110991 [11383]: Destroying timer event 0x25b9b80 "ctdb_control_timeout"
2012/10/08 20:25:55.111002 [11383]: Ending timer event 0x25b9ea0 "queue_next_trigger"
2012/10/08 20:25:55.111045 [11383]: server/ctdb_daemon.c:628 client request 2147483491 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111059 [11383]: Added timed event "queue_next_trigger": 0x25b9d80
2012/10/08 20:25:55.111069 [11383]: Added timed event "ctdb_control_timeout": 0x25b9ea0
2012/10/08 20:25:55.111079 [11383]: server/ctdb_server.c:331 ctdb request 2147483498 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111089 [11383]: Freeze priority 1
2012/10/08 20:25:55.111099 [11383]: Added timed event "queue_next_trigger": 0x25ba230
2012/10/08 20:25:55.111108 [11383]: Ending timer event 0x25b9d80 "queue_next_trigger"
2012/10/08 20:25:55.111118 [11383]: server/ctdb_server.c:331 ctdb request 2147483498 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.111131 [11383]: Destroying timer event 0x25b9ea0 "ctdb_control_timeout"
2012/10/08 20:25:55.111140 [11383]: Ending timer event 0x25ba230 "queue_next_trigger"
2012/10/08 20:25:55.111181 [11383]: server/ctdb_daemon.c:628 client request 2147483492 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111195 [11383]: Added timed event "queue_next_trigger": 0x25b96a0
2012/10/08 20:25:55.111205 [11383]: Added timed event "ctdb_control_timeout": 0x25ba210
2012/10/08 20:25:55.111215 [11383]: server/ctdb_server.c:331 ctdb request 2147483499 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111225 [11383]: Freeze priority 2
2012/10/08 20:25:55.111234 [11383]: Added timed event "queue_next_trigger": 0x25ba5c0
2012/10/08 20:25:55.111243 [11383]: Ending timer event 0x25b96a0 "queue_next_trigger"
2012/10/08 20:25:55.111253 [11383]: server/ctdb_server.c:331 ctdb request 2147483499 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.111265 [11383]: Destroying timer event 0x25ba210 "ctdb_control_timeout"
2012/10/08 20:25:55.111275 [11383]: Ending timer event 0x25ba5c0 "queue_next_trigger"
2012/10/08 20:25:55.111315 [11383]: server/ctdb_daemon.c:628 client request 2147483493 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111329 [11383]: Added timed event "queue_next_trigger": 0x25b96a0
2012/10/08 20:25:55.111339 [11383]: Added timed event "ctdb_control_timeout": 0x25ba5a0
2012/10/08 20:25:55.111349 [11383]: server/ctdb_server.c:331 ctdb request 2147483500 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111358 [11383]: Freeze priority 3
2012/10/08 20:25:55.111368 [11383]: Added timed event "queue_next_trigger": 0x25ba950
2012/10/08 20:25:55.111377 [11383]: Ending timer event 0x25b96a0 "queue_next_trigger"
2012/10/08 20:25:55.111386 [11383]: server/ctdb_server.c:331 ctdb request 2147483500 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.111399 [11383]: Destroying timer event 0x25ba5a0 "ctdb_control_timeout"
2012/10/08 20:25:55.111409 [11383]: Ending timer event 0x25ba950 "queue_next_trigger"
2012/10/08 20:25:55.111449 [11383]: server/ctdb_daemon.c:628 client request 2147483494 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111463 [11383]: Added timed event "queue_next_trigger": 0x25b96a0
2012/10/08 20:25:55.111473 [11383]: Added timed event "ctdb_control_timeout": 0x25ba930
2012/10/08 20:25:55.111500 [11383]: server/ctdb_server.c:331 ctdb request 2147483501 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111512 [11383]: Destroying timer event 0x25b99b0 "ctdb_drop_all_ips_event"
2012/10/08 20:25:55.111522 [11383]: Added timed event "ctdb_drop_all_ips_event": 0x25b9960
2012/10/08 20:25:55.111532 [11383]: Added timed event "queue_next_trigger": 0x25bad50
2012/10/08 20:25:55.111541 [11383]: Ending timer event 0x25b96a0 "queue_next_trigger"
2012/10/08 20:25:55.111551 [11383]: server/ctdb_server.c:331 ctdb request 2147483501 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.111564 [11383]: Destroying timer event 0x25ba930 "ctdb_control_timeout"
2012/10/08 20:25:55.111573 [11383]: Ending timer event 0x25bad50 "queue_next_trigger"
2012/10/08 20:25:55.111615 [11383]: server/ctdb_daemon.c:628 client request 2147483495 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111630 [11383]: Added timed event "queue_next_trigger": 0x25bac30
2012/10/08 20:25:55.111639 [11383]: Added timed event "ctdb_control_timeout": 0x25bad50
2012/10/08 20:25:55.111649 [11383]: server/ctdb_server.c:331 ctdb request 2147483502 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.111672 [11383]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:25:55.111682 [11383]: Monitoring has been disabled
2012/10/08 20:25:55.111694 [11383]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/08 20:25:55.111802 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:25:55.111839 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:25:55.111854 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:25:55.111873 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:25:55.112499 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.112609 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ada10
2012/10/08 20:25:55.112637 [11383]: Ending timer event 0x25bac30 "queue_next_trigger"
2012/10/08 20:25:55.113072 [eventscript-00.ctdb-startrecovery:11530]: Executing event script /etc/ctdb/events.d/00.ctdb startrecovery
2012/10/08 20:25:55.125381 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.125882 [11383]: SIGCHLD from 11530
2012/10/08 20:25:55.126163 [eventscript-01.reclock-startrecovery:11535]: Executing event script /etc/ctdb/events.d/01.reclock startrecovery
2012/10/08 20:25:55.135252 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.135736 [11383]: SIGCHLD from 11535
2012/10/08 20:25:55.136007 [eventscript-10.interface-startrecovery:11539]: Executing event script /etc/ctdb/events.d/10.interface startrecovery
2012/10/08 20:25:55.145248 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.145733 [11383]: SIGCHLD from 11539
2012/10/08 20:25:55.146013 [eventscript-11.natgw-startrecovery:11543]: Executing event script /etc/ctdb/events.d/11.natgw startrecovery
2012/10/08 20:25:55.155063 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.155542 [11383]: SIGCHLD from 11543
2012/10/08 20:25:55.155820 [eventscript-11.routing-startrecovery:11547]: Executing event script /etc/ctdb/events.d/11.routing startrecovery
2012/10/08 20:25:55.164656 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.165146 [11383]: SIGCHLD from 11547
2012/10/08 20:25:55.165423 [eventscript-13.per_ip_routing-startrecovery:11551]: Executing event script /etc/ctdb/events.d/13.per_ip_routing startrecovery
2012/10/08 20:25:55.177439 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.177929 [11383]: SIGCHLD from 11551
2012/10/08 20:25:55.178203 [eventscript-20.multipathd-startrecovery:11556]: Executing event script /etc/ctdb/events.d/20.multipathd startrecovery
2012/10/08 20:25:55.178800 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.179492 [eventscript-31.clamd-startrecovery:11557]: Executing event script /etc/ctdb/events.d/31.clamd startrecovery
2012/10/08 20:25:55.180057 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.180376 [11383]: SIGCHLD from 11556
2012/10/08 20:25:55.180642 [eventscript-40.vsftpd-startrecovery:11558]: Executing event script /etc/ctdb/events.d/40.vsftpd startrecovery
2012/10/08 20:25:55.186210 [11383]: SIGCHLD from 11557
2012/10/08 20:25:55.190080 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.190597 [11383]: SIGCHLD from 11558
2012/10/08 20:25:55.190889 [eventscript-40.fs_use-startrecovery:11562]: Executing event script /etc/ctdb/events.d/40.fs_use startrecovery
2012/10/08 20:25:55.191438 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.192120 [eventscript-41.httpd-startrecovery:11563]: Executing event script /etc/ctdb/events.d/41.httpd startrecovery
2012/10/08 20:25:55.194130 [11383]: SIGCHLD from 11562
2012/10/08 20:25:55.201280 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.201760 [11383]: SIGCHLD from 11563
2012/10/08 20:25:55.202044 [eventscript-50.samba-startrecovery:11567]: Executing event script /etc/ctdb/events.d/50.samba startrecovery
2012/10/08 20:25:55.214458 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.214934 [11383]: SIGCHLD from 11567
2012/10/08 20:25:55.215207 [eventscript-60.nfs-startrecovery:11572]: Executing event script /etc/ctdb/events.d/60.nfs startrecovery
2012/10/08 20:25:55.229620 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.230099 [11383]: SIGCHLD from 11572
2012/10/08 20:25:55.230431 [eventscript-60.ganesha-startrecovery:11578]: Executing event script /etc/ctdb/events.d/60.ganesha startrecovery
2012/10/08 20:25:55.231013 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.231696 [eventscript-62.cnfs-startrecovery:11579]: Executing event script /etc/ctdb/events.d/62.cnfs startrecovery
2012/10/08 20:25:55.233721 [11383]: SIGCHLD from 11578
2012/10/08 20:25:55.243411 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.243901 [11383]: SIGCHLD from 11579
2012/10/08 20:25:55.244184 [eventscript-70.iscsi-startrecovery:11584]: Executing event script /etc/ctdb/events.d/70.iscsi startrecovery
2012/10/08 20:25:55.252940 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:25:55.253418 [11383]: SIGCHLD from 11584
2012/10/08 20:25:55.253693 [eventscript-91.lvs-startrecovery:11588]: Executing event script /etc/ctdb/events.d/91.lvs startrecovery
2012/10/08 20:25:55.262122 [11383]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/08 20:25:55.262163 [11383]: Added timed event "queue_next_trigger": 0x25aa090
2012/10/08 20:25:55.262180 [11383]: Destroying timer event 0x25ada10 "ctdb_event_script_timeout"
2012/10/08 20:25:55.262208 [11383]: server/ctdb_server.c:331 ctdb request 2147483502 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.262288 [11383]: Destroying timer event 0x25bad50 "ctdb_control_timeout"
2012/10/08 20:25:55.262301 [11383]: Ending timer event 0x25aa090 "queue_next_trigger"
2012/10/08 20:25:55.262366 [11383]: server/ctdb_daemon.c:628 client request 2147483496 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.262381 [11383]: Added timed event "queue_next_trigger": 0x25ada10
2012/10/08 20:25:55.262392 [11383]: Added timed event "ctdb_control_timeout": 0x25b5f40
2012/10/08 20:25:55.262410 [11383]: server/ctdb_server.c:331 ctdb request 2147483503 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.262423 [11383]: Added timed event "queue_next_trigger": 0x25b42b0
2012/10/08 20:25:55.262432 [11383]: Ending timer event 0x25ada10 "queue_next_trigger"
2012/10/08 20:25:55.262442 [11383]: server/ctdb_server.c:331 ctdb request 2147483503 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.262455 [11383]: Destroying timer event 0x25b5f40 "ctdb_control_timeout"
2012/10/08 20:25:55.262465 [11383]: Ending timer event 0x25b42b0 "queue_next_trigger"
2012/10/08 20:25:55.262505 [11383]: server/ctdb_daemon.c:628 client request 2147483497 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.262519 [11383]: Added timed event "queue_next_trigger": 0x25aa090
2012/10/08 20:25:55.262529 [11383]: Added timed event "ctdb_control_timeout": 0x25b42b0
2012/10/08 20:25:55.262539 [11383]: server/ctdb_server.c:331 ctdb request 2147483504 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.262552 [11383]: Added timed event "queue_next_trigger": 0x25b6860
2012/10/08 20:25:55.262562 [11383]: Ending timer event 0x25aa090 "queue_next_trigger"
2012/10/08 20:25:55.262572 [11383]: server/ctdb_server.c:331 ctdb request 2147483504 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:55.262585 [11383]: Destroying timer event 0x25b42b0 "ctdb_control_timeout"
2012/10/08 20:25:55.262594 [11383]: Ending timer event 0x25b6860 "queue_next_trigger"
2012/10/08 20:25:55.262636 [11383]: server/ctdb_daemon.c:628 client request 2147483498 of type 7 length 72 from node 1 to 1
2012/10/08 20:25:55.262651 [11383]: Added timed event "queue_next_trigger": 0x25b42b0
2012/10/08 20:25:55.262660 [11383]: Added timed event "ctdb_control_timeout": 0x25ae300
2012/10/08 20:25:55.262670 [11383]: server/ctdb_server.c:331 ctdb request 2147483505 of type 7 length 72 from node 1 to 1
2012/10/08 20:25:55.262700 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:25:55.262711 [11383]: Added timed event "queue_next_trigger": 0x25aef60
2012/10/08 20:25:55.262720 [11383]: Ending timer event 0x25b42b0 "queue_next_trigger"
2012/10/08 20:25:55.262730 [11383]: server/ctdb_server.c:331 ctdb request 2147483505 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.262743 [11383]: Destroying timer event 0x25ae300 "ctdb_control_timeout"
2012/10/08 20:25:55.262753 [11383]: Ending timer event 0x25aef60 "queue_next_trigger"
2012/10/08 20:25:55.262780 [recoverd:11454]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:25:55.262809 [11383]: server/ctdb_daemon.c:628 client request 2147483499 of type 7 length 80 from node 1 to 1
2012/10/08 20:25:55.262823 [11383]: Added timed event "queue_next_trigger": 0x25b42b0
2012/10/08 20:25:55.262833 [11383]: Added timed event "ctdb_control_timeout": 0x25aef40
2012/10/08 20:25:55.262843 [11383]: server/ctdb_server.c:331 ctdb request 2147483506 of type 7 length 76 from node 1 to 1
2012/10/08 20:25:55.262871 [11383]: Added timed event "queue_next_trigger": 0x25b5470
2012/10/08 20:25:55.262883 [11383]: Ending timer event 0x25b42b0 "queue_next_trigger"
2012/10/08 20:25:55.262892 [11383]: server/ctdb_server.c:331 ctdb request 2147483506 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.262905 [11383]: Destroying timer event 0x25aef40 "ctdb_control_timeout"
2012/10/08 20:25:55.262915 [11383]: Ending timer event 0x25b5470 "queue_next_trigger"
2012/10/08 20:25:55.262956 [11383]: server/ctdb_daemon.c:628 client request 2147483500 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.262970 [11383]: Added timed event "queue_next_trigger": 0x25b42b0
2012/10/08 20:25:55.262980 [11383]: Added timed event "ctdb_control_timeout": 0x25b5dc0
2012/10/08 20:25:55.262990 [11383]: server/ctdb_server.c:331 ctdb request 2147483507 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263001 [11383]: Added timed event "queue_next_trigger": 0x25b57e0
2012/10/08 20:25:55.263010 [11383]: Ending timer event 0x25b42b0 "queue_next_trigger"
2012/10/08 20:25:55.263019 [11383]: server/ctdb_server.c:331 ctdb request 2147483507 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.263038 [11383]: Destroying timer event 0x25b5dc0 "ctdb_control_timeout"
2012/10/08 20:25:55.263048 [11383]: Ending timer event 0x25b57e0 "queue_next_trigger"
2012/10/08 20:25:55.263231 [11383]: SIGCHLD from 11588
2012/10/08 20:25:55.263264 [recoverd:11454]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:25:55.263277 [recoverd:11454]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:25:55.263303 [11383]: server/ctdb_daemon.c:628 client request 2147483501 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263318 [11383]: Added timed event "queue_next_trigger": 0x25b5600
2012/10/08 20:25:55.263327 [11383]: Added timed event "ctdb_control_timeout": 0x25b60d0
2012/10/08 20:25:55.263337 [11383]: server/ctdb_server.c:331 ctdb request 2147483508 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263348 [11383]: server/ctdb_freeze.c:523 num_nodes[2]
2012/10/08 20:25:55.263356 [11383]: server/ctdb_freeze.c:526 node[0].flags[0x3]
2012/10/08 20:25:55.263364 [11383]: server/ctdb_freeze.c:526 node[1].flags[0x2]
2012/10/08 20:25:55.263371 [11383]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/08 20:25:55.263381 [11383]: Added timed event "queue_next_trigger": 0x25b6a50
2012/10/08 20:25:55.263390 [11383]: Ending timer event 0x25b5600 "queue_next_trigger"
2012/10/08 20:25:55.263399 [11383]: server/ctdb_server.c:331 ctdb request 2147483508 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.263412 [11383]: Destroying timer event 0x25b60d0 "ctdb_control_timeout"
2012/10/08 20:25:55.263422 [11383]: Ending timer event 0x25b6a50 "queue_next_trigger"
2012/10/08 20:25:55.263448 [recoverd:11454]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:25:55.263476 [11383]: server/ctdb_daemon.c:628 client request 2147483502 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263490 [11383]: Added timed event "queue_next_trigger": 0x25b5dc0
2012/10/08 20:25:55.263499 [11383]: Added timed event "ctdb_control_timeout": 0x25b5670
2012/10/08 20:25:55.263509 [11383]: server/ctdb_server.c:331 ctdb request 2147483509 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263521 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:25:55.263530 [11383]: Ending timer event 0x25b5dc0 "queue_next_trigger"
2012/10/08 20:25:55.263539 [11383]: server/ctdb_server.c:331 ctdb request 2147483509 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.263552 [11383]: Destroying timer event 0x25b5670 "ctdb_control_timeout"
2012/10/08 20:25:55.263562 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:25:55.263604 [11383]: server/ctdb_daemon.c:628 client request 2147483503 of type 7 length 72 from node 1 to 1
2012/10/08 20:25:55.263620 [11383]: Added timed event "queue_next_trigger": 0x25b5dc0
2012/10/08 20:25:55.263630 [11383]: Added timed event "ctdb_control_timeout": 0x25b5a50
2012/10/08 20:25:55.263640 [11383]: server/ctdb_server.c:331 ctdb request 2147483510 of type 7 length 72 from node 1 to 1
2012/10/08 20:25:55.263651 [11383]: Added timed event "queue_next_trigger": 0x25b1a30
2012/10/08 20:25:55.263660 [11383]: Ending timer event 0x25b5dc0 "queue_next_trigger"
2012/10/08 20:25:55.263670 [11383]: server/ctdb_server.c:331 ctdb request 2147483510 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.263683 [11383]: Destroying timer event 0x25b5a50 "ctdb_control_timeout"
2012/10/08 20:25:55.263692 [11383]: Ending timer event 0x25b1a30 "queue_next_trigger"
2012/10/08 20:25:55.263717 [recoverd:11454]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:25:55.263744 [11383]: server/ctdb_daemon.c:628 client request 2147483504 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263757 [11383]: Added timed event "queue_next_trigger": 0x25b5dc0
2012/10/08 20:25:55.263767 [11383]: Added timed event "ctdb_control_timeout": 0x25b7070
2012/10/08 20:25:55.263777 [11383]: server/ctdb_server.c:331 ctdb request 2147483511 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263802 [11383]: Added timed event "queue_next_trigger": 0x25b1ce0
2012/10/08 20:25:55.263813 [11383]: Ending timer event 0x25b5dc0 "queue_next_trigger"
2012/10/08 20:25:55.263823 [11383]: server/ctdb_server.c:331 ctdb request 2147483511 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.263836 [11383]: Destroying timer event 0x25b7070 "ctdb_control_timeout"
2012/10/08 20:25:55.263846 [11383]: Ending timer event 0x25b1ce0 "queue_next_trigger"
2012/10/08 20:25:55.263872 [recoverd:11454]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:25:55.263900 [11383]: server/ctdb_daemon.c:628 client request 2147483505 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:25:55.263914 [11383]: Added timed event "queue_next_trigger": 0x25b1bc0
2012/10/08 20:25:55.263923 [11383]: Added timed event "ctdb_control_timeout": 0x25b1ce0
2012/10/08 20:25:55.263933 [11383]: server/ctdb_server.c:331 ctdb request 2147483512 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.263944 [11383]: Added timed event "queue_next_trigger": 0x25b4f60
2012/10/08 20:25:55.263953 [11383]: Ending timer event 0x25b1bc0 "queue_next_trigger"
2012/10/08 20:25:55.263963 [11383]: server/ctdb_server.c:331 ctdb request 2147483512 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.263975 [11383]: Destroying timer event 0x25b1ce0 "ctdb_control_timeout"
2012/10/08 20:25:55.263985 [11383]: Ending timer event 0x25b4f60 "queue_next_trigger"
2012/10/08 20:25:55.264024 [11383]: server/ctdb_daemon.c:628 client request 2147483506 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.264038 [11383]: Added timed event "queue_next_trigger": 0x25b4e40
2012/10/08 20:25:55.264048 [11383]: Added timed event "ctdb_control_timeout": 0x25b4f60
2012/10/08 20:25:55.264058 [11383]: server/ctdb_server.c:331 ctdb request 2147483513 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.264069 [11383]: Added timed event "queue_next_trigger": 0x25b21c0
2012/10/08 20:25:55.264079 [11383]: Ending timer event 0x25b4e40 "queue_next_trigger"
2012/10/08 20:25:55.264089 [11383]: server/ctdb_server.c:331 ctdb request 2147483513 of type 8 length 120 from node 1 to 1
2012/10/08 20:25:55.264102 [11383]: Destroying timer event 0x25b4f60 "ctdb_control_timeout"
2012/10/08 20:25:55.264111 [11383]: Ending timer event 0x25b21c0 "queue_next_trigger"
2012/10/08 20:25:55.264151 [11383]: server/ctdb_daemon.c:628 client request 2147483507 of type 7 length 72 from node 1 to 1
2012/10/08 20:25:55.264166 [11383]: Added timed event "queue_next_trigger": 0x25b6be0
2012/10/08 20:25:55.264176 [11383]: Added timed event "ctdb_control_timeout": 0x25b50f0
2012/10/08 20:25:55.264186 [11383]: server/ctdb_server.c:331 ctdb request 2147483514 of type 7 length 72 from node 1 to 1
2012/10/08 20:25:55.264195 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:25:55.264205 [11383]: Added timed event "queue_next_trigger": 0x25b23e0
2012/10/08 20:25:55.264215 [11383]: Ending timer event 0x25b6be0 "queue_next_trigger"
2012/10/08 20:25:55.264224 [11383]: server/ctdb_server.c:331 ctdb request 2147483514 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.264237 [11383]: Destroying timer event 0x25b50f0 "ctdb_control_timeout"
2012/10/08 20:25:55.264247 [11383]: Ending timer event 0x25b23e0 "queue_next_trigger"
2012/10/08 20:25:55.264272 [recoverd:11454]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:25:55.264299 [11383]: server/ctdb_daemon.c:628 client request 2147483508 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.264313 [11383]: Added timed event "queue_next_trigger": 0x25b5dc0
2012/10/08 20:25:55.264323 [11383]: Added timed event "ctdb_control_timeout": 0x25b23c0
2012/10/08 20:25:55.264332 [11383]: server/ctdb_server.c:331 ctdb request 2147483515 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.264343 [11383]: Destroying timer event 0x25b9960 "ctdb_drop_all_ips_event"
2012/10/08 20:25:55.264352 [11383]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:25:55.264366 [11383]: Thawing priority 1
2012/10/08 20:25:55.264376 [11383]: Release freeze handler for prio 1
2012/10/08 20:25:55.264392 [11383]: Thawing priority 2
2012/10/08 20:25:55.264401 [11383]: Release freeze handler for prio 2
2012/10/08 20:25:55.264415 [11383]: Thawing priority 3
2012/10/08 20:25:55.264424 [11383]: Release freeze handler for prio 3
2012/10/08 20:25:55.264801 [11383]: server/ctdb_recover.c:808 Created PIPE FD:11 for setrecmode
2012/10/08 20:25:55.264840 [11383]: Added timed event "ctdb_set_recmode_timeout": 0x25b28f0
2012/10/08 20:25:55.264917 [11383]: Ending timer event 0x25b5dc0 "queue_next_trigger"
2012/10/08 20:25:55.265779 [11383]: SIGCHLD from 11451
2012/10/08 20:25:55.265800 [11383]: SIGCHLD from 11452
2012/10/08 20:25:55.265817 [11383]: SIGCHLD from 11453
2012/10/08 20:25:55.269784 [11383]: Destroying timer event 0x25b28f0 "ctdb_set_recmode_timeout"
2012/10/08 20:25:55.269803 [11383]: Added timed event "queue_next_trigger": 0x25b9af0
2012/10/08 20:25:55.269821 [11383]: server/ctdb_server.c:331 ctdb request 2147483515 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.269872 [11383]: Destroying timer event 0x25b23c0 "ctdb_control_timeout"
2012/10/08 20:25:55.269884 [11383]: Ending timer event 0x25b9af0 "queue_next_trigger"
2012/10/08 20:25:55.270052 [11383]: SIGCHLD from 11592
2012/10/08 20:25:55.270090 [recoverd:11454]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:25:55.270120 [11383]: server/ctdb_daemon.c:628 client request 2147483509 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.270146 [11383]: Added timed event "queue_next_trigger": 0x25b23c0
2012/10/08 20:25:55.270157 [11383]: Added timed event "ctdb_control_timeout": 0x25b9ad0
2012/10/08 20:25:55.270167 [11383]: server/ctdb_server.c:331 ctdb request 2147483516 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.270179 [11383]: Added timed event "queue_next_trigger": 0x25b9e50
2012/10/08 20:25:55.270188 [11383]: Ending timer event 0x25b23c0 "queue_next_trigger"
2012/10/08 20:25:55.270198 [11383]: server/ctdb_server.c:331 ctdb request 2147483516 of type 8 length 80 from node 1 to 1
2012/10/08 20:25:55.270212 [11383]: Destroying timer event 0x25b9ad0 "ctdb_control_timeout"
2012/10/08 20:25:55.270221 [11383]: Ending timer event 0x25b9e50 "queue_next_trigger"
2012/10/08 20:25:55.270300 [11383]: server/ctdb_daemon.c:628 client request 2147483510 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.270315 [11383]: Added timed event "queue_next_trigger": 0x25b9cf0
2012/10/08 20:25:55.270324 [11383]: Added timed event "ctdb_control_timeout": 0x25b9e10
2012/10/08 20:25:55.270334 [11383]: server/ctdb_server.c:331 ctdb request 2147483517 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.270346 [11383]: Added timed event "queue_next_trigger": 0x25ba170
2012/10/08 20:25:55.270355 [11383]: Ending timer event 0x25b9cf0 "queue_next_trigger"
2012/10/08 20:25:55.270365 [11383]: server/ctdb_server.c:331 ctdb request 2147483517 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.270378 [11383]: Destroying timer event 0x25b9e10 "ctdb_control_timeout"
2012/10/08 20:25:55.270387 [11383]: Ending timer event 0x25ba170 "queue_next_trigger"
2012/10/08 20:25:55.270445 [recoverd:11454]: ----------------------------------------
2012/10/08 20:25:55.270456 [recoverd:11454]: CONSIDERING MOVES (UNASSIGNED)
2012/10/08 20:25:55.270470 [recoverd:11454]: ----------------------------------------
2012/10/08 20:25:55.270505 [recoverd:11454]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:25:55.270533 [11383]: server/ctdb_daemon.c:628 client request 2147483511 of type 7 length 80 from node 1 to 1
2012/10/08 20:25:55.270547 [11383]: Added timed event "queue_next_trigger": 0x25ba050
2012/10/08 20:25:55.270557 [11383]: Added timed event "ctdb_control_timeout": 0x25ba170
2012/10/08 20:25:55.270567 [11383]: server/ctdb_server.c:331 ctdb request 2147483518 of type 7 length 80 from node 1 to 1
2012/10/08 20:25:55.270598 [11383]: Redundant release of IP 10.100.55.198/24 on interface __none__ (ip not held)
2012/10/08 20:25:55.270617 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/08 20:25:55.270629 [11383]: Added timed event "queue_next_trigger": 0x25ba500
2012/10/08 20:25:55.270639 [11383]: Ending timer event 0x25ba050 "queue_next_trigger"
2012/10/08 20:25:55.270648 [11383]: server/ctdb_server.c:331 ctdb request 2147483518 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.270662 [11383]: Destroying timer event 0x25ba170 "ctdb_control_timeout"
2012/10/08 20:25:55.270671 [11383]: Ending timer event 0x25ba500 "queue_next_trigger"
2012/10/08 20:25:55.270722 [11383]: server/ctdb_daemon.c:628 client request 2147483512 of type 7 length 80 from node 1 to 1
2012/10/08 20:25:55.270736 [11383]: Added timed event "queue_next_trigger": 0x25ba3e0
2012/10/08 20:25:55.270746 [11383]: Added timed event "ctdb_control_timeout": 0x25ba500
2012/10/08 20:25:55.270756 [11383]: server/ctdb_server.c:331 ctdb request 2147483519 of type 7 length 74 from node 1 to 1
2012/10/08 20:25:55.270768 [11383]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:25:55.270776 [11383]: Monitoring has been disabled
2012/10/08 20:25:55.270798 [11383]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/08 20:25:55.270903 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:25:55.270938 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:25:55.270953 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:25:55.270971 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:25:55.271448 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.271565 [11383]: Added timed event "ctdb_event_script_timeout": 0x25b6560
2012/10/08 20:25:55.271593 [11383]: Ending timer event 0x25ba3e0 "queue_next_trigger"
2012/10/08 20:25:55.272016 [eventscript-00.ctdb-ipreallocated:11593]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/00.ctdb ipreallocated
2012/10/08 20:25:55.283918 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.284427 [11383]: SIGCHLD from 11593
2012/10/08 20:25:55.284694 [eventscript-01.reclock-ipreallocated:11598]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/01.reclock ipreallocated
2012/10/08 20:25:55.293670 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.294175 [11383]: SIGCHLD from 11598
2012/10/08 20:25:55.294509 [eventscript-10.interface-ipreallocated:11602]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/10.interface ipreallocated
2012/10/08 20:25:55.303588 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.304076 [11383]: SIGCHLD from 11602
2012/10/08 20:25:55.304346 [eventscript-11.natgw-ipreallocated:11606]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.natgw ipreallocated
2012/10/08 20:25:55.313643 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.314149 [11383]: SIGCHLD from 11606
2012/10/08 20:25:55.314474 [eventscript-11.routing-ipreallocated:11610]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.routing ipreallocated
2012/10/08 20:25:55.323300 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.323783 [11383]: SIGCHLD from 11610
2012/10/08 20:25:55.324054 [eventscript-13.per_ip_routing-ipreallocated:11614]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/13.per_ip_routing ipreallocated
2012/10/08 20:25:55.335590 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.336093 [11383]: SIGCHLD from 11614
2012/10/08 20:25:55.336370 [eventscript-20.multipathd-ipreallocated:11619]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/20.multipathd ipreallocated
2012/10/08 20:25:55.336945 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.337639 [eventscript-31.clamd-ipreallocated:11620]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/31.clamd ipreallocated
2012/10/08 20:25:55.338178 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.338551 [11383]: SIGCHLD from 11619
2012/10/08 20:25:55.338839 [eventscript-40.vsftpd-ipreallocated:11621]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.vsftpd ipreallocated
2012/10/08 20:25:55.342451 [11383]: SIGCHLD from 11620
2012/10/08 20:25:55.348164 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.348660 [11383]: SIGCHLD from 11621
2012/10/08 20:25:55.348931 [eventscript-40.fs_use-ipreallocated:11625]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.fs_use ipreallocated
2012/10/08 20:25:55.349494 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.350292 [eventscript-41.httpd-ipreallocated:11626]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/41.httpd ipreallocated
2012/10/08 20:25:55.352291 [11383]: SIGCHLD from 11625
2012/10/08 20:25:55.359360 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.359866 [11383]: SIGCHLD from 11626
2012/10/08 20:25:55.360135 [eventscript-50.samba-ipreallocated:11630]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/50.samba ipreallocated
2012/10/08 20:25:55.372169 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.372669 [11383]: SIGCHLD from 11630
2012/10/08 20:25:55.372978 [eventscript-60.nfs-ipreallocated:11635]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.nfs ipreallocated
2012/10/08 20:25:55.387121 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.387612 [11383]: SIGCHLD from 11635
2012/10/08 20:25:55.387884 [eventscript-60.ganesha-ipreallocated:11641]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.ganesha ipreallocated
2012/10/08 20:25:55.388427 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.389161 [eventscript-62.cnfs-ipreallocated:11642]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/62.cnfs ipreallocated
2012/10/08 20:25:55.391171 [11383]: SIGCHLD from 11641
2012/10/08 20:25:55.400779 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.401269 [11383]: SIGCHLD from 11642
2012/10/08 20:25:55.401539 [eventscript-70.iscsi-ipreallocated:11647]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/70.iscsi ipreallocated
2012/10/08 20:25:55.410410 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.410906 [11383]: SIGCHLD from 11647
2012/10/08 20:25:55.411208 [eventscript-91.lvs-ipreallocated:11651]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/91.lvs ipreallocated
2012/10/08 20:25:55.420263 [11383]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/08 20:25:55.420313 [11383]: Monitoring has been enabled
2012/10/08 20:25:55.420329 [11383]: Added timed event "queue_next_trigger": 0x25ada10
2012/10/08 20:25:55.420344 [11383]: Destroying timer event 0x25b6560 "ctdb_event_script_timeout"
2012/10/08 20:25:55.420355 [11383]: server/ctdb_server.c:331 ctdb request 2147483519 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.420393 [11383]: Destroying timer event 0x25ba500 "ctdb_control_timeout"
2012/10/08 20:25:55.420405 [11383]: Ending timer event 0x25ada10 "queue_next_trigger"
2012/10/08 20:25:55.420462 [11383]: server/ctdb_daemon.c:628 client request 2147483513 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.420486 [11383]: Added timed event "queue_next_trigger": 0x25ada10
2012/10/08 20:25:55.420497 [11383]: Added timed event "ctdb_control_timeout": 0x25b5f40
2012/10/08 20:25:55.420507 [11383]: server/ctdb_server.c:331 ctdb request 2147483520 of type 7 length 64 from node 1 to 1
2012/10/08 20:25:55.420517 [11383]: Recovery has finished
2012/10/08 20:25:55.420526 [11383]: Monitoring has been disabled
2012/10/08 20:25:55.420536 [11383]: server/eventscript.c:754 Starting eventscript recovered
2012/10/08 20:25:55.420630 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:25:55.420681 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:25:55.420698 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:25:55.420716 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:25:55.421174 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.421268 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ad830
2012/10/08 20:25:55.421330 [11383]: Ending timer event 0x25ada10 "queue_next_trigger"
2012/10/08 20:25:55.421694 [11383]: SIGCHLD from 11651
2012/10/08 20:25:55.421961 [eventscript-00.ctdb-recovered:11655]: Executing event script /etc/ctdb/events.d/00.ctdb recovered
2012/10/08 20:25:55.433623 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.434096 [11383]: SIGCHLD from 11655
2012/10/08 20:25:55.434399 [eventscript-01.reclock-recovered:11660]: Executing event script /etc/ctdb/events.d/01.reclock recovered
2012/10/08 20:25:55.443359 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.443797 [11383]: SIGCHLD from 11660
2012/10/08 20:25:55.444077 [eventscript-10.interface-recovered:11664]: Executing event script /etc/ctdb/events.d/10.interface recovered
2012/10/08 20:25:55.451991 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.452348 [11383]: SIGCHLD from 11664
2012/10/08 20:25:55.452551 [eventscript-11.natgw-recovered:11668]: Executing event script /etc/ctdb/events.d/11.natgw recovered
2012/10/08 20:25:55.459136 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.459476 [11383]: SIGCHLD from 11668
2012/10/08 20:25:55.459686 [eventscript-11.routing-recovered:11672]: Executing event script /etc/ctdb/events.d/11.routing recovered
2012/10/08 20:25:55.466287 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.466635 [11383]: SIGCHLD from 11672
2012/10/08 20:25:55.466841 [eventscript-13.per_ip_routing-recovered:11676]: Executing event script /etc/ctdb/events.d/13.per_ip_routing recovered
2012/10/08 20:25:55.475410 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.475740 [11383]: SIGCHLD from 11676
2012/10/08 20:25:55.475952 [eventscript-20.multipathd-recovered:11681]: Executing event script /etc/ctdb/events.d/20.multipathd recovered
2012/10/08 20:25:55.476363 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.476856 [eventscript-31.clamd-recovered:11682]: Executing event script /etc/ctdb/events.d/31.clamd recovered
2012/10/08 20:25:55.477298 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.477522 [11383]: SIGCHLD from 11681
2012/10/08 20:25:55.477726 [eventscript-40.vsftpd-recovered:11683]: Executing event script /etc/ctdb/events.d/40.vsftpd recovered
2012/10/08 20:25:55.481813 [11383]: SIGCHLD from 11682
2012/10/08 20:25:55.484665 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.485022 [11383]: SIGCHLD from 11683
2012/10/08 20:25:55.485233 [eventscript-40.fs_use-recovered:11687]: Executing event script /etc/ctdb/events.d/40.fs_use recovered
2012/10/08 20:25:55.485676 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.486164 [eventscript-41.httpd-recovered:11688]: Executing event script /etc/ctdb/events.d/41.httpd recovered
2012/10/08 20:25:55.490196 [11383]: SIGCHLD from 11687
2012/10/08 20:25:55.493009 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.493369 [11383]: SIGCHLD from 11688
2012/10/08 20:25:55.493569 [eventscript-50.samba-recovered:11692]: Executing event script /etc/ctdb/events.d/50.samba recovered
2012/10/08 20:25:55.502535 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.502867 [11383]: SIGCHLD from 11692
2012/10/08 20:25:55.503080 [eventscript-60.nfs-recovered:11697]: Executing event script /etc/ctdb/events.d/60.nfs recovered
2012/10/08 20:25:55.513627 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.513987 [11383]: SIGCHLD from 11697
2012/10/08 20:25:55.514189 [eventscript-60.ganesha-recovered:11703]: Executing event script /etc/ctdb/events.d/60.ganesha recovered
2012/10/08 20:25:55.514656 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.515138 [eventscript-62.cnfs-recovered:11704]: Executing event script /etc/ctdb/events.d/62.cnfs recovered
2012/10/08 20:25:55.519154 [11383]: SIGCHLD from 11703
2012/10/08 20:25:55.523876 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.524227 [11383]: SIGCHLD from 11704
2012/10/08 20:25:55.524427 [eventscript-70.iscsi-recovered:11709]: Executing event script /etc/ctdb/events.d/70.iscsi recovered
2012/10/08 20:25:55.531065 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:25:55.531394 [11383]: SIGCHLD from 11709
2012/10/08 20:25:55.531607 [eventscript-91.lvs-recovered:11713]: Executing event script /etc/ctdb/events.d/91.lvs recovered
2012/10/08 20:25:55.537879 [11383]: server/eventscript.c:485 Eventscript recovered finished with state 0
2012/10/08 20:25:55.537916 [11383]: Monitoring has been enabled
2012/10/08 20:25:55.537936 [11383]: Added timed event "queue_next_trigger": 0x25b71b0
2012/10/08 20:25:55.537980 [11383]: Destroying timer event 0x25ad830 "ctdb_event_script_timeout"
2012/10/08 20:25:55.537989 [11383]: server/ctdb_server.c:331 ctdb request 2147483520 of type 8 length 48 from node 1 to 1
2012/10/08 20:25:55.538018 [11383]: Destroying timer event 0x25b5f40 "ctdb_control_timeout"
2012/10/08 20:25:55.538027 [11383]: Ending timer event 0x25b71b0 "queue_next_trigger"
2012/10/08 20:25:55.538056 [recoverd:11454]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:25:55.538080 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 48 from node 1 to 4026531844
2012/10/08 20:25:55.538091 [11383]: Added timed event "queue_next_trigger": 0x25ad830
2012/10/08 20:25:55.538099 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 5 length 48 from node 1 to 1
2012/10/08 20:25:55.538108 [11383]: Ending timer event 0x25ad830 "queue_next_trigger"
2012/10/08 20:25:55.538121 [recoverd:11454]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:25:55.538129 [recoverd:11454]: Resetting ban count to 0 for all nodes
2012/10/08 20:25:55.538136 [recoverd:11454]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:25:55.538302 [11383]: SIGCHLD from 11713
2012/10/08 20:25:55.938417 [11383]: Added timed event "ctdb_statistics_update": 0x25b71b0
2012/10/08 20:25:55.938433 [11383]: Ending timer event 0x25b6020 "ctdb_statistics_update"
2012/10/08 20:25:56.095498 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:56.095514 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:56.095535 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b42b0
2012/10/08 20:25:56.095545 [11383]: Ending timer event 0x25b64b0 "ctdb_wait_until_recovered"
2012/10/08 20:25:56.095555 [11383]: Added timed event "ctdb_time_tick": 0x25a9fe0
2012/10/08 20:25:56.095563 [11383]: Ending timer event 0x25b67b0 "ctdb_time_tick"
2012/10/08 20:25:56.099660 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:25:56.099717 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6780
2012/10/08 20:25:56.099729 [11383]: Ending timer event 0x25b53c0 "ctdb_tcp_node_connect"
2012/10/08 20:25:56.100028 [11383]: Destroying timer event 0x25b6780 "ctdb_tcp_node_connect"
2012/10/08 20:25:56.100052 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6780
2012/10/08 20:25:56.939118 [11383]: Added timed event "ctdb_statistics_update": 0x25aeed0
2012/10/08 20:25:56.939137 [11383]: Ending timer event 0x25b71b0 "ctdb_statistics_update"
2012/10/08 20:25:57.093201 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25b71b0
2012/10/08 20:25:57.093220 [11383]: Ending timer event 0x25adb70 "ctdb_check_for_dead_nodes"
2012/10/08 20:25:57.096271 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:57.096287 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:57.096298 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b63d0
2012/10/08 20:25:57.096307 [11383]: Ending timer event 0x25b42b0 "ctdb_wait_until_recovered"
2012/10/08 20:25:57.096317 [11383]: Added timed event "ctdb_time_tick": 0x25b42b0
2012/10/08 20:25:57.096325 [11383]: Ending timer event 0x25a9fe0 "ctdb_time_tick"
2012/10/08 20:25:57.100384 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:25:57.100416 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6e00
2012/10/08 20:25:57.100427 [11383]: Ending timer event 0x25b6780 "ctdb_tcp_node_connect"
2012/10/08 20:25:57.100682 [11383]: Destroying timer event 0x25b6e00 "ctdb_tcp_node_connect"
2012/10/08 20:25:57.100703 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6e00
2012/10/08 20:25:57.939768 [11383]: Added timed event "ctdb_statistics_update": 0x25a9fe0
2012/10/08 20:25:57.939784 [11383]: Ending timer event 0x25aeed0 "ctdb_statistics_update"
2012/10/08 20:25:58.096837 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:58.096854 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:58.096866 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b7070
2012/10/08 20:25:58.096875 [11383]: Ending timer event 0x25b63d0 "ctdb_wait_until_recovered"
2012/10/08 20:25:58.096885 [11383]: Added timed event "ctdb_time_tick": 0x25b63d0
2012/10/08 20:25:58.096893 [11383]: Ending timer event 0x25b42b0 "ctdb_time_tick"
2012/10/08 20:25:58.100954 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:25:58.100996 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5850
2012/10/08 20:25:58.101008 [11383]: Ending timer event 0x25b6e00 "ctdb_tcp_node_connect"
2012/10/08 20:25:58.101279 [11383]: Destroying timer event 0x25b5850 "ctdb_tcp_node_connect"
2012/10/08 20:25:58.101300 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5850
2012/10/08 20:25:58.940349 [11383]: Added timed event "ctdb_statistics_update": 0x25b59e0
2012/10/08 20:25:58.940367 [11383]: Ending timer event 0x25a9fe0 "ctdb_statistics_update"
2012/10/08 20:25:59.097428 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:25:59.097444 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:59.097456 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b5fb0
2012/10/08 20:25:59.097465 [11383]: Ending timer event 0x25b7070 "ctdb_wait_until_recovered"
2012/10/08 20:25:59.097474 [11383]: Added timed event "ctdb_time_tick": 0x25b7070
2012/10/08 20:25:59.097483 [11383]: Ending timer event 0x25b63d0 "ctdb_time_tick"
2012/10/08 20:25:59.101544 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:25:59.101589 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b60d0
2012/10/08 20:25:59.101600 [11383]: Ending timer event 0x25b5850 "ctdb_tcp_node_connect"
2012/10/08 20:25:59.101878 [11383]: Destroying timer event 0x25b60d0 "ctdb_tcp_node_connect"
2012/10/08 20:25:59.101899 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b60d0
2012/10/08 20:25:59.940966 [11383]: Added timed event "ctdb_statistics_update": 0x25b5b90
2012/10/08 20:25:59.940981 [11383]: Ending timer event 0x25b59e0 "ctdb_statistics_update"
2012/10/08 20:26:00.098152 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:00.098168 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:00.098180 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b5d90
2012/10/08 20:26:00.098188 [11383]: Ending timer event 0x25b5fb0 "ctdb_wait_until_recovered"
2012/10/08 20:26:00.098198 [11383]: Added timed event "ctdb_time_tick": 0x25b5fb0
2012/10/08 20:26:00.098207 [11383]: Ending timer event 0x25b7070 "ctdb_time_tick"
2012/10/08 20:26:00.102271 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:00.102306 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6a50
2012/10/08 20:26:00.102317 [11383]: Ending timer event 0x25b60d0 "ctdb_tcp_node_connect"
2012/10/08 20:26:00.102602 [11383]: Destroying timer event 0x25b6a50 "ctdb_tcp_node_connect"
2012/10/08 20:26:00.102623 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6a50
2012/10/08 20:26:00.941714 [11383]: Added timed event "ctdb_statistics_update": 0x25b6be0
2012/10/08 20:26:00.941731 [11383]: Ending timer event 0x25b5b90 "ctdb_statistics_update"
2012/10/08 20:26:01.098772 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:01.098788 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:01.098800 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b5430
2012/10/08 20:26:01.098808 [11383]: Ending timer event 0x25b5d90 "ctdb_wait_until_recovered"
2012/10/08 20:26:01.098818 [11383]: Added timed event "ctdb_time_tick": 0x25b5d90
2012/10/08 20:26:01.098827 [11383]: Ending timer event 0x25b5fb0 "ctdb_time_tick"
2012/10/08 20:26:01.102885 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:01.102918 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5550
2012/10/08 20:26:01.102930 [11383]: Ending timer event 0x25b6a50 "ctdb_tcp_node_connect"
2012/10/08 20:26:01.103201 [11383]: Destroying timer event 0x25b5550 "ctdb_tcp_node_connect"
2012/10/08 20:26:01.103222 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6a50
2012/10/08 20:26:01.942276 [11383]: Added timed event "ctdb_statistics_update": 0x25b5fb0
2012/10/08 20:26:01.942292 [11383]: Ending timer event 0x25b6be0 "ctdb_statistics_update"
2012/10/08 20:26:02.093353 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25b6be0
2012/10/08 20:26:02.093371 [11383]: Ending timer event 0x25b71b0 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:02.099421 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:02.099437 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:02.099449 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b1c60
2012/10/08 20:26:02.099458 [11383]: Ending timer event 0x25b5430 "ctdb_wait_until_recovered"
2012/10/08 20:26:02.099467 [11383]: Added timed event "ctdb_time_tick": 0x25b5430
2012/10/08 20:26:02.099476 [11383]: Ending timer event 0x25b5d90 "ctdb_time_tick"
2012/10/08 20:26:02.103532 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:02.103567 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1d80
2012/10/08 20:26:02.103579 [11383]: Ending timer event 0x25b6a50 "ctdb_tcp_node_connect"
2012/10/08 20:26:02.103889 [11383]: Destroying timer event 0x25b1d80 "ctdb_tcp_node_connect"
2012/10/08 20:26:02.103910 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1d80
2012/10/08 20:26:02.942967 [11383]: Added timed event "ctdb_statistics_update": 0x25b4e40
2012/10/08 20:26:02.942990 [11383]: Ending timer event 0x25b5fb0 "ctdb_statistics_update"
2012/10/08 20:26:03.100074 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:03.100091 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:03.100102 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b5040
2012/10/08 20:26:03.100111 [11383]: Ending timer event 0x25b1c60 "ctdb_wait_until_recovered"
2012/10/08 20:26:03.100121 [11383]: Added timed event "ctdb_time_tick": 0x25b1c60
2012/10/08 20:26:03.100129 [11383]: Ending timer event 0x25b5430 "ctdb_time_tick"
2012/10/08 20:26:03.104186 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:03.104219 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5160
2012/10/08 20:26:03.104230 [11383]: Ending timer event 0x25b1d80 "ctdb_tcp_node_connect"
2012/10/08 20:26:03.104489 [11383]: Destroying timer event 0x25b5160 "ctdb_tcp_node_connect"
2012/10/08 20:26:03.104510 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5160
2012/10/08 20:26:03.943757 [11383]: Added timed event "ctdb_statistics_update": 0x25b2160
2012/10/08 20:26:03.943774 [11383]: Ending timer event 0x25b4e40 "ctdb_statistics_update"
2012/10/08 20:26:04.100845 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:04.100862 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:04.100874 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b2360
2012/10/08 20:26:04.100882 [11383]: Ending timer event 0x25b5040 "ctdb_wait_until_recovered"
2012/10/08 20:26:04.100892 [11383]: Added timed event "ctdb_time_tick": 0x25b5040
2012/10/08 20:26:04.100901 [11383]: Ending timer event 0x25b1c60 "ctdb_time_tick"
2012/10/08 20:26:04.104958 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:04.104990 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2480
2012/10/08 20:26:04.105002 [11383]: Ending timer event 0x25b5160 "ctdb_tcp_node_connect"
2012/10/08 20:26:04.105260 [11383]: Destroying timer event 0x25b2480 "ctdb_tcp_node_connect"
2012/10/08 20:26:04.105280 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2480
2012/10/08 20:26:04.944512 [11383]: Added timed event "ctdb_statistics_update": 0x25b2610
2012/10/08 20:26:04.944528 [11383]: Ending timer event 0x25b2160 "ctdb_statistics_update"
2012/10/08 20:26:05.101606 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:05.101621 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:05.101633 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b2810
2012/10/08 20:26:05.101641 [11383]: Ending timer event 0x25b2360 "ctdb_wait_until_recovered"
2012/10/08 20:26:05.101651 [11383]: Added timed event "ctdb_time_tick": 0x25b2360
2012/10/08 20:26:05.101659 [11383]: Ending timer event 0x25b5040 "ctdb_time_tick"
2012/10/08 20:26:05.105717 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:05.105750 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2930
2012/10/08 20:26:05.105762 [11383]: Ending timer event 0x25b2480 "ctdb_tcp_node_connect"
2012/10/08 20:26:05.106016 [11383]: Destroying timer event 0x25b2930 "ctdb_tcp_node_connect"
2012/10/08 20:26:05.106036 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2930
2012/10/08 20:26:05.548187 [recoverd:11454]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:26:05.548240 [11383]: server/ctdb_daemon.c:628 client request 2147483514 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.548258 [11383]: Added timed event "queue_next_trigger": 0x25b9c90
2012/10/08 20:26:05.548268 [11383]: Added timed event "ctdb_control_timeout": 0x25b9db0
2012/10/08 20:26:05.548279 [11383]: server/ctdb_server.c:331 ctdb request 2147483521 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548290 [11383]: Destroying timer event 0x25ae1e0 "ctdb_recd_ping_timeout"
2012/10/08 20:26:05.548300 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:26:05.548317 [11383]: Added timed event "queue_next_trigger": 0x25ba0d0
2012/10/08 20:26:05.548327 [11383]: Ending timer event 0x25b9c90 "queue_next_trigger"
2012/10/08 20:26:05.548337 [11383]: server/ctdb_server.c:331 ctdb request 2147483521 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.548351 [11383]: Destroying timer event 0x25b9db0 "ctdb_control_timeout"
2012/10/08 20:26:05.548361 [11383]: Ending timer event 0x25ba0d0 "queue_next_trigger"
2012/10/08 20:26:05.548402 [11383]: server/ctdb_daemon.c:628 client request 2147483515 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.548417 [11383]: Added timed event "queue_next_trigger": 0x25b99f0
2012/10/08 20:26:05.548427 [11383]: Added timed event "ctdb_control_timeout": 0x25ba260
2012/10/08 20:26:05.548437 [11383]: server/ctdb_server.c:331 ctdb request 2147483522 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548448 [11383]: Added timed event "queue_next_trigger": 0x25ba4a0
2012/10/08 20:26:05.548457 [11383]: Ending timer event 0x25b99f0 "queue_next_trigger"
2012/10/08 20:26:05.548467 [11383]: server/ctdb_server.c:331 ctdb request 2147483522 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.548479 [11383]: Destroying timer event 0x25ba260 "ctdb_control_timeout"
2012/10/08 20:26:05.548489 [11383]: Ending timer event 0x25ba4a0 "queue_next_trigger"
2012/10/08 20:26:05.548529 [11383]: server/ctdb_daemon.c:628 client request 2147483516 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.548543 [11383]: Added timed event "queue_next_trigger": 0x25b99f0
2012/10/08 20:26:05.548553 [11383]: Added timed event "ctdb_control_timeout": 0x25ba460
2012/10/08 20:26:05.548563 [11383]: server/ctdb_server.c:331 ctdb request 2147483523 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548575 [11383]: Added timed event "queue_next_trigger": 0x25ba920
2012/10/08 20:26:05.548584 [11383]: Ending timer event 0x25b99f0 "queue_next_trigger"
2012/10/08 20:26:05.548594 [11383]: server/ctdb_server.c:331 ctdb request 2147483523 of type 8 length 228 from node 1 to 1
2012/10/08 20:26:05.548606 [11383]: Destroying timer event 0x25ba460 "ctdb_control_timeout"
2012/10/08 20:26:05.548616 [11383]: Ending timer event 0x25ba920 "queue_next_trigger"
2012/10/08 20:26:05.548657 [11383]: server/ctdb_daemon.c:628 client request 2147483517 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.548671 [11383]: Added timed event "queue_next_trigger": 0x25ba6a0
2012/10/08 20:26:05.548681 [11383]: Added timed event "ctdb_control_timeout": 0x25ba7c0
2012/10/08 20:26:05.548691 [11383]: server/ctdb_server.c:331 ctdb request 2147483524 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548702 [11383]: Added timed event "queue_next_trigger": 0x25bab50
2012/10/08 20:26:05.548712 [11383]: Ending timer event 0x25ba6a0 "queue_next_trigger"
2012/10/08 20:26:05.548721 [11383]: server/ctdb_server.c:331 ctdb request 2147483524 of type 8 length 67 from node 1 to 1
2012/10/08 20:26:05.548734 [11383]: Destroying timer event 0x25ba7c0 "ctdb_control_timeout"
2012/10/08 20:26:05.548744 [11383]: Ending timer event 0x25bab50 "queue_next_trigger"
2012/10/08 20:26:05.548785 [11383]: server/ctdb_daemon.c:628 client request 2147483518 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.548799 [11383]: Added timed event "queue_next_trigger": 0x25ba460
2012/10/08 20:26:05.548809 [11383]: Added timed event "ctdb_control_timeout": 0x25bab20
2012/10/08 20:26:05.548819 [11383]: server/ctdb_server.c:331 ctdb request 2147483525 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548830 [11383]: Added timed event "queue_next_trigger": 0x25bae70
2012/10/08 20:26:05.548839 [11383]: Ending timer event 0x25ba460 "queue_next_trigger"
2012/10/08 20:26:05.548848 [11383]: server/ctdb_server.c:331 ctdb request 2147483525 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.548861 [11383]: Destroying timer event 0x25bab20 "ctdb_control_timeout"
2012/10/08 20:26:05.548871 [11383]: Ending timer event 0x25bae70 "queue_next_trigger"
2012/10/08 20:26:05.548916 [11383]: server/ctdb_daemon.c:628 client request 2147483519 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548931 [11383]: Added timed event "queue_next_trigger": 0x25b9fb0
2012/10/08 20:26:05.548940 [11383]: Added timed event "ctdb_control_timeout": 0x25ba0d0
2012/10/08 20:26:05.548950 [11383]: server/ctdb_server.c:331 ctdb request 2147483526 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.548962 [11383]: Added timed event "queue_next_trigger": 0x25b7380
2012/10/08 20:26:05.548972 [11383]: Ending timer event 0x25b9fb0 "queue_next_trigger"
2012/10/08 20:26:05.548981 [11383]: server/ctdb_server.c:331 ctdb request 2147483526 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:05.548994 [11383]: Destroying timer event 0x25ba0d0 "ctdb_control_timeout"
2012/10/08 20:26:05.549003 [11383]: Ending timer event 0x25b7380 "queue_next_trigger"
2012/10/08 20:26:05.549044 [11383]: server/ctdb_daemon.c:628 client request 2147483520 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549058 [11383]: Added timed event "queue_next_trigger": 0x25b7380
2012/10/08 20:26:05.549068 [11383]: Added timed event "ctdb_control_timeout": 0x25bb100
2012/10/08 20:26:05.549078 [11383]: server/ctdb_server.c:331 ctdb request 2147483527 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549090 [11383]: Added timed event "queue_next_trigger": 0x25b76b0
2012/10/08 20:26:05.549100 [11383]: Ending timer event 0x25b7380 "queue_next_trigger"
2012/10/08 20:26:05.549109 [11383]: server/ctdb_server.c:331 ctdb request 2147483527 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:05.549122 [11383]: Destroying timer event 0x25bb100 "ctdb_control_timeout"
2012/10/08 20:26:05.549132 [11383]: Ending timer event 0x25b76b0 "queue_next_trigger"
2012/10/08 20:26:05.549172 [11383]: server/ctdb_daemon.c:628 client request 2147483521 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549187 [11383]: Added timed event "queue_next_trigger": 0x25ba460
2012/10/08 20:26:05.549197 [11383]: Added timed event "ctdb_control_timeout": 0x25b7580
2012/10/08 20:26:05.549207 [11383]: server/ctdb_server.c:331 ctdb request 2147483528 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549218 [11383]: Added timed event "queue_next_trigger": 0x25b78d0
2012/10/08 20:26:05.549227 [11383]: Ending timer event 0x25ba460 "queue_next_trigger"
2012/10/08 20:26:05.549236 [11383]: server/ctdb_server.c:331 ctdb request 2147483528 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.549249 [11383]: Destroying timer event 0x25b7580 "ctdb_control_timeout"
2012/10/08 20:26:05.549259 [11383]: Ending timer event 0x25b78d0 "queue_next_trigger"
2012/10/08 20:26:05.549298 [11383]: server/ctdb_daemon.c:628 client request 2147483522 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549312 [11383]: Added timed event "queue_next_trigger": 0x25ba460
2012/10/08 20:26:05.549322 [11383]: Added timed event "ctdb_control_timeout": 0x25b78b0
2012/10/08 20:26:05.549332 [11383]: server/ctdb_server.c:331 ctdb request 2147483529 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549343 [11383]: Added timed event "queue_next_trigger": 0x25b7d20
2012/10/08 20:26:05.549353 [11383]: Ending timer event 0x25ba460 "queue_next_trigger"
2012/10/08 20:26:05.549362 [11383]: server/ctdb_server.c:331 ctdb request 2147483529 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:05.549375 [11383]: Destroying timer event 0x25b78b0 "ctdb_control_timeout"
2012/10/08 20:26:05.549384 [11383]: Ending timer event 0x25b7d20 "queue_next_trigger"
2012/10/08 20:26:05.549425 [11383]: server/ctdb_daemon.c:628 client request 2147483523 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.549439 [11383]: Added timed event "queue_next_trigger": 0x25b7ad0
2012/10/08 20:26:05.549448 [11383]: Added timed event "ctdb_control_timeout": 0x25b7bf0
2012/10/08 20:26:05.549458 [11383]: server/ctdb_server.c:331 ctdb request 2147483530 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549491 [11383]: Added timed event "queue_next_trigger": 0x25b8050
2012/10/08 20:26:05.549508 [11383]: Ending timer event 0x25b7ad0 "queue_next_trigger"
2012/10/08 20:26:05.549528 [11383]: server/ctdb_server.c:331 ctdb request 2147483530 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:05.549543 [11383]: Destroying timer event 0x25b7bf0 "ctdb_control_timeout"
2012/10/08 20:26:05.549552 [11383]: Ending timer event 0x25b8050 "queue_next_trigger"
2012/10/08 20:26:05.549593 [11383]: server/ctdb_daemon.c:628 client request 2147483524 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.549607 [11383]: Added timed event "queue_next_trigger": 0x25ba460
2012/10/08 20:26:05.549617 [11383]: Added timed event "ctdb_control_timeout": 0x25b7f20
2012/10/08 20:26:05.549626 [11383]: server/ctdb_server.c:331 ctdb request 2147483531 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549638 [11383]: Added timed event "queue_next_trigger": 0x25b8280
2012/10/08 20:26:05.549648 [11383]: Ending timer event 0x25ba460 "queue_next_trigger"
2012/10/08 20:26:05.549657 [11383]: server/ctdb_server.c:331 ctdb request 2147483531 of type 8 length 72 from node 1 to 1
2012/10/08 20:26:05.549670 [11383]: Destroying timer event 0x25b7f20 "ctdb_control_timeout"
2012/10/08 20:26:05.549680 [11383]: Ending timer event 0x25b8280 "queue_next_trigger"
2012/10/08 20:26:05.549705 [recoverd:11454]: The interfaces status has changed on local node 1 - force takeover run
2012/10/08 20:26:05.549732 [11383]: server/ctdb_daemon.c:628 client request 2147483525 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.549746 [11383]: Added timed event "queue_next_trigger": 0x25ba460
2012/10/08 20:26:05.549756 [11383]: Added timed event "ctdb_control_timeout": 0x25b8250
2012/10/08 20:26:05.549766 [11383]: server/ctdb_server.c:331 ctdb request 2147483532 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549777 [11383]: Added timed event "queue_next_trigger": 0x25b85e0
2012/10/08 20:26:05.549787 [11383]: Ending timer event 0x25ba460 "queue_next_trigger"
2012/10/08 20:26:05.549796 [11383]: server/ctdb_server.c:331 ctdb request 2147483532 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:05.549809 [11383]: Destroying timer event 0x25b8250 "ctdb_control_timeout"
2012/10/08 20:26:05.549818 [11383]: Ending timer event 0x25b85e0 "queue_next_trigger"
2012/10/08 20:26:05.549858 [11383]: server/ctdb_daemon.c:628 client request 2147483526 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.549872 [11383]: Added timed event "queue_next_trigger": 0x25b8480
2012/10/08 20:26:05.549882 [11383]: Added timed event "ctdb_control_timeout": 0x25b85a0
2012/10/08 20:26:05.549892 [11383]: server/ctdb_server.c:331 ctdb request 2147483533 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.549903 [11383]: Added timed event "queue_next_trigger": 0x25b8a20
2012/10/08 20:26:05.549912 [11383]: Ending timer event 0x25b8480 "queue_next_trigger"
2012/10/08 20:26:05.549922 [11383]: server/ctdb_server.c:331 ctdb request 2147483533 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:05.549934 [11383]: Destroying timer event 0x25b85a0 "ctdb_control_timeout"
2012/10/08 20:26:05.549944 [11383]: Ending timer event 0x25b8a20 "queue_next_trigger"
2012/10/08 20:26:05.550001 [recoverd:11454]: Trigger takeoverrun
2012/10/08 20:26:05.550028 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 64 from node 1 to 1
2012/10/08 20:26:05.550069 [11383]: server/ctdb_daemon.c:628 client request 2147483527 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.550083 [11383]: Added timed event "queue_next_trigger": 0x25b87e0
2012/10/08 20:26:05.550093 [11383]: Added timed event "ctdb_control_timeout": 0x25b8900
2012/10/08 20:26:05.550103 [11383]: server/ctdb_server.c:331 ctdb request 2147483534 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.550114 [11383]: Added timed event "queue_next_trigger": 0x25b8bd0
2012/10/08 20:26:05.550123 [11383]: Ending timer event 0x25b87e0 "queue_next_trigger"
2012/10/08 20:26:05.550133 [11383]: server/ctdb_server.c:331 ctdb request 2147483534 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.550152 [11383]: Destroying timer event 0x25b8900 "ctdb_control_timeout"
2012/10/08 20:26:05.550162 [11383]: Ending timer event 0x25b8bd0 "queue_next_trigger"
2012/10/08 20:26:05.550208 [11383]: server/ctdb_daemon.c:628 client request 2147483528 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.550246 [11383]: Added timed event "queue_next_trigger": 0x25b85a0
2012/10/08 20:26:05.550260 [11383]: Added timed event "ctdb_control_timeout": 0x25b8bb0
2012/10/08 20:26:05.550271 [11383]: server/ctdb_server.c:331 ctdb request 2147483535 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.550282 [11383]: Added timed event "queue_next_trigger": 0x25b8ef0
2012/10/08 20:26:05.550292 [11383]: Ending timer event 0x25b85a0 "queue_next_trigger"
2012/10/08 20:26:05.550301 [11383]: server/ctdb_server.c:331 ctdb request 2147483535 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.550315 [11383]: Destroying timer event 0x25b8bb0 "ctdb_control_timeout"
2012/10/08 20:26:05.550325 [11383]: Ending timer event 0x25b8ef0 "queue_next_trigger"
2012/10/08 20:26:05.550691 [recoverd:11454]: server/ctdb_recoverd.c:2810 Created PIPE FD:17 for check_recovery_lock
2012/10/08 20:26:05.553094 [recoverd:11454]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:26:05.553108 [recoverd:11454]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:26:05.554357 [11383]: server/ctdb_daemon.c:628 client request 2147483529 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:05.554374 [11383]: Added timed event "queue_next_trigger": 0x25bad50
2012/10/08 20:26:05.554385 [11383]: Added timed event "ctdb_control_timeout": 0x25bae70
2012/10/08 20:26:05.554395 [11383]: server/ctdb_server.c:331 ctdb request 2147483536 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:05.554407 [11383]: Added timed event "queue_next_trigger": 0x25b9210
2012/10/08 20:26:05.554416 [11383]: Ending timer event 0x25bad50 "queue_next_trigger"
2012/10/08 20:26:05.554426 [11383]: server/ctdb_server.c:331 ctdb request 2147483536 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.554439 [11383]: Destroying timer event 0x25bae70 "ctdb_control_timeout"
2012/10/08 20:26:05.554449 [11383]: Ending timer event 0x25b9210 "queue_next_trigger"
2012/10/08 20:26:05.554513 [recoverd:11454]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:26:05.554527 [recoverd:11454]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:26:05.554536 [recoverd:11454]: Taking out recovery lock from recovery daemon
2012/10/08 20:26:05.554544 [recoverd:11454]: Take the recovery lock
2012/10/08 20:26:05.560196 [recoverd:11454]: Recovery lock taken successfully
2012/10/08 20:26:05.560211 [recoverd:11454]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:26:05.560240 [11383]: server/ctdb_daemon.c:628 client request 2147483530 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:05.560256 [11383]: Added timed event "queue_next_trigger": 0x25b85a0
2012/10/08 20:26:05.560266 [11383]: Added timed event "ctdb_control_timeout": 0x25b8eb0
2012/10/08 20:26:05.560276 [11383]: server/ctdb_server.c:331 ctdb request 2147483537 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:05.560287 [11383]: Added timed event "queue_next_trigger": 0x25b9530
2012/10/08 20:26:05.560297 [11383]: Ending timer event 0x25b85a0 "queue_next_trigger"
2012/10/08 20:26:05.560306 [11383]: server/ctdb_server.c:331 ctdb request 2147483537 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.560331 [11383]: Destroying timer event 0x25b8eb0 "ctdb_control_timeout"
2012/10/08 20:26:05.560341 [11383]: Ending timer event 0x25b9530 "queue_next_trigger"
2012/10/08 20:26:05.560369 [recoverd:11454]: RECD SIGCHLD from 11717
2012/10/08 20:26:05.560388 [recoverd:11454]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:26:05.560398 [recoverd:11454]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 1
2012/10/08 20:26:05.560423 [11383]: server/ctdb_daemon.c:628 client request 2147483531 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.560444 [11383]: Added timed event "queue_next_trigger": 0x25b6c90
2012/10/08 20:26:05.560455 [11383]: Added timed event "ctdb_control_timeout": 0x25b1f40
2012/10/08 20:26:05.560465 [11383]: server/ctdb_server.c:331 ctdb request 2147483538 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.560476 [11383]: Added timed event "queue_next_trigger": 0x25b2750
2012/10/08 20:26:05.560486 [11383]: Ending timer event 0x25b6c90 "queue_next_trigger"
2012/10/08 20:26:05.560495 [11383]: server/ctdb_server.c:331 ctdb request 2147483538 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.560508 [11383]: Destroying timer event 0x25b1f40 "ctdb_control_timeout"
2012/10/08 20:26:05.560518 [11383]: Ending timer event 0x25b2750 "queue_next_trigger"
2012/10/08 20:26:05.560546 [recoverd:11454]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:26:05.560557 [recoverd:11454]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:26:05.560584 [11383]: server/ctdb_daemon.c:628 client request 2147483532 of type 7 length 88 from node 1 to 1
2012/10/08 20:26:05.560597 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:05.560607 [11383]: Added timed event "ctdb_control_timeout": 0x25b26c0
2012/10/08 20:26:05.560617 [11383]: server/ctdb_server.c:331 ctdb request 2147483539 of type 7 length 83 from node 1 to 1
2012/10/08 20:26:05.560628 [11383]: Added timed event "queue_next_trigger": 0x25ae320
2012/10/08 20:26:05.560637 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:05.560647 [11383]: server/ctdb_server.c:331 ctdb request 2147483539 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.560660 [11383]: Destroying timer event 0x25b26c0 "ctdb_control_timeout"
2012/10/08 20:26:05.560669 [11383]: Ending timer event 0x25ae320 "queue_next_trigger"
2012/10/08 20:26:05.560711 [11383]: server/ctdb_daemon.c:628 client request 2147483533 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.560725 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:05.560735 [11383]: Added timed event "ctdb_control_timeout": 0x25ae300
2012/10/08 20:26:05.560745 [11383]: server/ctdb_server.c:331 ctdb request 2147483540 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.560755 [11383]: Freeze priority 1
2012/10/08 20:26:05.561272 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:05.561644 [11383]: Added timed event "queue_next_trigger": 0x25b4340
2012/10/08 20:26:05.561659 [11383]: server/ctdb_server.c:331 ctdb request 2147483540 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.561731 [11383]: Destroying timer event 0x25ae300 "ctdb_control_timeout"
2012/10/08 20:26:05.561744 [11383]: Ending timer event 0x25b4340 "queue_next_trigger"
2012/10/08 20:26:05.561787 [11383]: server/ctdb_daemon.c:628 client request 2147483534 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.561801 [11383]: Added timed event "queue_next_trigger": 0x25b42b0
2012/10/08 20:26:05.561811 [11383]: Added timed event "ctdb_control_timeout": 0x25b2480
2012/10/08 20:26:05.561821 [11383]: server/ctdb_server.c:331 ctdb request 2147483541 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.561831 [11383]: Freeze priority 2
2012/10/08 20:26:05.562340 [11383]: Ending timer event 0x25b42b0 "queue_next_trigger"
2012/10/08 20:26:05.562758 [11383]: Added timed event "queue_next_trigger": 0x25b2270
2012/10/08 20:26:05.562788 [11383]: server/ctdb_server.c:331 ctdb request 2147483541 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.562863 [11383]: Destroying timer event 0x25b2480 "ctdb_control_timeout"
2012/10/08 20:26:05.562875 [11383]: Ending timer event 0x25b2270 "queue_next_trigger"
2012/10/08 20:26:05.562919 [11383]: server/ctdb_daemon.c:628 client request 2147483535 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.562933 [11383]: Added timed event "queue_next_trigger": 0x25aef40
2012/10/08 20:26:05.562943 [11383]: Added timed event "ctdb_control_timeout": 0x25bafc0
2012/10/08 20:26:05.562960 [11383]: server/ctdb_server.c:331 ctdb request 2147483542 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.562990 [11383]: Freeze priority 3
2012/10/08 20:26:05.563462 [11383]: Ending timer event 0x25aef40 "queue_next_trigger"
2012/10/08 20:26:05.563859 [11383]: Added timed event "queue_next_trigger": 0x25b60c0
2012/10/08 20:26:05.563873 [11383]: server/ctdb_server.c:331 ctdb request 2147483542 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.563962 [11383]: Destroying timer event 0x25bafc0 "ctdb_control_timeout"
2012/10/08 20:26:05.563974 [11383]: Ending timer event 0x25b60c0 "queue_next_trigger"
2012/10/08 20:26:05.564018 [11383]: server/ctdb_daemon.c:628 client request 2147483536 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.564033 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:26:05.564043 [11383]: Added timed event "ctdb_control_timeout": 0x25b6f20
2012/10/08 20:26:05.564053 [11383]: server/ctdb_server.c:331 ctdb request 2147483543 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.564064 [11383]: Added timed event "ctdb_drop_all_ips_event": 0x25ae300
2012/10/08 20:26:05.564072 [11383]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:26:05.564082 [11383]: Added timed event "queue_next_trigger": 0x25b67c0
2012/10/08 20:26:05.564091 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:26:05.564100 [11383]: server/ctdb_server.c:331 ctdb request 2147483543 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.564113 [11383]: Destroying timer event 0x25b6f20 "ctdb_control_timeout"
2012/10/08 20:26:05.564123 [11383]: Ending timer event 0x25b67c0 "queue_next_trigger"
2012/10/08 20:26:05.564164 [11383]: server/ctdb_daemon.c:628 client request 2147483537 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.564178 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:26:05.564188 [11383]: Added timed event "ctdb_control_timeout": 0x25b6710
2012/10/08 20:26:05.564198 [11383]: server/ctdb_server.c:331 ctdb request 2147483544 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.564207 [11383]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:26:05.564217 [11383]: Monitoring has been disabled
2012/10/08 20:26:05.564246 [11383]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/08 20:26:05.564383 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:05.564420 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:05.564435 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:05.564453 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:05.564905 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.564998 [11383]: Added timed event "ctdb_event_script_timeout": 0x25adb70
2012/10/08 20:26:05.565035 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:26:05.565483 [eventscript-00.ctdb-startrecovery:11721]: Executing event script /etc/ctdb/events.d/00.ctdb startrecovery
2012/10/08 20:26:05.578271 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.578771 [11383]: SIGCHLD from 11721
2012/10/08 20:26:05.579054 [eventscript-01.reclock-startrecovery:11726]: Executing event script /etc/ctdb/events.d/01.reclock startrecovery
2012/10/08 20:26:05.588325 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.588854 [11383]: SIGCHLD from 11726
2012/10/08 20:26:05.589144 [eventscript-10.interface-startrecovery:11730]: Executing event script /etc/ctdb/events.d/10.interface startrecovery
2012/10/08 20:26:05.598569 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.599081 [11383]: SIGCHLD from 11730
2012/10/08 20:26:05.599378 [eventscript-11.natgw-startrecovery:11734]: Executing event script /etc/ctdb/events.d/11.natgw startrecovery
2012/10/08 20:26:05.608489 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.609004 [11383]: SIGCHLD from 11734
2012/10/08 20:26:05.609307 [eventscript-11.routing-startrecovery:11738]: Executing event script /etc/ctdb/events.d/11.routing startrecovery
2012/10/08 20:26:05.618350 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.618854 [11383]: SIGCHLD from 11738
2012/10/08 20:26:05.619145 [eventscript-13.per_ip_routing-startrecovery:11742]: Executing event script /etc/ctdb/events.d/13.per_ip_routing startrecovery
2012/10/08 20:26:05.631211 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.631750 [11383]: SIGCHLD from 11742
2012/10/08 20:26:05.632048 [eventscript-20.multipathd-startrecovery:11747]: Executing event script /etc/ctdb/events.d/20.multipathd startrecovery
2012/10/08 20:26:05.632680 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.633425 [eventscript-31.clamd-startrecovery:11748]: Executing event script /etc/ctdb/events.d/31.clamd startrecovery
2012/10/08 20:26:05.634074 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.634380 [11383]: SIGCHLD from 11747
2012/10/08 20:26:05.634663 [eventscript-40.vsftpd-startrecovery:11749]: Executing event script /etc/ctdb/events.d/40.vsftpd startrecovery
2012/10/08 20:26:05.638467 [11383]: SIGCHLD from 11748
2012/10/08 20:26:05.644021 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.644634 [11383]: SIGCHLD from 11749
2012/10/08 20:26:05.644922 [eventscript-40.fs_use-startrecovery:11753]: Executing event script /etc/ctdb/events.d/40.fs_use startrecovery
2012/10/08 20:26:05.645472 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.646294 [eventscript-41.httpd-startrecovery:11754]: Executing event script /etc/ctdb/events.d/41.httpd startrecovery
2012/10/08 20:26:05.648326 [11383]: SIGCHLD from 11753
2012/10/08 20:26:05.655573 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.656099 [11383]: SIGCHLD from 11754
2012/10/08 20:26:05.656388 [eventscript-50.samba-startrecovery:11758]: Executing event script /etc/ctdb/events.d/50.samba startrecovery
2012/10/08 20:26:05.668647 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.669165 [11383]: SIGCHLD from 11758
2012/10/08 20:26:05.669458 [eventscript-60.nfs-startrecovery:11763]: Executing event script /etc/ctdb/events.d/60.nfs startrecovery
2012/10/08 20:26:05.683976 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.684505 [11383]: SIGCHLD from 11763
2012/10/08 20:26:05.684779 [eventscript-60.ganesha-startrecovery:11769]: Executing event script /etc/ctdb/events.d/60.ganesha startrecovery
2012/10/08 20:26:05.685315 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.686075 [eventscript-62.cnfs-startrecovery:11770]: Executing event script /etc/ctdb/events.d/62.cnfs startrecovery
2012/10/08 20:26:05.688099 [11383]: SIGCHLD from 11769
2012/10/08 20:26:05.698024 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.698561 [11383]: SIGCHLD from 11770
2012/10/08 20:26:05.698855 [eventscript-70.iscsi-startrecovery:11775]: Executing event script /etc/ctdb/events.d/70.iscsi startrecovery
2012/10/08 20:26:05.707935 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:05.708330 [11383]: SIGCHLD from 11775
2012/10/08 20:26:05.708553 [eventscript-91.lvs-startrecovery:11779]: Executing event script /etc/ctdb/events.d/91.lvs startrecovery
2012/10/08 20:26:05.715084 [11383]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/08 20:26:05.715152 [11383]: Added timed event "queue_next_trigger": 0x25b1f40
2012/10/08 20:26:05.715167 [11383]: Destroying timer event 0x25adb70 "ctdb_event_script_timeout"
2012/10/08 20:26:05.715193 [11383]: server/ctdb_server.c:331 ctdb request 2147483544 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.715243 [11383]: Destroying timer event 0x25b6710 "ctdb_control_timeout"
2012/10/08 20:26:05.715253 [11383]: Ending timer event 0x25b1f40 "queue_next_trigger"
2012/10/08 20:26:05.715303 [11383]: server/ctdb_daemon.c:628 client request 2147483538 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.715315 [11383]: Added timed event "queue_next_trigger": 0x25b6360
2012/10/08 20:26:05.715323 [11383]: Added timed event "ctdb_control_timeout": 0x25b6710
2012/10/08 20:26:05.715331 [11383]: server/ctdb_server.c:331 ctdb request 2147483545 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.715339 [11383]: Added timed event "queue_next_trigger": 0x25ad7c0
2012/10/08 20:26:05.715347 [11383]: Ending timer event 0x25b6360 "queue_next_trigger"
2012/10/08 20:26:05.715354 [11383]: server/ctdb_server.c:331 ctdb request 2147483545 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.715364 [11383]: Destroying timer event 0x25b6710 "ctdb_control_timeout"
2012/10/08 20:26:05.715371 [11383]: Ending timer event 0x25ad7c0 "queue_next_trigger"
2012/10/08 20:26:05.715401 [11383]: server/ctdb_daemon.c:628 client request 2147483539 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.715412 [11383]: Added timed event "queue_next_trigger": 0x25ad7c0
2012/10/08 20:26:05.715419 [11383]: Added timed event "ctdb_control_timeout": 0x25b26c0
2012/10/08 20:26:05.715427 [11383]: server/ctdb_server.c:331 ctdb request 2147483546 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.715437 [11383]: Added timed event "queue_next_trigger": 0x25b6900
2012/10/08 20:26:05.715444 [11383]: Ending timer event 0x25ad7c0 "queue_next_trigger"
2012/10/08 20:26:05.715451 [11383]: server/ctdb_server.c:331 ctdb request 2147483546 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:05.715461 [11383]: Destroying timer event 0x25b26c0 "ctdb_control_timeout"
2012/10/08 20:26:05.715468 [11383]: Ending timer event 0x25b6900 "queue_next_trigger"
2012/10/08 20:26:05.715499 [11383]: server/ctdb_daemon.c:628 client request 2147483540 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.715510 [11383]: Added timed event "queue_next_trigger": 0x25b6710
2012/10/08 20:26:05.715517 [11383]: Added timed event "ctdb_control_timeout": 0x25aa050
2012/10/08 20:26:05.715526 [11383]: server/ctdb_server.c:331 ctdb request 2147483547 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.715534 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:05.715541 [11383]: Added timed event "queue_next_trigger": 0x25b2170
2012/10/08 20:26:05.715548 [11383]: Ending timer event 0x25b6710 "queue_next_trigger"
2012/10/08 20:26:05.715555 [11383]: server/ctdb_server.c:331 ctdb request 2147483547 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.715565 [11383]: Destroying timer event 0x25aa050 "ctdb_control_timeout"
2012/10/08 20:26:05.715572 [11383]: Ending timer event 0x25b2170 "queue_next_trigger"
2012/10/08 20:26:05.715592 [recoverd:11454]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:26:05.715616 [11383]: server/ctdb_daemon.c:628 client request 2147483541 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.715626 [11383]: Added timed event "queue_next_trigger": 0x25b6710
2012/10/08 20:26:05.715633 [11383]: Added timed event "ctdb_control_timeout": 0x25aefb0
2012/10/08 20:26:05.715641 [11383]: server/ctdb_server.c:331 ctdb request 2147483548 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.715649 [11383]: Added timed event "queue_next_trigger": 0x25b2410
2012/10/08 20:26:05.715657 [11383]: Ending timer event 0x25b6710 "queue_next_trigger"
2012/10/08 20:26:05.715668 [11383]: server/ctdb_server.c:331 ctdb request 2147483548 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.715679 [11383]: Destroying timer event 0x25aefb0 "ctdb_control_timeout"
2012/10/08 20:26:05.715686 [11383]: Ending timer event 0x25b2410 "queue_next_trigger"
2012/10/08 20:26:05.715716 [11383]: server/ctdb_daemon.c:628 client request 2147483542 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.715727 [11383]: Added timed event "queue_next_trigger": 0x25b6710
2012/10/08 20:26:05.715734 [11383]: Added timed event "ctdb_control_timeout": 0x25b4320
2012/10/08 20:26:05.715741 [11383]: server/ctdb_server.c:331 ctdb request 2147483549 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.715750 [11383]: Added timed event "queue_next_trigger": 0x25b6e90
2012/10/08 20:26:05.715757 [11383]: Ending timer event 0x25b6710 "queue_next_trigger"
2012/10/08 20:26:05.715764 [11383]: server/ctdb_server.c:331 ctdb request 2147483549 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.715773 [11383]: Destroying timer event 0x25b4320 "ctdb_control_timeout"
2012/10/08 20:26:05.715781 [11383]: Ending timer event 0x25b6e90 "queue_next_trigger"
2012/10/08 20:26:05.715922 [11383]: SIGCHLD from 11779
2012/10/08 20:26:05.715947 [recoverd:11454]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:26:05.715956 [recoverd:11454]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:26:05.715976 [11383]: server/ctdb_daemon.c:628 client request 2147483543 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.715986 [11383]: Added timed event "queue_next_trigger": 0x25b7020
2012/10/08 20:26:05.715996 [11383]: Added timed event "ctdb_control_timeout": 0x25b6e70
2012/10/08 20:26:05.716003 [11383]: server/ctdb_server.c:331 ctdb request 2147483550 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716011 [11383]: server/ctdb_freeze.c:523 num_nodes[2]
2012/10/08 20:26:05.716018 [11383]: server/ctdb_freeze.c:526 node[0].flags[0x3]
2012/10/08 20:26:05.716024 [11383]: server/ctdb_freeze.c:526 node[1].flags[0x2]
2012/10/08 20:26:05.716030 [11383]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/08 20:26:05.716037 [11383]: Added timed event "queue_next_trigger": 0x25b5970
2012/10/08 20:26:05.716044 [11383]: Ending timer event 0x25b7020 "queue_next_trigger"
2012/10/08 20:26:05.716051 [11383]: server/ctdb_server.c:331 ctdb request 2147483550 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.716061 [11383]: Destroying timer event 0x25b6e70 "ctdb_control_timeout"
2012/10/08 20:26:05.716068 [11383]: Ending timer event 0x25b5970 "queue_next_trigger"
2012/10/08 20:26:05.716088 [recoverd:11454]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:26:05.716108 [11383]: server/ctdb_daemon.c:628 client request 2147483544 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716119 [11383]: Added timed event "queue_next_trigger": 0x25b5850
2012/10/08 20:26:05.716126 [11383]: Added timed event "ctdb_control_timeout": 0x25b5970
2012/10/08 20:26:05.716133 [11383]: server/ctdb_server.c:331 ctdb request 2147483551 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716142 [11383]: Added timed event "queue_next_trigger": 0x25b60d0
2012/10/08 20:26:05.716149 [11383]: Ending timer event 0x25b5850 "queue_next_trigger"
2012/10/08 20:26:05.716156 [11383]: server/ctdb_server.c:331 ctdb request 2147483551 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.716166 [11383]: Destroying timer event 0x25b5970 "ctdb_control_timeout"
2012/10/08 20:26:05.716173 [11383]: Ending timer event 0x25b60d0 "queue_next_trigger"
2012/10/08 20:26:05.716205 [11383]: server/ctdb_daemon.c:628 client request 2147483545 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.716217 [11383]: Added timed event "queue_next_trigger": 0x25b5fb0
2012/10/08 20:26:05.716224 [11383]: Added timed event "ctdb_control_timeout": 0x25b60d0
2012/10/08 20:26:05.716232 [11383]: server/ctdb_server.c:331 ctdb request 2147483552 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.716240 [11383]: Added timed event "queue_next_trigger": 0x25b5d20
2012/10/08 20:26:05.716261 [11383]: Ending timer event 0x25b5fb0 "queue_next_trigger"
2012/10/08 20:26:05.716269 [11383]: server/ctdb_server.c:331 ctdb request 2147483552 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.716279 [11383]: Destroying timer event 0x25b60d0 "ctdb_control_timeout"
2012/10/08 20:26:05.716287 [11383]: Ending timer event 0x25b5d20 "queue_next_trigger"
2012/10/08 20:26:05.716306 [recoverd:11454]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:26:05.716327 [11383]: server/ctdb_daemon.c:628 client request 2147483546 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716337 [11383]: Added timed event "queue_next_trigger": 0x25b5c00
2012/10/08 20:26:05.716344 [11383]: Added timed event "ctdb_control_timeout": 0x25b5d20
2012/10/08 20:26:05.716352 [11383]: server/ctdb_server.c:331 ctdb request 2147483553 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716360 [11383]: Added timed event "queue_next_trigger": 0x25b54e0
2012/10/08 20:26:05.716367 [11383]: Ending timer event 0x25b5c00 "queue_next_trigger"
2012/10/08 20:26:05.716374 [11383]: server/ctdb_server.c:331 ctdb request 2147483553 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.716384 [11383]: Destroying timer event 0x25b5d20 "ctdb_control_timeout"
2012/10/08 20:26:05.716391 [11383]: Ending timer event 0x25b54e0 "queue_next_trigger"
2012/10/08 20:26:05.716410 [recoverd:11454]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:26:05.716431 [11383]: server/ctdb_daemon.c:628 client request 2147483547 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:05.716441 [11383]: Added timed event "queue_next_trigger": 0x25b7020
2012/10/08 20:26:05.716449 [11383]: Added timed event "ctdb_control_timeout": 0x25b5430
2012/10/08 20:26:05.716456 [11383]: server/ctdb_server.c:331 ctdb request 2147483554 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716464 [11383]: Added timed event "queue_next_trigger": 0x25b1b50
2012/10/08 20:26:05.716471 [11383]: Ending timer event 0x25b7020 "queue_next_trigger"
2012/10/08 20:26:05.716478 [11383]: server/ctdb_server.c:331 ctdb request 2147483554 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.716487 [11383]: Destroying timer event 0x25b5430 "ctdb_control_timeout"
2012/10/08 20:26:05.716495 [11383]: Ending timer event 0x25b1b50 "queue_next_trigger"
2012/10/08 20:26:05.716524 [11383]: server/ctdb_daemon.c:628 client request 2147483548 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716534 [11383]: Added timed event "queue_next_trigger": 0x25b1a30
2012/10/08 20:26:05.716541 [11383]: Added timed event "ctdb_control_timeout": 0x25b1b50
2012/10/08 20:26:05.716549 [11383]: server/ctdb_server.c:331 ctdb request 2147483555 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716558 [11383]: Added timed event "queue_next_trigger": 0x25b4f70
2012/10/08 20:26:05.716565 [11383]: Ending timer event 0x25b1a30 "queue_next_trigger"
2012/10/08 20:26:05.716572 [11383]: server/ctdb_server.c:331 ctdb request 2147483555 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:05.716582 [11383]: Destroying timer event 0x25b1b50 "ctdb_control_timeout"
2012/10/08 20:26:05.716590 [11383]: Ending timer event 0x25b4f70 "queue_next_trigger"
2012/10/08 20:26:05.716620 [11383]: server/ctdb_daemon.c:628 client request 2147483549 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.716630 [11383]: Added timed event "queue_next_trigger": 0x25b6830
2012/10/08 20:26:05.716638 [11383]: Added timed event "ctdb_control_timeout": 0x25b5090
2012/10/08 20:26:05.716645 [11383]: server/ctdb_server.c:331 ctdb request 2147483556 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:05.716652 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:05.716660 [11383]: Added timed event "queue_next_trigger": 0x25b4e60
2012/10/08 20:26:05.716667 [11383]: Ending timer event 0x25b6830 "queue_next_trigger"
2012/10/08 20:26:05.716674 [11383]: server/ctdb_server.c:331 ctdb request 2147483556 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.716708 [11383]: Destroying timer event 0x25b5090 "ctdb_control_timeout"
2012/10/08 20:26:05.716716 [11383]: Ending timer event 0x25b4e60 "queue_next_trigger"
2012/10/08 20:26:05.716737 [recoverd:11454]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:26:05.716758 [11383]: server/ctdb_daemon.c:628 client request 2147483550 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716768 [11383]: Added timed event "queue_next_trigger": 0x25b51b0
2012/10/08 20:26:05.716775 [11383]: Added timed event "ctdb_control_timeout": 0x25b4e40
2012/10/08 20:26:05.716783 [11383]: server/ctdb_server.c:331 ctdb request 2147483557 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.716791 [11383]: Destroying timer event 0x25ae300 "ctdb_drop_all_ips_event"
2012/10/08 20:26:05.716798 [11383]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:26:05.716804 [11383]: Thawing priority 1
2012/10/08 20:26:05.716811 [11383]: Release freeze handler for prio 1
2012/10/08 20:26:05.716823 [11383]: Thawing priority 2
2012/10/08 20:26:05.716830 [11383]: Release freeze handler for prio 2
2012/10/08 20:26:05.716840 [11383]: Thawing priority 3
2012/10/08 20:26:05.716847 [11383]: Release freeze handler for prio 3
2012/10/08 20:26:05.717142 [11383]: server/ctdb_recover.c:808 Created PIPE FD:11 for setrecmode
2012/10/08 20:26:05.717192 [11383]: Added timed event "ctdb_set_recmode_timeout": 0x25ba900
2012/10/08 20:26:05.717251 [11383]: Ending timer event 0x25b51b0 "queue_next_trigger"
2012/10/08 20:26:05.717956 [11383]: SIGCHLD from 11718
2012/10/08 20:26:05.717969 [11383]: SIGCHLD from 11719
2012/10/08 20:26:05.717978 [11383]: SIGCHLD from 11720
2012/10/08 20:26:05.721797 [11383]: Destroying timer event 0x25ba900 "ctdb_set_recmode_timeout"
2012/10/08 20:26:05.721825 [11383]: Added timed event "queue_next_trigger": 0x25b99b0
2012/10/08 20:26:05.721840 [11383]: server/ctdb_server.c:331 ctdb request 2147483557 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.721887 [11383]: Destroying timer event 0x25b4e40 "ctdb_control_timeout"
2012/10/08 20:26:05.721896 [11383]: Ending timer event 0x25b99b0 "queue_next_trigger"
2012/10/08 20:26:05.722023 [11383]: SIGCHLD from 11783
2012/10/08 20:26:05.722051 [recoverd:11454]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:26:05.722076 [11383]: server/ctdb_daemon.c:628 client request 2147483551 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.722096 [11383]: Added timed event "queue_next_trigger": 0x25b9920
2012/10/08 20:26:05.722104 [11383]: Added timed event "ctdb_control_timeout": 0x25b9b40
2012/10/08 20:26:05.722111 [11383]: server/ctdb_server.c:331 ctdb request 2147483558 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.722121 [11383]: Added timed event "queue_next_trigger": 0x25b9dc0
2012/10/08 20:26:05.722128 [11383]: Ending timer event 0x25b9920 "queue_next_trigger"
2012/10/08 20:26:05.722135 [11383]: server/ctdb_server.c:331 ctdb request 2147483558 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:05.722145 [11383]: Destroying timer event 0x25b9b40 "ctdb_control_timeout"
2012/10/08 20:26:05.722152 [11383]: Ending timer event 0x25b9dc0 "queue_next_trigger"
2012/10/08 20:26:05.722185 [11383]: server/ctdb_daemon.c:628 client request 2147483552 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.722196 [11383]: Added timed event "queue_next_trigger": 0x25b9c60
2012/10/08 20:26:05.722203 [11383]: Added timed event "ctdb_control_timeout": 0x25b9d80
2012/10/08 20:26:05.722210 [11383]: server/ctdb_server.c:331 ctdb request 2147483559 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.722219 [11383]: Added timed event "queue_next_trigger": 0x25ba0e0
2012/10/08 20:26:05.722244 [11383]: Ending timer event 0x25b9c60 "queue_next_trigger"
2012/10/08 20:26:05.722253 [11383]: server/ctdb_server.c:331 ctdb request 2147483559 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.722263 [11383]: Destroying timer event 0x25b9d80 "ctdb_control_timeout"
2012/10/08 20:26:05.722292 [11383]: Ending timer event 0x25ba0e0 "queue_next_trigger"
2012/10/08 20:26:05.722318 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:05.722327 [recoverd:11454]: CONSIDERING MOVES (UNASSIGNED)
2012/10/08 20:26:05.722333 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:05.722342 [recoverd:11454]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:26:05.722362 [11383]: server/ctdb_daemon.c:628 client request 2147483553 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:05.722372 [11383]: Added timed event "queue_next_trigger": 0x25b9fc0
2012/10/08 20:26:05.722380 [11383]: Added timed event "ctdb_control_timeout": 0x25ba0e0
2012/10/08 20:26:05.722387 [11383]: server/ctdb_server.c:331 ctdb request 2147483560 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:05.722413 [11383]: Redundant release of IP 10.100.55.198/24 on interface __none__ (ip not held)
2012/10/08 20:26:05.722422 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/08 20:26:05.722431 [11383]: Added timed event "queue_next_trigger": 0x25ba400
2012/10/08 20:26:05.722438 [11383]: Ending timer event 0x25b9fc0 "queue_next_trigger"
2012/10/08 20:26:05.722445 [11383]: server/ctdb_server.c:331 ctdb request 2147483560 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.722455 [11383]: Destroying timer event 0x25ba0e0 "ctdb_control_timeout"
2012/10/08 20:26:05.722462 [11383]: Ending timer event 0x25ba400 "queue_next_trigger"
2012/10/08 20:26:05.722494 [11383]: server/ctdb_daemon.c:628 client request 2147483554 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:05.722505 [11383]: Added timed event "queue_next_trigger": 0x25ba2e0
2012/10/08 20:26:05.722513 [11383]: Added timed event "ctdb_control_timeout": 0x25ba400
2012/10/08 20:26:05.722520 [11383]: server/ctdb_server.c:331 ctdb request 2147483561 of type 7 length 74 from node 1 to 1
2012/10/08 20:26:05.722529 [11383]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:26:05.722536 [11383]: Monitoring has been disabled
2012/10/08 20:26:05.722545 [11383]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/08 20:26:05.722612 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:05.722638 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:05.722649 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:05.722663 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:05.723002 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.723066 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ade00
2012/10/08 20:26:05.723095 [11383]: Ending timer event 0x25ba2e0 "queue_next_trigger"
2012/10/08 20:26:05.723427 [eventscript-00.ctdb-ipreallocated:11784]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/00.ctdb ipreallocated
2012/10/08 20:26:05.732340 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.732695 [11383]: SIGCHLD from 11784
2012/10/08 20:26:05.732911 [eventscript-01.reclock-ipreallocated:11789]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/01.reclock ipreallocated
2012/10/08 20:26:05.739628 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.739961 [11383]: SIGCHLD from 11789
2012/10/08 20:26:05.740174 [eventscript-10.interface-ipreallocated:11793]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/10.interface ipreallocated
2012/10/08 20:26:05.747082 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.747433 [11383]: SIGCHLD from 11793
2012/10/08 20:26:05.747638 [eventscript-11.natgw-ipreallocated:11797]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.natgw ipreallocated
2012/10/08 20:26:05.754203 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.754585 [11383]: SIGCHLD from 11797
2012/10/08 20:26:05.754790 [eventscript-11.routing-ipreallocated:11801]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.routing ipreallocated
2012/10/08 20:26:05.761476 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.761838 [11383]: SIGCHLD from 11801
2012/10/08 20:26:05.762049 [eventscript-13.per_ip_routing-ipreallocated:11805]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/13.per_ip_routing ipreallocated
2012/10/08 20:26:05.770698 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.771058 [11383]: SIGCHLD from 11805
2012/10/08 20:26:05.771257 [eventscript-20.multipathd-ipreallocated:11810]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/20.multipathd ipreallocated
2012/10/08 20:26:05.771660 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.772174 [eventscript-31.clamd-ipreallocated:11811]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/31.clamd ipreallocated
2012/10/08 20:26:05.772595 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.772820 [11383]: SIGCHLD from 11810
2012/10/08 20:26:05.773030 [eventscript-40.vsftpd-ipreallocated:11812]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.vsftpd ipreallocated
2012/10/08 20:26:05.777213 [11383]: SIGCHLD from 11811
2012/10/08 20:26:05.780046 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.780396 [11383]: SIGCHLD from 11812
2012/10/08 20:26:05.780615 [eventscript-40.fs_use-ipreallocated:11816]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.fs_use ipreallocated
2012/10/08 20:26:05.781031 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.781529 [eventscript-41.httpd-ipreallocated:11817]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/41.httpd ipreallocated
2012/10/08 20:26:05.785541 [11383]: SIGCHLD from 11816
2012/10/08 20:26:05.788385 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.788735 [11383]: SIGCHLD from 11817
2012/10/08 20:26:05.788946 [eventscript-50.samba-ipreallocated:11821]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/50.samba ipreallocated
2012/10/08 20:26:05.797890 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.798267 [11383]: SIGCHLD from 11821
2012/10/08 20:26:05.798481 [eventscript-60.nfs-ipreallocated:11826]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.nfs ipreallocated
2012/10/08 20:26:05.809133 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.809494 [11383]: SIGCHLD from 11826
2012/10/08 20:26:05.809702 [eventscript-60.ganesha-ipreallocated:11832]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.ganesha ipreallocated
2012/10/08 20:26:05.810131 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.810654 [eventscript-62.cnfs-ipreallocated:11833]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/62.cnfs ipreallocated
2012/10/08 20:26:05.814666 [11383]: SIGCHLD from 11832
2012/10/08 20:26:05.819472 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.819830 [11383]: SIGCHLD from 11833
2012/10/08 20:26:05.820050 [eventscript-70.iscsi-ipreallocated:11838]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/70.iscsi ipreallocated
2012/10/08 20:26:05.826740 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.827104 [11383]: SIGCHLD from 11838
2012/10/08 20:26:05.827302 [eventscript-91.lvs-ipreallocated:11842]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/91.lvs ipreallocated
2012/10/08 20:26:05.833519 [11383]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/08 20:26:05.833569 [11383]: Monitoring has been enabled
2012/10/08 20:26:05.833582 [11383]: Added timed event "queue_next_trigger": 0x25b6c90
2012/10/08 20:26:05.833593 [11383]: Destroying timer event 0x25ade00 "ctdb_event_script_timeout"
2012/10/08 20:26:05.833601 [11383]: server/ctdb_server.c:331 ctdb request 2147483561 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.833651 [11383]: Destroying timer event 0x25ba400 "ctdb_control_timeout"
2012/10/08 20:26:05.833660 [11383]: Ending timer event 0x25b6c90 "queue_next_trigger"
2012/10/08 20:26:05.833706 [11383]: server/ctdb_daemon.c:628 client request 2147483555 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.833717 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:05.833725 [11383]: Added timed event "ctdb_control_timeout": 0x25b29e0
2012/10/08 20:26:05.833733 [11383]: server/ctdb_server.c:331 ctdb request 2147483562 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:05.833740 [11383]: Recovery has finished
2012/10/08 20:26:05.833747 [11383]: Monitoring has been disabled
2012/10/08 20:26:05.833755 [11383]: server/eventscript.c:754 Starting eventscript recovered
2012/10/08 20:26:05.833837 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:05.833863 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:05.833874 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:05.833888 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:05.834267 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.834340 [11383]: Added timed event "ctdb_event_script_timeout": 0x25b7290
2012/10/08 20:26:05.834377 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:05.834647 [11383]: SIGCHLD from 11842
2012/10/08 20:26:05.834852 [eventscript-00.ctdb-recovered:11846]: Executing event script /etc/ctdb/events.d/00.ctdb recovered
2012/10/08 20:26:05.844501 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.844965 [11383]: SIGCHLD from 11846
2012/10/08 20:26:05.845256 [eventscript-01.reclock-recovered:11851]: Executing event script /etc/ctdb/events.d/01.reclock recovered
2012/10/08 20:26:05.854050 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.854581 [11383]: SIGCHLD from 11851
2012/10/08 20:26:05.854875 [eventscript-10.interface-recovered:11855]: Executing event script /etc/ctdb/events.d/10.interface recovered
2012/10/08 20:26:05.863953 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.864434 [11383]: SIGCHLD from 11855
2012/10/08 20:26:05.864717 [eventscript-11.natgw-recovered:11859]: Executing event script /etc/ctdb/events.d/11.natgw recovered
2012/10/08 20:26:05.873596 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.874079 [11383]: SIGCHLD from 11859
2012/10/08 20:26:05.874404 [eventscript-11.routing-recovered:11863]: Executing event script /etc/ctdb/events.d/11.routing recovered
2012/10/08 20:26:05.883271 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.883747 [11383]: SIGCHLD from 11863
2012/10/08 20:26:05.884032 [eventscript-13.per_ip_routing-recovered:11867]: Executing event script /etc/ctdb/events.d/13.per_ip_routing recovered
2012/10/08 20:26:05.895509 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.895997 [11383]: SIGCHLD from 11867
2012/10/08 20:26:05.896300 [eventscript-20.multipathd-recovered:11872]: Executing event script /etc/ctdb/events.d/20.multipathd recovered
2012/10/08 20:26:05.896850 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.897525 [eventscript-31.clamd-recovered:11873]: Executing event script /etc/ctdb/events.d/31.clamd recovered
2012/10/08 20:26:05.898050 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.898395 [11383]: SIGCHLD from 11872
2012/10/08 20:26:05.898683 [eventscript-40.vsftpd-recovered:11874]: Executing event script /etc/ctdb/events.d/40.vsftpd recovered
2012/10/08 20:26:05.902463 [11383]: SIGCHLD from 11873
2012/10/08 20:26:05.907924 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.908388 [11383]: SIGCHLD from 11874
2012/10/08 20:26:05.908682 [eventscript-40.fs_use-recovered:11878]: Executing event script /etc/ctdb/events.d/40.fs_use recovered
2012/10/08 20:26:05.909190 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.909934 [eventscript-41.httpd-recovered:11879]: Executing event script /etc/ctdb/events.d/41.httpd recovered
2012/10/08 20:26:05.911976 [11383]: SIGCHLD from 11878
2012/10/08 20:26:05.919044 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.919531 [11383]: SIGCHLD from 11879
2012/10/08 20:26:05.919812 [eventscript-50.samba-recovered:11883]: Executing event script /etc/ctdb/events.d/50.samba recovered
2012/10/08 20:26:05.931764 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.932249 [11383]: SIGCHLD from 11883
2012/10/08 20:26:05.932537 [eventscript-60.nfs-recovered:11888]: Executing event script /etc/ctdb/events.d/60.nfs recovered
2012/10/08 20:26:05.945414 [11383]: Added timed event "ctdb_statistics_update": 0x25adb70
2012/10/08 20:26:05.945431 [11383]: Ending timer event 0x25b2610 "ctdb_statistics_update"
2012/10/08 20:26:05.947236 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.947696 [11383]: SIGCHLD from 11888
2012/10/08 20:26:05.947977 [eventscript-60.ganesha-recovered:11894]: Executing event script /etc/ctdb/events.d/60.ganesha recovered
2012/10/08 20:26:05.948514 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.949216 [eventscript-62.cnfs-recovered:11895]: Executing event script /etc/ctdb/events.d/62.cnfs recovered
2012/10/08 20:26:05.951244 [11383]: SIGCHLD from 11894
2012/10/08 20:26:05.960778 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.961209 [11383]: SIGCHLD from 11895
2012/10/08 20:26:05.961480 [eventscript-70.iscsi-recovered:11900]: Executing event script /etc/ctdb/events.d/70.iscsi recovered
2012/10/08 20:26:05.970302 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:05.970788 [11383]: SIGCHLD from 11900
2012/10/08 20:26:05.971069 [eventscript-91.lvs-recovered:11904]: Executing event script /etc/ctdb/events.d/91.lvs recovered
2012/10/08 20:26:05.979165 [11383]: server/eventscript.c:485 Eventscript recovered finished with state 0
2012/10/08 20:26:05.979223 [11383]: Monitoring has been enabled
2012/10/08 20:26:05.979236 [11383]: Added timed event "queue_next_trigger": 0x25b6a50
2012/10/08 20:26:05.979260 [11383]: Destroying timer event 0x25b7290 "ctdb_event_script_timeout"
2012/10/08 20:26:05.979270 [11383]: server/ctdb_server.c:331 ctdb request 2147483562 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:05.979309 [11383]: Destroying timer event 0x25b29e0 "ctdb_control_timeout"
2012/10/08 20:26:05.979318 [11383]: Ending timer event 0x25b6a50 "queue_next_trigger"
2012/10/08 20:26:05.979346 [recoverd:11454]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:26:05.979369 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 48 from node 1 to 4026531844
2012/10/08 20:26:05.979387 [11383]: Added timed event "queue_next_trigger": 0x25b1fb0
2012/10/08 20:26:05.979395 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 5 length 48 from node 1 to 1
2012/10/08 20:26:05.979403 [11383]: Ending timer event 0x25b1fb0 "queue_next_trigger"
2012/10/08 20:26:05.979416 [recoverd:11454]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:26:05.979424 [recoverd:11454]: Resetting ban count to 0 for all nodes
2012/10/08 20:26:05.979431 [recoverd:11454]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:26:05.979578 [11383]: SIGCHLD from 11904
2012/10/08 20:26:06.102639 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:06.102660 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:06.102677 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b1fb0
2012/10/08 20:26:06.102684 [11383]: Ending timer event 0x25b2810 "ctdb_wait_until_recovered"
2012/10/08 20:26:06.102692 [11383]: Added timed event "ctdb_time_tick": 0x25b2810
2012/10/08 20:26:06.102698 [11383]: Ending timer event 0x25b2360 "ctdb_time_tick"
2012/10/08 20:26:06.106762 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:06.106823 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25aa090
2012/10/08 20:26:06.106838 [11383]: Ending timer event 0x25b2930 "ctdb_tcp_node_connect"
2012/10/08 20:26:06.107150 [11383]: Destroying timer event 0x25aa090 "ctdb_tcp_node_connect"
2012/10/08 20:26:06.107181 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25ae300
2012/10/08 20:26:06.946302 [11383]: Added timed event "ctdb_statistics_update": 0x25a9fe0
2012/10/08 20:26:06.946318 [11383]: Ending timer event 0x25adb70 "ctdb_statistics_update"
2012/10/08 20:26:07.094374 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25adb70
2012/10/08 20:26:07.094401 [11383]: Ending timer event 0x25b6be0 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:07.103452 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:07.103470 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:07.103484 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b42b0
2012/10/08 20:26:07.103493 [11383]: Ending timer event 0x25b1fb0 "ctdb_wait_until_recovered"
2012/10/08 20:26:07.103503 [11383]: Added timed event "ctdb_time_tick": 0x25b1fb0
2012/10/08 20:26:07.103512 [11383]: Ending timer event 0x25b2810 "ctdb_time_tick"
2012/10/08 20:26:07.107561 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:07.107597 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b43d0
2012/10/08 20:26:07.107609 [11383]: Ending timer event 0x25ae300 "ctdb_tcp_node_connect"
2012/10/08 20:26:07.107874 [11383]: Destroying timer event 0x25b43d0 "ctdb_tcp_node_connect"
2012/10/08 20:26:07.107895 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2810
2012/10/08 20:26:07.947127 [11383]: Added timed event "ctdb_statistics_update": 0x25b5fb0
2012/10/08 20:26:07.947142 [11383]: Ending timer event 0x25a9fe0 "ctdb_statistics_update"
2012/10/08 20:26:08.104213 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:08.104227 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:08.104239 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b6140
2012/10/08 20:26:08.104247 [11383]: Ending timer event 0x25b42b0 "ctdb_wait_until_recovered"
2012/10/08 20:26:08.104257 [11383]: Added timed event "ctdb_time_tick": 0x25b42b0
2012/10/08 20:26:08.104265 [11383]: Ending timer event 0x25b1fb0 "ctdb_time_tick"
2012/10/08 20:26:08.108318 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:08.108350 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b66a0
2012/10/08 20:26:08.108361 [11383]: Ending timer event 0x25b2810 "ctdb_tcp_node_connect"
2012/10/08 20:26:08.108635 [11383]: Destroying timer event 0x25b66a0 "ctdb_tcp_node_connect"
2012/10/08 20:26:08.108663 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b66a0
2012/10/08 20:26:08.947722 [11383]: Added timed event "ctdb_statistics_update": 0x25b6830
2012/10/08 20:26:08.947739 [11383]: Ending timer event 0x25b5fb0 "ctdb_statistics_update"
2012/10/08 20:26:09.104795 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:09.104809 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:09.104827 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b5850
2012/10/08 20:26:09.104837 [11383]: Ending timer event 0x25b6140 "ctdb_wait_until_recovered"
2012/10/08 20:26:09.104846 [11383]: Added timed event "ctdb_time_tick": 0x25b6140
2012/10/08 20:26:09.104855 [11383]: Ending timer event 0x25b42b0 "ctdb_time_tick"
2012/10/08 20:26:09.108910 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:09.108941 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5970
2012/10/08 20:26:09.108953 [11383]: Ending timer event 0x25b66a0 "ctdb_tcp_node_connect"
2012/10/08 20:26:09.109289 [11383]: Destroying timer event 0x25b5970 "ctdb_tcp_node_connect"
2012/10/08 20:26:09.109310 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5970
2012/10/08 20:26:09.948364 [11383]: Added timed event "ctdb_statistics_update": 0x25b6e00
2012/10/08 20:26:09.948381 [11383]: Ending timer event 0x25b6830 "ctdb_statistics_update"
2012/10/08 20:26:10.105429 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:10.105444 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:10.105455 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b7000
2012/10/08 20:26:10.105464 [11383]: Ending timer event 0x25b5850 "ctdb_wait_until_recovered"
2012/10/08 20:26:10.105474 [11383]: Added timed event "ctdb_time_tick": 0x25b5850
2012/10/08 20:26:10.105482 [11383]: Ending timer event 0x25b6140 "ctdb_time_tick"
2012/10/08 20:26:10.109539 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:10.109570 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b62f0
2012/10/08 20:26:10.109582 [11383]: Ending timer event 0x25b5970 "ctdb_tcp_node_connect"
2012/10/08 20:26:10.109881 [11383]: Destroying timer event 0x25b62f0 "ctdb_tcp_node_connect"
2012/10/08 20:26:10.109901 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b62f0
2012/10/08 20:26:10.949031 [11383]: Added timed event "ctdb_statistics_update": 0x25b6480
2012/10/08 20:26:10.949091 [11383]: Ending timer event 0x25b6e00 "ctdb_statistics_update"
2012/10/08 20:26:11.106347 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:11.106366 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:11.106380 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b5c00
2012/10/08 20:26:11.106389 [11383]: Ending timer event 0x25b7000 "ctdb_wait_until_recovered"
2012/10/08 20:26:11.106399 [11383]: Added timed event "ctdb_time_tick": 0x25b7000
2012/10/08 20:26:11.106407 [11383]: Ending timer event 0x25b5850 "ctdb_time_tick"
2012/10/08 20:26:11.110480 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:11.110546 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5d20
2012/10/08 20:26:11.110558 [11383]: Ending timer event 0x25b62f0 "ctdb_tcp_node_connect"
2012/10/08 20:26:11.110895 [11383]: Destroying timer event 0x25b5d20 "ctdb_tcp_node_connect"
2012/10/08 20:26:11.110926 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5d20
2012/10/08 20:26:11.949996 [11383]: Added timed event "ctdb_statistics_update": 0x25b5350
2012/10/08 20:26:11.950036 [11383]: Ending timer event 0x25b6480 "ctdb_statistics_update"
2012/10/08 20:26:12.093106 [11383]: Added timed event "ctdb_update_tcp_tickles": 0x25b6480
2012/10/08 20:26:12.093125 [11383]: Ending timer event 0x25adff0 "ctdb_update_tcp_tickles"
2012/10/08 20:26:12.095179 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25adff0
2012/10/08 20:26:12.095195 [11383]: Ending timer event 0x25adb70 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:12.107264 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:12.107282 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:12.107295 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b1a30
2012/10/08 20:26:12.107303 [11383]: Ending timer event 0x25b5c00 "ctdb_wait_until_recovered"
2012/10/08 20:26:12.107313 [11383]: Added timed event "ctdb_time_tick": 0x25b5c00
2012/10/08 20:26:12.107322 [11383]: Ending timer event 0x25b7000 "ctdb_time_tick"
2012/10/08 20:26:12.111393 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:12.111450 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1b50
2012/10/08 20:26:12.111462 [11383]: Ending timer event 0x25b5d20 "ctdb_tcp_node_connect"
2012/10/08 20:26:12.111793 [11383]: Destroying timer event 0x25b1b50 "ctdb_tcp_node_connect"
2012/10/08 20:26:12.111820 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1b50
2012/10/08 20:26:12.950885 [11383]: Added timed event "ctdb_statistics_update": 0x25b1ce0
2012/10/08 20:26:12.950917 [11383]: Ending timer event 0x25b5350 "ctdb_statistics_update"
2012/10/08 20:26:13.107969 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:13.107987 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:13.108000 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b4e40
2012/10/08 20:26:13.108009 [11383]: Ending timer event 0x25b1a30 "ctdb_wait_until_recovered"
2012/10/08 20:26:13.108019 [11383]: Added timed event "ctdb_time_tick": 0x25b1a30
2012/10/08 20:26:13.108027 [11383]: Ending timer event 0x25b5c00 "ctdb_time_tick"
2012/10/08 20:26:13.112090 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:13.112138 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b4f60
2012/10/08 20:26:13.112150 [11383]: Ending timer event 0x25b1b50 "ctdb_tcp_node_connect"
2012/10/08 20:26:13.112457 [11383]: Destroying timer event 0x25b4f60 "ctdb_tcp_node_connect"
2012/10/08 20:26:13.112483 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b4f60
2012/10/08 20:26:13.951540 [11383]: Added timed event "ctdb_statistics_update": 0x25b50f0
2012/10/08 20:26:13.951566 [11383]: Ending timer event 0x25b1ce0 "ctdb_statistics_update"
2012/10/08 20:26:14.108617 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:14.108634 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:14.108646 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b2160
2012/10/08 20:26:14.108655 [11383]: Ending timer event 0x25b4e40 "ctdb_wait_until_recovered"
2012/10/08 20:26:14.108665 [11383]: Added timed event "ctdb_time_tick": 0x25b4e40
2012/10/08 20:26:14.108674 [11383]: Ending timer event 0x25b1a30 "ctdb_time_tick"
2012/10/08 20:26:14.112735 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:14.112775 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2280
2012/10/08 20:26:14.112787 [11383]: Ending timer event 0x25b4f60 "ctdb_tcp_node_connect"
2012/10/08 20:26:14.113087 [11383]: Destroying timer event 0x25b2280 "ctdb_tcp_node_connect"
2012/10/08 20:26:14.113111 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2280
2012/10/08 20:26:14.952179 [11383]: Added timed event "ctdb_statistics_update": 0x25b2410
2012/10/08 20:26:14.952202 [11383]: Ending timer event 0x25b50f0 "ctdb_statistics_update"
2012/10/08 20:26:15.109253 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:15.109272 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:15.109284 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b2610
2012/10/08 20:26:15.109293 [11383]: Ending timer event 0x25b2160 "ctdb_wait_until_recovered"
2012/10/08 20:26:15.109303 [11383]: Added timed event "ctdb_time_tick": 0x25b2160
2012/10/08 20:26:15.109312 [11383]: Ending timer event 0x25b4e40 "ctdb_time_tick"
2012/10/08 20:26:15.113374 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:15.113434 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b98b0
2012/10/08 20:26:15.113446 [11383]: Ending timer event 0x25b2280 "ctdb_tcp_node_connect"
2012/10/08 20:26:15.113756 [11383]: Destroying timer event 0x25b98b0 "ctdb_tcp_node_connect"
2012/10/08 20:26:15.113780 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b98b0
2012/10/08 20:26:15.952851 [11383]: Added timed event "ctdb_statistics_update": 0x25b9a40
2012/10/08 20:26:15.952885 [11383]: Ending timer event 0x25b2410 "ctdb_statistics_update"
2012/10/08 20:26:15.989489 [recoverd:11454]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:26:15.989562 [11383]: server/ctdb_daemon.c:628 client request 2147483556 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.989583 [11383]: Added timed event "queue_next_trigger": 0x25b9f80
2012/10/08 20:26:15.989594 [11383]: Added timed event "ctdb_control_timeout": 0x25ba0a0
2012/10/08 20:26:15.989605 [11383]: server/ctdb_server.c:331 ctdb request 2147483563 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.989616 [11383]: Destroying timer event 0x25ae1e0 "ctdb_recd_ping_timeout"
2012/10/08 20:26:15.989626 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:26:15.989640 [11383]: Added timed event "queue_next_trigger": 0x25ba3c0
2012/10/08 20:26:15.989649 [11383]: Ending timer event 0x25b9f80 "queue_next_trigger"
2012/10/08 20:26:15.989659 [11383]: server/ctdb_server.c:331 ctdb request 2147483563 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.989674 [11383]: Destroying timer event 0x25ba0a0 "ctdb_control_timeout"
2012/10/08 20:26:15.989683 [11383]: Ending timer event 0x25ba3c0 "queue_next_trigger"
2012/10/08 20:26:15.989725 [11383]: server/ctdb_daemon.c:628 client request 2147483557 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.989740 [11383]: Added timed event "queue_next_trigger": 0x25ba340
2012/10/08 20:26:15.989750 [11383]: Added timed event "ctdb_control_timeout": 0x25b7290
2012/10/08 20:26:15.989760 [11383]: server/ctdb_server.c:331 ctdb request 2147483564 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.989771 [11383]: Added timed event "queue_next_trigger": 0x25b7440
2012/10/08 20:26:15.989781 [11383]: Ending timer event 0x25ba340 "queue_next_trigger"
2012/10/08 20:26:15.989790 [11383]: server/ctdb_server.c:331 ctdb request 2147483564 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.989803 [11383]: Destroying timer event 0x25b7290 "ctdb_control_timeout"
2012/10/08 20:26:15.989813 [11383]: Ending timer event 0x25b7440 "queue_next_trigger"
2012/10/08 20:26:15.989853 [11383]: server/ctdb_daemon.c:628 client request 2147483558 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.989867 [11383]: Added timed event "queue_next_trigger": 0x25ba340
2012/10/08 20:26:15.989877 [11383]: Added timed event "ctdb_control_timeout": 0x25b9f20
2012/10/08 20:26:15.989887 [11383]: server/ctdb_server.c:331 ctdb request 2147483565 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.989899 [11383]: Added timed event "queue_next_trigger": 0x25b78c0
2012/10/08 20:26:15.989909 [11383]: Ending timer event 0x25ba340 "queue_next_trigger"
2012/10/08 20:26:15.989918 [11383]: server/ctdb_server.c:331 ctdb request 2147483565 of type 8 length 228 from node 1 to 1
2012/10/08 20:26:15.989931 [11383]: Destroying timer event 0x25b9f20 "ctdb_control_timeout"
2012/10/08 20:26:15.989941 [11383]: Ending timer event 0x25b78c0 "queue_next_trigger"
2012/10/08 20:26:15.989981 [11383]: server/ctdb_daemon.c:628 client request 2147483559 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.989996 [11383]: Added timed event "queue_next_trigger": 0x25b7640
2012/10/08 20:26:15.990005 [11383]: Added timed event "ctdb_control_timeout": 0x25b7760
2012/10/08 20:26:15.990015 [11383]: server/ctdb_server.c:331 ctdb request 2147483566 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990027 [11383]: Added timed event "queue_next_trigger": 0x25b7af0
2012/10/08 20:26:15.990036 [11383]: Ending timer event 0x25b7640 "queue_next_trigger"
2012/10/08 20:26:15.990059 [11383]: server/ctdb_server.c:331 ctdb request 2147483566 of type 8 length 67 from node 1 to 1
2012/10/08 20:26:15.990072 [11383]: Destroying timer event 0x25b7760 "ctdb_control_timeout"
2012/10/08 20:26:15.990082 [11383]: Ending timer event 0x25b7af0 "queue_next_trigger"
2012/10/08 20:26:15.990124 [11383]: server/ctdb_daemon.c:628 client request 2147483560 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.990138 [11383]: Added timed event "queue_next_trigger": 0x25b7a50
2012/10/08 20:26:15.990147 [11383]: Added timed event "ctdb_control_timeout": 0x25ba310
2012/10/08 20:26:15.990157 [11383]: server/ctdb_server.c:331 ctdb request 2147483567 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990169 [11383]: Added timed event "queue_next_trigger": 0x25b7e10
2012/10/08 20:26:15.990178 [11383]: Ending timer event 0x25b7a50 "queue_next_trigger"
2012/10/08 20:26:15.990187 [11383]: server/ctdb_server.c:331 ctdb request 2147483567 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.990259 [11383]: Destroying timer event 0x25ba310 "ctdb_control_timeout"
2012/10/08 20:26:15.990270 [11383]: Ending timer event 0x25b7e10 "queue_next_trigger"
2012/10/08 20:26:15.990311 [11383]: server/ctdb_daemon.c:628 client request 2147483561 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990325 [11383]: Added timed event "queue_next_trigger": 0x25b7cf0
2012/10/08 20:26:15.990335 [11383]: Added timed event "ctdb_control_timeout": 0x25b7e10
2012/10/08 20:26:15.990345 [11383]: server/ctdb_server.c:331 ctdb request 2147483568 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990357 [11383]: Added timed event "queue_next_trigger": 0x25b81a0
2012/10/08 20:26:15.990366 [11383]: Ending timer event 0x25b7cf0 "queue_next_trigger"
2012/10/08 20:26:15.990376 [11383]: server/ctdb_server.c:331 ctdb request 2147483568 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:15.990389 [11383]: Destroying timer event 0x25b7e10 "ctdb_control_timeout"
2012/10/08 20:26:15.990398 [11383]: Ending timer event 0x25b81a0 "queue_next_trigger"
2012/10/08 20:26:15.990439 [11383]: server/ctdb_daemon.c:628 client request 2147483562 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990453 [11383]: Added timed event "queue_next_trigger": 0x25b7e10
2012/10/08 20:26:15.990463 [11383]: Added timed event "ctdb_control_timeout": 0x25b8110
2012/10/08 20:26:15.990472 [11383]: server/ctdb_server.c:331 ctdb request 2147483569 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990485 [11383]: Added timed event "queue_next_trigger": 0x25b84d0
2012/10/08 20:26:15.990495 [11383]: Ending timer event 0x25b7e10 "queue_next_trigger"
2012/10/08 20:26:15.990504 [11383]: server/ctdb_server.c:331 ctdb request 2147483569 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:15.990517 [11383]: Destroying timer event 0x25b8110 "ctdb_control_timeout"
2012/10/08 20:26:15.990526 [11383]: Ending timer event 0x25b84d0 "queue_next_trigger"
2012/10/08 20:26:15.990567 [11383]: server/ctdb_daemon.c:628 client request 2147483563 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990581 [11383]: Added timed event "queue_next_trigger": 0x25b7640
2012/10/08 20:26:15.990590 [11383]: Added timed event "ctdb_control_timeout": 0x25b83a0
2012/10/08 20:26:15.990600 [11383]: server/ctdb_server.c:331 ctdb request 2147483570 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990611 [11383]: Added timed event "queue_next_trigger": 0x25b86f0
2012/10/08 20:26:15.990620 [11383]: Ending timer event 0x25b7640 "queue_next_trigger"
2012/10/08 20:26:15.990630 [11383]: server/ctdb_server.c:331 ctdb request 2147483570 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.990643 [11383]: Destroying timer event 0x25b83a0 "ctdb_control_timeout"
2012/10/08 20:26:15.990652 [11383]: Ending timer event 0x25b86f0 "queue_next_trigger"
2012/10/08 20:26:15.990692 [11383]: server/ctdb_daemon.c:628 client request 2147483564 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990706 [11383]: Added timed event "queue_next_trigger": 0x25b84c0
2012/10/08 20:26:15.990722 [11383]: Added timed event "ctdb_control_timeout": 0x25b86d0
2012/10/08 20:26:15.990733 [11383]: server/ctdb_server.c:331 ctdb request 2147483571 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990745 [11383]: Added timed event "queue_next_trigger": 0x25b8b40
2012/10/08 20:26:15.990754 [11383]: Ending timer event 0x25b84c0 "queue_next_trigger"
2012/10/08 20:26:15.990764 [11383]: server/ctdb_server.c:331 ctdb request 2147483571 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:15.990777 [11383]: Destroying timer event 0x25b86d0 "ctdb_control_timeout"
2012/10/08 20:26:15.990786 [11383]: Ending timer event 0x25b8b40 "queue_next_trigger"
2012/10/08 20:26:15.990827 [11383]: server/ctdb_daemon.c:628 client request 2147483565 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.990841 [11383]: Added timed event "queue_next_trigger": 0x25b88f0
2012/10/08 20:26:15.990851 [11383]: Added timed event "ctdb_control_timeout": 0x25b8a10
2012/10/08 20:26:15.990861 [11383]: server/ctdb_server.c:331 ctdb request 2147483572 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.990873 [11383]: Added timed event "queue_next_trigger": 0x25b8e70
2012/10/08 20:26:15.990882 [11383]: Ending timer event 0x25b88f0 "queue_next_trigger"
2012/10/08 20:26:15.990892 [11383]: server/ctdb_server.c:331 ctdb request 2147483572 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:15.990905 [11383]: Destroying timer event 0x25b8a10 "ctdb_control_timeout"
2012/10/08 20:26:15.990915 [11383]: Ending timer event 0x25b8e70 "queue_next_trigger"
2012/10/08 20:26:15.990954 [11383]: server/ctdb_daemon.c:628 client request 2147483566 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.990968 [11383]: Added timed event "queue_next_trigger": 0x25b86d0
2012/10/08 20:26:15.990978 [11383]: Added timed event "ctdb_control_timeout": 0x25b8d40
2012/10/08 20:26:15.990987 [11383]: server/ctdb_server.c:331 ctdb request 2147483573 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991000 [11383]: Added timed event "queue_next_trigger": 0x25b90a0
2012/10/08 20:26:15.991009 [11383]: Ending timer event 0x25b86d0 "queue_next_trigger"
2012/10/08 20:26:15.991018 [11383]: server/ctdb_server.c:331 ctdb request 2147483573 of type 8 length 72 from node 1 to 1
2012/10/08 20:26:15.991031 [11383]: Destroying timer event 0x25b8d40 "ctdb_control_timeout"
2012/10/08 20:26:15.991041 [11383]: Ending timer event 0x25b90a0 "queue_next_trigger"
2012/10/08 20:26:15.991103 [11383]: server/ctdb_daemon.c:628 client request 2147483567 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.991118 [11383]: Added timed event "queue_next_trigger": 0x25b86d0
2012/10/08 20:26:15.991128 [11383]: Added timed event "ctdb_control_timeout": 0x25b9070
2012/10/08 20:26:15.991138 [11383]: server/ctdb_server.c:331 ctdb request 2147483574 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991150 [11383]: Added timed event "queue_next_trigger": 0x25b9400
2012/10/08 20:26:15.991159 [11383]: Ending timer event 0x25b86d0 "queue_next_trigger"
2012/10/08 20:26:15.991168 [11383]: server/ctdb_server.c:331 ctdb request 2147483574 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:15.991181 [11383]: Destroying timer event 0x25b9070 "ctdb_control_timeout"
2012/10/08 20:26:15.991191 [11383]: Ending timer event 0x25b9400 "queue_next_trigger"
2012/10/08 20:26:15.991231 [11383]: server/ctdb_daemon.c:628 client request 2147483568 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:15.991245 [11383]: Added timed event "queue_next_trigger": 0x25b9590
2012/10/08 20:26:15.991254 [11383]: Added timed event "ctdb_control_timeout": 0x25b96b0
2012/10/08 20:26:15.991264 [11383]: server/ctdb_server.c:331 ctdb request 2147483575 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991276 [11383]: Added timed event "queue_next_trigger": 0x25bf860
2012/10/08 20:26:15.991286 [11383]: Ending timer event 0x25b9590 "queue_next_trigger"
2012/10/08 20:26:15.991295 [11383]: server/ctdb_server.c:331 ctdb request 2147483575 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:15.991315 [11383]: Destroying timer event 0x25b96b0 "ctdb_control_timeout"
2012/10/08 20:26:15.991325 [11383]: Ending timer event 0x25bf860 "queue_next_trigger"
2012/10/08 20:26:15.991386 [11383]: server/ctdb_daemon.c:628 client request 2147483569 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991400 [11383]: Added timed event "queue_next_trigger": 0x25bf790
2012/10/08 20:26:15.991410 [11383]: Added timed event "ctdb_control_timeout": 0x25b9310
2012/10/08 20:26:15.991420 [11383]: server/ctdb_server.c:331 ctdb request 2147483576 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991431 [11383]: Added timed event "queue_next_trigger": 0x25bfaf0
2012/10/08 20:26:15.991440 [11383]: Ending timer event 0x25bf790 "queue_next_trigger"
2012/10/08 20:26:15.991450 [11383]: server/ctdb_server.c:331 ctdb request 2147483576 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.991462 [11383]: Destroying timer event 0x25b9310 "ctdb_control_timeout"
2012/10/08 20:26:15.991472 [11383]: Ending timer event 0x25bfaf0 "queue_next_trigger"
2012/10/08 20:26:15.991513 [11383]: server/ctdb_daemon.c:628 client request 2147483570 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991527 [11383]: Added timed event "queue_next_trigger": 0x25b96b0
2012/10/08 20:26:15.991537 [11383]: Added timed event "ctdb_control_timeout": 0x25bfad0
2012/10/08 20:26:15.991547 [11383]: server/ctdb_server.c:331 ctdb request 2147483577 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:15.991558 [11383]: Added timed event "queue_next_trigger": 0x25bfe10
2012/10/08 20:26:15.991567 [11383]: Ending timer event 0x25b96b0 "queue_next_trigger"
2012/10/08 20:26:15.991576 [11383]: server/ctdb_server.c:331 ctdb request 2147483577 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.991589 [11383]: Destroying timer event 0x25bfad0 "ctdb_control_timeout"
2012/10/08 20:26:15.991599 [11383]: Ending timer event 0x25bfe10 "queue_next_trigger"
2012/10/08 20:26:15.992007 [recoverd:11454]: server/ctdb_recoverd.c:2810 Created PIPE FD:17 for check_recovery_lock
2012/10/08 20:26:15.994448 [recoverd:11454]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:26:15.994463 [recoverd:11454]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:26:15.995697 [11383]: server/ctdb_daemon.c:628 client request 2147483571 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:15.995713 [11383]: Added timed event "queue_next_trigger": 0x25bfcf0
2012/10/08 20:26:15.995723 [11383]: Added timed event "ctdb_control_timeout": 0x25bfe10
2012/10/08 20:26:15.995734 [11383]: server/ctdb_server.c:331 ctdb request 2147483578 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:15.995745 [11383]: Added timed event "queue_next_trigger": 0x25c0130
2012/10/08 20:26:15.995755 [11383]: Ending timer event 0x25bfcf0 "queue_next_trigger"
2012/10/08 20:26:15.995764 [11383]: server/ctdb_server.c:331 ctdb request 2147483578 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:15.995778 [11383]: Destroying timer event 0x25bfe10 "ctdb_control_timeout"
2012/10/08 20:26:15.995788 [11383]: Ending timer event 0x25c0130 "queue_next_trigger"
2012/10/08 20:26:15.995858 [recoverd:11454]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:26:15.995870 [recoverd:11454]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:26:15.995879 [recoverd:11454]: Taking out recovery lock from recovery daemon
2012/10/08 20:26:15.995887 [recoverd:11454]: Take the recovery lock
2012/10/08 20:26:16.001480 [recoverd:11454]: Recovery lock taken successfully
2012/10/08 20:26:16.001495 [recoverd:11454]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:26:16.001524 [11383]: server/ctdb_daemon.c:628 client request 2147483572 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:16.001539 [11383]: Added timed event "queue_next_trigger": 0x25c0010
2012/10/08 20:26:16.001549 [11383]: Added timed event "ctdb_control_timeout": 0x25c0130
2012/10/08 20:26:16.001566 [11383]: server/ctdb_server.c:331 ctdb request 2147483579 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:16.001578 [11383]: Added timed event "queue_next_trigger": 0x25c03e0
2012/10/08 20:26:16.001588 [11383]: Ending timer event 0x25c0010 "queue_next_trigger"
2012/10/08 20:26:16.001597 [11383]: server/ctdb_server.c:331 ctdb request 2147483579 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.001611 [11383]: Destroying timer event 0x25c0130 "ctdb_control_timeout"
2012/10/08 20:26:16.001621 [11383]: Ending timer event 0x25c03e0 "queue_next_trigger"
2012/10/08 20:26:16.001652 [recoverd:11454]: RECD SIGCHLD from 11908
2012/10/08 20:26:16.001671 [recoverd:11454]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:26:16.001681 [recoverd:11454]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 1
2012/10/08 20:26:16.001706 [11383]: server/ctdb_daemon.c:628 client request 2147483573 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.001720 [11383]: Added timed event "queue_next_trigger": 0x25c02c0
2012/10/08 20:26:16.001730 [11383]: Added timed event "ctdb_control_timeout": 0x25c03e0
2012/10/08 20:26:16.001740 [11383]: server/ctdb_server.c:331 ctdb request 2147483580 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.001751 [11383]: Added timed event "queue_next_trigger": 0x25c0700
2012/10/08 20:26:16.001761 [11383]: Ending timer event 0x25c02c0 "queue_next_trigger"
2012/10/08 20:26:16.001771 [11383]: server/ctdb_server.c:331 ctdb request 2147483580 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.001784 [11383]: Destroying timer event 0x25c03e0 "ctdb_control_timeout"
2012/10/08 20:26:16.001794 [11383]: Ending timer event 0x25c0700 "queue_next_trigger"
2012/10/08 20:26:16.001821 [recoverd:11454]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:26:16.001833 [recoverd:11454]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:26:16.001860 [11383]: server/ctdb_daemon.c:628 client request 2147483574 of type 7 length 88 from node 1 to 1
2012/10/08 20:26:16.001874 [11383]: Added timed event "queue_next_trigger": 0x25c05e0
2012/10/08 20:26:16.001883 [11383]: Added timed event "ctdb_control_timeout": 0x25c0700
2012/10/08 20:26:16.001893 [11383]: server/ctdb_server.c:331 ctdb request 2147483581 of type 7 length 83 from node 1 to 1
2012/10/08 20:26:16.001905 [11383]: Added timed event "queue_next_trigger": 0x25c0a20
2012/10/08 20:26:16.001914 [11383]: Ending timer event 0x25c05e0 "queue_next_trigger"
2012/10/08 20:26:16.001923 [11383]: server/ctdb_server.c:331 ctdb request 2147483581 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.001936 [11383]: Destroying timer event 0x25c0700 "ctdb_control_timeout"
2012/10/08 20:26:16.001946 [11383]: Ending timer event 0x25c0a20 "queue_next_trigger"
2012/10/08 20:26:16.001988 [11383]: server/ctdb_daemon.c:628 client request 2147483575 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.002009 [11383]: Added timed event "queue_next_trigger": 0x25c0900
2012/10/08 20:26:16.002020 [11383]: Added timed event "ctdb_control_timeout": 0x25c0a20
2012/10/08 20:26:16.002030 [11383]: server/ctdb_server.c:331 ctdb request 2147483582 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.002040 [11383]: Freeze priority 1
2012/10/08 20:26:16.002574 [11383]: Ending timer event 0x25c0900 "queue_next_trigger"
2012/10/08 20:26:16.002933 [11383]: Added timed event "queue_next_trigger": 0x25c0dd0
2012/10/08 20:26:16.002948 [11383]: server/ctdb_server.c:331 ctdb request 2147483582 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.003027 [11383]: Destroying timer event 0x25c0a20 "ctdb_control_timeout"
2012/10/08 20:26:16.003055 [11383]: Ending timer event 0x25c0dd0 "queue_next_trigger"
2012/10/08 20:26:16.003101 [11383]: server/ctdb_daemon.c:628 client request 2147483576 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.003116 [11383]: Added timed event "queue_next_trigger": 0x25c0d40
2012/10/08 20:26:16.003126 [11383]: Added timed event "ctdb_control_timeout": 0x25c0f60
2012/10/08 20:26:16.003143 [11383]: server/ctdb_server.c:331 ctdb request 2147483583 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.003153 [11383]: Freeze priority 2
2012/10/08 20:26:16.003633 [11383]: Ending timer event 0x25c0d40 "queue_next_trigger"
2012/10/08 20:26:16.004037 [11383]: Added timed event "queue_next_trigger": 0x25c1230
2012/10/08 20:26:16.004051 [11383]: server/ctdb_server.c:331 ctdb request 2147483583 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.004125 [11383]: Destroying timer event 0x25c0f60 "ctdb_control_timeout"
2012/10/08 20:26:16.004153 [11383]: Ending timer event 0x25c1230 "queue_next_trigger"
2012/10/08 20:26:16.004199 [11383]: server/ctdb_daemon.c:628 client request 2147483577 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.004213 [11383]: Added timed event "queue_next_trigger": 0x25c11a0
2012/10/08 20:26:16.004223 [11383]: Added timed event "ctdb_control_timeout": 0x25c13c0
2012/10/08 20:26:16.004233 [11383]: server/ctdb_server.c:331 ctdb request 2147483584 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.004243 [11383]: Freeze priority 3
2012/10/08 20:26:16.004735 [11383]: Ending timer event 0x25c11a0 "queue_next_trigger"
2012/10/08 20:26:16.005131 [11383]: Added timed event "queue_next_trigger": 0x25c1690
2012/10/08 20:26:16.005161 [11383]: server/ctdb_server.c:331 ctdb request 2147483584 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.005222 [11383]: Destroying timer event 0x25c13c0 "ctdb_control_timeout"
2012/10/08 20:26:16.005250 [11383]: Ending timer event 0x25c1690 "queue_next_trigger"
2012/10/08 20:26:16.005297 [11383]: server/ctdb_daemon.c:628 client request 2147483578 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.005312 [11383]: Added timed event "queue_next_trigger": 0x25c1600
2012/10/08 20:26:16.005322 [11383]: Added timed event "ctdb_control_timeout": 0x25c1820
2012/10/08 20:26:16.005332 [11383]: server/ctdb_server.c:331 ctdb request 2147483585 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.005343 [11383]: Added timed event "ctdb_drop_all_ips_event": 0x25c1940
2012/10/08 20:26:16.005351 [11383]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:26:16.005361 [11383]: Added timed event "queue_next_trigger": 0x25c1bf0
2012/10/08 20:26:16.005370 [11383]: Ending timer event 0x25c1600 "queue_next_trigger"
2012/10/08 20:26:16.005379 [11383]: server/ctdb_server.c:331 ctdb request 2147483585 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.005392 [11383]: Destroying timer event 0x25c1820 "ctdb_control_timeout"
2012/10/08 20:26:16.005402 [11383]: Ending timer event 0x25c1bf0 "queue_next_trigger"
2012/10/08 20:26:16.005443 [11383]: server/ctdb_daemon.c:628 client request 2147483579 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.005492 [11383]: Added timed event "queue_next_trigger": 0x25c1ad0
2012/10/08 20:26:16.005504 [11383]: Added timed event "ctdb_control_timeout": 0x25c1bf0
2012/10/08 20:26:16.005515 [11383]: server/ctdb_server.c:331 ctdb request 2147483586 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.005524 [11383]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:26:16.005534 [11383]: Monitoring has been disabled
2012/10/08 20:26:16.005561 [11383]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/08 20:26:16.005748 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:16.005785 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:16.005800 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:16.005818 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:16.006353 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.006481 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ade00
2012/10/08 20:26:16.006516 [11383]: Ending timer event 0x25c1ad0 "queue_next_trigger"
2012/10/08 20:26:16.006974 [eventscript-00.ctdb-startrecovery:11912]: Executing event script /etc/ctdb/events.d/00.ctdb startrecovery
2012/10/08 20:26:16.019474 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.019997 [11383]: SIGCHLD from 11912
2012/10/08 20:26:16.020265 [eventscript-01.reclock-startrecovery:11917]: Executing event script /etc/ctdb/events.d/01.reclock startrecovery
2012/10/08 20:26:16.029540 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.030076 [11383]: SIGCHLD from 11917
2012/10/08 20:26:16.030393 [eventscript-10.interface-startrecovery:11921]: Executing event script /etc/ctdb/events.d/10.interface startrecovery
2012/10/08 20:26:16.040263 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.040780 [11383]: SIGCHLD from 11921
2012/10/08 20:26:16.041060 [eventscript-11.natgw-startrecovery:11925]: Executing event script /etc/ctdb/events.d/11.natgw startrecovery
2012/10/08 20:26:16.050127 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.050676 [11383]: SIGCHLD from 11925
2012/10/08 20:26:16.050974 [eventscript-11.routing-startrecovery:11929]: Executing event script /etc/ctdb/events.d/11.routing startrecovery
2012/10/08 20:26:16.060047 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.060550 [11383]: SIGCHLD from 11929
2012/10/08 20:26:16.060825 [eventscript-13.per_ip_routing-startrecovery:11933]: Executing event script /etc/ctdb/events.d/13.per_ip_routing startrecovery
2012/10/08 20:26:16.072910 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.073421 [11383]: SIGCHLD from 11933
2012/10/08 20:26:16.073722 [eventscript-20.multipathd-startrecovery:11938]: Executing event script /etc/ctdb/events.d/20.multipathd startrecovery
2012/10/08 20:26:16.074279 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.074999 [eventscript-31.clamd-startrecovery:11939]: Executing event script /etc/ctdb/events.d/31.clamd startrecovery
2012/10/08 20:26:16.075569 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.075888 [11383]: SIGCHLD from 11938
2012/10/08 20:26:16.076204 [eventscript-40.vsftpd-startrecovery:11940]: Executing event script /etc/ctdb/events.d/40.vsftpd startrecovery
2012/10/08 20:26:16.081847 [11383]: SIGCHLD from 11939
2012/10/08 20:26:16.085762 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.086294 [11383]: SIGCHLD from 11940
2012/10/08 20:26:16.086577 [eventscript-40.fs_use-startrecovery:11944]: Executing event script /etc/ctdb/events.d/40.fs_use startrecovery
2012/10/08 20:26:16.087133 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.087868 [eventscript-41.httpd-startrecovery:11945]: Executing event script /etc/ctdb/events.d/41.httpd startrecovery
2012/10/08 20:26:16.089877 [11383]: SIGCHLD from 11944
2012/10/08 20:26:16.097301 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.097810 [11383]: SIGCHLD from 11945
2012/10/08 20:26:16.098089 [eventscript-50.samba-startrecovery:11949]: Executing event script /etc/ctdb/events.d/50.samba startrecovery
2012/10/08 20:26:16.109858 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:16.109887 [11383]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/08 20:26:16.109901 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25ad7c0
2012/10/08 20:26:16.109909 [11383]: Ending timer event 0x25b2610 "ctdb_wait_until_recovered"
2012/10/08 20:26:16.109944 [11383]: Added timed event "ctdb_time_tick": 0x25b6530
2012/10/08 20:26:16.109955 [11383]: Ending timer event 0x25b2160 "ctdb_time_tick"
2012/10/08 20:26:16.111156 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.111651 [11383]: SIGCHLD from 11949
2012/10/08 20:26:16.111923 [eventscript-60.nfs-startrecovery:11954]: Executing event script /etc/ctdb/events.d/60.nfs startrecovery
2012/10/08 20:26:16.114710 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:18
2012/10/08 20:26:16.114777 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25c1ca0
2012/10/08 20:26:16.114789 [11383]: Ending timer event 0x25b98b0 "ctdb_tcp_node_connect"
2012/10/08 20:26:16.117099 [11383]: Destroying timer event 0x25c1ca0 "ctdb_tcp_node_connect"
2012/10/08 20:26:16.117125 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b98b0
2012/10/08 20:26:16.126884 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.127409 [11383]: SIGCHLD from 11954
2012/10/08 20:26:16.127703 [eventscript-60.ganesha-startrecovery:11960]: Executing event script /etc/ctdb/events.d/60.ganesha startrecovery
2012/10/08 20:26:16.128309 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.129065 [eventscript-62.cnfs-startrecovery:11961]: Executing event script /etc/ctdb/events.d/62.cnfs startrecovery
2012/10/08 20:26:16.131249 [11383]: SIGCHLD from 11960
2012/10/08 20:26:16.141098 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.141616 [11383]: SIGCHLD from 11961
2012/10/08 20:26:16.141893 [eventscript-70.iscsi-startrecovery:11966]: Executing event script /etc/ctdb/events.d/70.iscsi startrecovery
2012/10/08 20:26:16.151001 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:16.151503 [11383]: SIGCHLD from 11966
2012/10/08 20:26:16.151780 [eventscript-91.lvs-startrecovery:11970]: Executing event script /etc/ctdb/events.d/91.lvs startrecovery
2012/10/08 20:26:16.160315 [11383]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/08 20:26:16.160410 [11383]: Added timed event "queue_next_trigger": 0x25adb70
2012/10/08 20:26:16.160428 [11383]: Destroying timer event 0x25ade00 "ctdb_event_script_timeout"
2012/10/08 20:26:16.160464 [11383]: server/ctdb_server.c:331 ctdb request 2147483586 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.160521 [11383]: Destroying timer event 0x25c1bf0 "ctdb_control_timeout"
2012/10/08 20:26:16.160533 [11383]: Ending timer event 0x25adb70 "queue_next_trigger"
2012/10/08 20:26:16.160594 [11383]: server/ctdb_daemon.c:628 client request 2147483580 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:16.160611 [11383]: Added timed event "queue_next_trigger": 0x25b9960
2012/10/08 20:26:16.160621 [11383]: Added timed event "ctdb_control_timeout": 0x25b1fb0
2012/10/08 20:26:16.160632 [11383]: server/ctdb_server.c:331 ctdb request 2147483587 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.160643 [11383]: Added timed event "queue_next_trigger": 0x25ae290
2012/10/08 20:26:16.160653 [11383]: Ending timer event 0x25b9960 "queue_next_trigger"
2012/10/08 20:26:16.160662 [11383]: server/ctdb_server.c:331 ctdb request 2147483587 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.160695 [11383]: Destroying timer event 0x25b1fb0 "ctdb_control_timeout"
2012/10/08 20:26:16.160705 [11383]: Ending timer event 0x25ae290 "queue_next_trigger"
2012/10/08 20:26:16.160747 [11383]: server/ctdb_daemon.c:628 client request 2147483581 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.160761 [11383]: Added timed event "queue_next_trigger": 0x25aa050
2012/10/08 20:26:16.160771 [11383]: Added timed event "ctdb_control_timeout": 0x25aeed0
2012/10/08 20:26:16.160781 [11383]: server/ctdb_server.c:331 ctdb request 2147483588 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.160794 [11383]: Added timed event "queue_next_trigger": 0x25b94b0
2012/10/08 20:26:16.160803 [11383]: Ending timer event 0x25aa050 "queue_next_trigger"
2012/10/08 20:26:16.160813 [11383]: server/ctdb_server.c:331 ctdb request 2147483588 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:16.160835 [11383]: Destroying timer event 0x25aeed0 "ctdb_control_timeout"
2012/10/08 20:26:16.160845 [11383]: Ending timer event 0x25b94b0 "queue_next_trigger"
2012/10/08 20:26:16.160886 [11383]: server/ctdb_daemon.c:628 client request 2147483582 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.160900 [11383]: Added timed event "queue_next_trigger": 0x25aeff0
2012/10/08 20:26:16.160910 [11383]: Added timed event "ctdb_control_timeout": 0x25b9640
2012/10/08 20:26:16.160920 [11383]: server/ctdb_server.c:331 ctdb request 2147483589 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.160931 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:16.160941 [11383]: Added timed event "queue_next_trigger": 0x25b43d0
2012/10/08 20:26:16.160950 [11383]: Ending timer event 0x25aeff0 "queue_next_trigger"
2012/10/08 20:26:16.160960 [11383]: server/ctdb_server.c:331 ctdb request 2147483589 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.160972 [11383]: Destroying timer event 0x25b9640 "ctdb_control_timeout"
2012/10/08 20:26:16.160982 [11383]: Ending timer event 0x25b43d0 "queue_next_trigger"
2012/10/08 20:26:16.161009 [recoverd:11454]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:26:16.161039 [11383]: server/ctdb_daemon.c:628 client request 2147483583 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.161053 [11383]: Added timed event "queue_next_trigger": 0x25b42b0
2012/10/08 20:26:16.161063 [11383]: Added timed event "ctdb_control_timeout": 0x25b43d0
2012/10/08 20:26:16.161073 [11383]: server/ctdb_server.c:331 ctdb request 2147483590 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.161110 [11383]: Added timed event "queue_next_trigger": 0x25b5e00
2012/10/08 20:26:16.161123 [11383]: Ending timer event 0x25b42b0 "queue_next_trigger"
2012/10/08 20:26:16.161132 [11383]: server/ctdb_server.c:331 ctdb request 2147483590 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.161146 [11383]: Destroying timer event 0x25b43d0 "ctdb_control_timeout"
2012/10/08 20:26:16.161155 [11383]: Ending timer event 0x25b5e00 "queue_next_trigger"
2012/10/08 20:26:16.161196 [11383]: server/ctdb_daemon.c:628 client request 2147483584 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.161210 [11383]: Added timed event "queue_next_trigger": 0x25b5ce0
2012/10/08 20:26:16.161220 [11383]: Added timed event "ctdb_control_timeout": 0x25b5e00
2012/10/08 20:26:16.161230 [11383]: server/ctdb_server.c:331 ctdb request 2147483591 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.161242 [11383]: Added timed event "queue_next_trigger": 0x25b7070
2012/10/08 20:26:16.161251 [11383]: Ending timer event 0x25b5ce0 "queue_next_trigger"
2012/10/08 20:26:16.161260 [11383]: server/ctdb_server.c:331 ctdb request 2147483591 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.161273 [11383]: Destroying timer event 0x25b5e00 "ctdb_control_timeout"
2012/10/08 20:26:16.161283 [11383]: Ending timer event 0x25b7070 "queue_next_trigger"
2012/10/08 20:26:16.161471 [11383]: SIGCHLD from 11970
2012/10/08 20:26:16.161504 [recoverd:11454]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:26:16.161517 [recoverd:11454]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:26:16.161551 [11383]: server/ctdb_daemon.c:628 client request 2147483585 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.161566 [11383]: Added timed event "queue_next_trigger": 0x25b6ff0
2012/10/08 20:26:16.161576 [11383]: Added timed event "ctdb_control_timeout": 0x25b58c0
2012/10/08 20:26:16.161586 [11383]: server/ctdb_server.c:331 ctdb request 2147483592 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.161596 [11383]: server/ctdb_freeze.c:523 num_nodes[2]
2012/10/08 20:26:16.161604 [11383]: server/ctdb_freeze.c:526 node[0].flags[0x3]
2012/10/08 20:26:16.161612 [11383]: server/ctdb_freeze.c:526 node[1].flags[0x2]
2012/10/08 20:26:16.161620 [11383]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/08 20:26:16.161648 [11383]: Added timed event "queue_next_trigger": 0x25b5fd0
2012/10/08 20:26:16.161659 [11383]: Ending timer event 0x25b6ff0 "queue_next_trigger"
2012/10/08 20:26:16.161668 [11383]: server/ctdb_server.c:331 ctdb request 2147483592 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.161682 [11383]: Destroying timer event 0x25b58c0 "ctdb_control_timeout"
2012/10/08 20:26:16.161692 [11383]: Ending timer event 0x25b5fd0 "queue_next_trigger"
2012/10/08 20:26:16.161718 [recoverd:11454]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:26:16.161746 [11383]: server/ctdb_daemon.c:628 client request 2147483586 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.161760 [11383]: Added timed event "queue_next_trigger": 0x25b5f40
2012/10/08 20:26:16.161770 [11383]: Added timed event "ctdb_control_timeout": 0x25b6f50
2012/10/08 20:26:16.161780 [11383]: server/ctdb_server.c:331 ctdb request 2147483593 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.161791 [11383]: Added timed event "queue_next_trigger": 0x25b67c0
2012/10/08 20:26:16.161801 [11383]: Ending timer event 0x25b5f40 "queue_next_trigger"
2012/10/08 20:26:16.161810 [11383]: server/ctdb_server.c:331 ctdb request 2147483593 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.161823 [11383]: Destroying timer event 0x25b6f50 "ctdb_control_timeout"
2012/10/08 20:26:16.161833 [11383]: Ending timer event 0x25b67c0 "queue_next_trigger"
2012/10/08 20:26:16.161891 [11383]: server/ctdb_daemon.c:628 client request 2147483587 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.161906 [11383]: Added timed event "queue_next_trigger": 0x25b6160
2012/10/08 20:26:16.161916 [11383]: Added timed event "ctdb_control_timeout": 0x25b6710
2012/10/08 20:26:16.161926 [11383]: server/ctdb_server.c:331 ctdb request 2147483594 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.161937 [11383]: Added timed event "queue_next_trigger": 0x25b6b70
2012/10/08 20:26:16.161946 [11383]: Ending timer event 0x25b6160 "queue_next_trigger"
2012/10/08 20:26:16.161956 [11383]: server/ctdb_server.c:331 ctdb request 2147483594 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.161969 [11383]: Destroying timer event 0x25b6710 "ctdb_control_timeout"
2012/10/08 20:26:16.161978 [11383]: Ending timer event 0x25b6b70 "queue_next_trigger"
2012/10/08 20:26:16.162003 [recoverd:11454]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:26:16.162030 [11383]: server/ctdb_daemon.c:628 client request 2147483588 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162044 [11383]: Added timed event "queue_next_trigger": 0x25b6160
2012/10/08 20:26:16.162054 [11383]: Added timed event "ctdb_control_timeout": 0x25b6ac0
2012/10/08 20:26:16.162064 [11383]: server/ctdb_server.c:331 ctdb request 2147483595 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162075 [11383]: Added timed event "queue_next_trigger": 0x25c1720
2012/10/08 20:26:16.162084 [11383]: Ending timer event 0x25b6160 "queue_next_trigger"
2012/10/08 20:26:16.162093 [11383]: server/ctdb_server.c:331 ctdb request 2147483595 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.162106 [11383]: Destroying timer event 0x25b6ac0 "ctdb_control_timeout"
2012/10/08 20:26:16.162115 [11383]: Ending timer event 0x25c1720 "queue_next_trigger"
2012/10/08 20:26:16.162141 [recoverd:11454]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:26:16.162169 [11383]: server/ctdb_daemon.c:628 client request 2147483589 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:16.162183 [11383]: Added timed event "queue_next_trigger": 0x25b6160
2012/10/08 20:26:16.162192 [11383]: Added timed event "ctdb_control_timeout": 0x25c1670
2012/10/08 20:26:16.162202 [11383]: server/ctdb_server.c:331 ctdb request 2147483596 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162212 [11383]: Added timed event "queue_next_trigger": 0x25c1b10
2012/10/08 20:26:16.162248 [11383]: Ending timer event 0x25b6160 "queue_next_trigger"
2012/10/08 20:26:16.162262 [11383]: server/ctdb_server.c:331 ctdb request 2147483596 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.162306 [11383]: Destroying timer event 0x25c1670 "ctdb_control_timeout"
2012/10/08 20:26:16.162317 [11383]: Ending timer event 0x25c1b10 "queue_next_trigger"
2012/10/08 20:26:16.162358 [11383]: server/ctdb_daemon.c:628 client request 2147483590 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162372 [11383]: Added timed event "queue_next_trigger": 0x25b6160
2012/10/08 20:26:16.162382 [11383]: Added timed event "ctdb_control_timeout": 0x25c1a60
2012/10/08 20:26:16.162391 [11383]: server/ctdb_server.c:331 ctdb request 2147483597 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162403 [11383]: Added timed event "queue_next_trigger": 0x25c1380
2012/10/08 20:26:16.162413 [11383]: Ending timer event 0x25b6160 "queue_next_trigger"
2012/10/08 20:26:16.162422 [11383]: server/ctdb_server.c:331 ctdb request 2147483597 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:16.162437 [11383]: Destroying timer event 0x25c1a60 "ctdb_control_timeout"
2012/10/08 20:26:16.162447 [11383]: Ending timer event 0x25c1380 "queue_next_trigger"
2012/10/08 20:26:16.162487 [11383]: server/ctdb_daemon.c:628 client request 2147483591 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.162501 [11383]: Added timed event "queue_next_trigger": 0x25c1130
2012/10/08 20:26:16.162511 [11383]: Added timed event "ctdb_control_timeout": 0x25c1250
2012/10/08 20:26:16.162521 [11383]: server/ctdb_server.c:331 ctdb request 2147483598 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:16.162530 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:16.162540 [11383]: Added timed event "queue_next_trigger": 0x25c0df0
2012/10/08 20:26:16.162549 [11383]: Ending timer event 0x25c1130 "queue_next_trigger"
2012/10/08 20:26:16.162559 [11383]: server/ctdb_server.c:331 ctdb request 2147483598 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.162571 [11383]: Destroying timer event 0x25c1250 "ctdb_control_timeout"
2012/10/08 20:26:16.162581 [11383]: Ending timer event 0x25c0df0 "queue_next_trigger"
2012/10/08 20:26:16.162607 [recoverd:11454]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:26:16.162634 [11383]: server/ctdb_daemon.c:628 client request 2147483592 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162648 [11383]: Added timed event "queue_next_trigger": 0x25b62f0
2012/10/08 20:26:16.162657 [11383]: Added timed event "ctdb_control_timeout": 0x25c0cd0
2012/10/08 20:26:16.162667 [11383]: server/ctdb_server.c:331 ctdb request 2147483599 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.162677 [11383]: Destroying timer event 0x25c1940 "ctdb_drop_all_ips_event"
2012/10/08 20:26:16.162686 [11383]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:26:16.162695 [11383]: Thawing priority 1
2012/10/08 20:26:16.162704 [11383]: Release freeze handler for prio 1
2012/10/08 20:26:16.162731 [11383]: Thawing priority 2
2012/10/08 20:26:16.162742 [11383]: Release freeze handler for prio 2
2012/10/08 20:26:16.162756 [11383]: Thawing priority 3
2012/10/08 20:26:16.162766 [11383]: Release freeze handler for prio 3
2012/10/08 20:26:16.163180 [11383]: server/ctdb_recover.c:808 Created PIPE FD:11 for setrecmode
2012/10/08 20:26:16.163254 [11383]: Added timed event "ctdb_set_recmode_timeout": 0x25b5580
2012/10/08 20:26:16.163333 [11383]: Ending timer event 0x25b62f0 "queue_next_trigger"
2012/10/08 20:26:16.164235 [11383]: SIGCHLD from 11909
2012/10/08 20:26:16.164252 [11383]: SIGCHLD from 11910
2012/10/08 20:26:16.164293 [11383]: SIGCHLD from 11911
2012/10/08 20:26:16.168108 [11383]: Destroying timer event 0x25b5580 "ctdb_set_recmode_timeout"
2012/10/08 20:26:16.168128 [11383]: Added timed event "queue_next_trigger": 0x25ba210
2012/10/08 20:26:16.168164 [11383]: server/ctdb_server.c:331 ctdb request 2147483599 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.168228 [11383]: Destroying timer event 0x25c0cd0 "ctdb_control_timeout"
2012/10/08 20:26:16.168255 [11383]: Ending timer event 0x25ba210 "queue_next_trigger"
2012/10/08 20:26:16.168428 [11383]: SIGCHLD from 11974
2012/10/08 20:26:16.168466 [recoverd:11454]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:26:16.168497 [11383]: server/ctdb_daemon.c:628 client request 2147483593 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.168512 [11383]: Added timed event "queue_next_trigger": 0x25ba3a0
2012/10/08 20:26:16.168522 [11383]: Added timed event "ctdb_control_timeout": 0x25ba1f0
2012/10/08 20:26:16.168532 [11383]: server/ctdb_server.c:331 ctdb request 2147483600 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.168544 [11383]: Added timed event "queue_next_trigger": 0x25b1c00
2012/10/08 20:26:16.168554 [11383]: Ending timer event 0x25ba3a0 "queue_next_trigger"
2012/10/08 20:26:16.168563 [11383]: server/ctdb_server.c:331 ctdb request 2147483600 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:16.168576 [11383]: Destroying timer event 0x25ba1f0 "ctdb_control_timeout"
2012/10/08 20:26:16.168586 [11383]: Ending timer event 0x25b1c00 "queue_next_trigger"
2012/10/08 20:26:16.168628 [11383]: server/ctdb_daemon.c:628 client request 2147483594 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.168643 [11383]: Added timed event "queue_next_trigger": 0x25ba3a0
2012/10/08 20:26:16.168653 [11383]: Added timed event "ctdb_control_timeout": 0x25b1aa0
2012/10/08 20:26:16.168662 [11383]: server/ctdb_server.c:331 ctdb request 2147483601 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.168674 [11383]: Added timed event "queue_next_trigger": 0x25b9b80
2012/10/08 20:26:16.168684 [11383]: Ending timer event 0x25ba3a0 "queue_next_trigger"
2012/10/08 20:26:16.168693 [11383]: server/ctdb_server.c:331 ctdb request 2147483601 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.168706 [11383]: Destroying timer event 0x25b1aa0 "ctdb_control_timeout"
2012/10/08 20:26:16.168716 [11383]: Ending timer event 0x25b9b80 "queue_next_trigger"
2012/10/08 20:26:16.168746 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:16.168757 [recoverd:11454]: CONSIDERING MOVES (UNASSIGNED)
2012/10/08 20:26:16.168766 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:16.168777 [recoverd:11454]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:26:16.168802 [11383]: server/ctdb_daemon.c:628 client request 2147483595 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:16.168816 [11383]: Added timed event "queue_next_trigger": 0x25b6160
2012/10/08 20:26:16.168826 [11383]: Added timed event "ctdb_control_timeout": 0x25b9af0
2012/10/08 20:26:16.168837 [11383]: server/ctdb_server.c:331 ctdb request 2147483602 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:16.168867 [11383]: Redundant release of IP 10.100.55.198/24 on interface __none__ (ip not held)
2012/10/08 20:26:16.168880 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/08 20:26:16.168891 [11383]: Added timed event "queue_next_trigger": 0x25b9ea0
2012/10/08 20:26:16.168901 [11383]: Ending timer event 0x25b6160 "queue_next_trigger"
2012/10/08 20:26:16.168910 [11383]: server/ctdb_server.c:331 ctdb request 2147483602 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.168923 [11383]: Destroying timer event 0x25b9af0 "ctdb_control_timeout"
2012/10/08 20:26:16.168933 [11383]: Ending timer event 0x25b9ea0 "queue_next_trigger"
2012/10/08 20:26:16.168976 [11383]: server/ctdb_daemon.c:628 client request 2147483596 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:16.168990 [11383]: Added timed event "queue_next_trigger": 0x25b9d80
2012/10/08 20:26:16.169000 [11383]: Added timed event "ctdb_control_timeout": 0x25b9ea0
2012/10/08 20:26:16.169010 [11383]: server/ctdb_server.c:331 ctdb request 2147483603 of type 7 length 74 from node 1 to 1
2012/10/08 20:26:16.169022 [11383]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:26:16.169042 [11383]: Monitoring has been disabled
2012/10/08 20:26:16.169056 [11383]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/08 20:26:16.169184 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:16.169220 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:16.169235 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:16.169253 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:16.169748 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.169857 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ba3e0
2012/10/08 20:26:16.169888 [11383]: Ending timer event 0x25b9d80 "queue_next_trigger"
2012/10/08 20:26:16.170383 [eventscript-00.ctdb-ipreallocated:11975]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/00.ctdb ipreallocated
2012/10/08 20:26:16.182159 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.182672 [11383]: SIGCHLD from 11975
2012/10/08 20:26:16.182942 [eventscript-01.reclock-ipreallocated:11980]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/01.reclock ipreallocated
2012/10/08 20:26:16.192016 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.192508 [11383]: SIGCHLD from 11980
2012/10/08 20:26:16.192779 [eventscript-10.interface-ipreallocated:11984]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/10.interface ipreallocated
2012/10/08 20:26:16.201912 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.202425 [11383]: SIGCHLD from 11984
2012/10/08 20:26:16.202696 [eventscript-11.natgw-ipreallocated:11988]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.natgw ipreallocated
2012/10/08 20:26:16.211646 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.212138 [11383]: SIGCHLD from 11988
2012/10/08 20:26:16.212410 [eventscript-11.routing-ipreallocated:11992]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.routing ipreallocated
2012/10/08 20:26:16.221298 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.221777 [11383]: SIGCHLD from 11992
2012/10/08 20:26:16.222049 [eventscript-13.per_ip_routing-ipreallocated:11996]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/13.per_ip_routing ipreallocated
2012/10/08 20:26:16.233641 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.234132 [11383]: SIGCHLD from 11996
2012/10/08 20:26:16.234638 [eventscript-20.multipathd-ipreallocated:12001]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/20.multipathd ipreallocated
2012/10/08 20:26:16.235324 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.236021 [eventscript-31.clamd-ipreallocated:12002]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/31.clamd ipreallocated
2012/10/08 20:26:16.236566 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.237284 [11383]: SIGCHLD from 12001
2012/10/08 20:26:16.237544 [eventscript-40.vsftpd-ipreallocated:12003]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.vsftpd ipreallocated
2012/10/08 20:26:16.242457 [11383]: SIGCHLD from 12002
2012/10/08 20:26:16.246889 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.247364 [11383]: SIGCHLD from 12003
2012/10/08 20:26:16.247638 [eventscript-40.fs_use-ipreallocated:12007]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.fs_use ipreallocated
2012/10/08 20:26:16.248183 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.248871 [eventscript-41.httpd-ipreallocated:12008]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/41.httpd ipreallocated
2012/10/08 20:26:16.250962 [11383]: SIGCHLD from 12007
2012/10/08 20:26:16.258037 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.258541 [11383]: SIGCHLD from 12008
2012/10/08 20:26:16.258815 [eventscript-50.samba-ipreallocated:12012]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/50.samba ipreallocated
2012/10/08 20:26:16.270921 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.271404 [11383]: SIGCHLD from 12012
2012/10/08 20:26:16.271689 [eventscript-60.nfs-ipreallocated:12017]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.nfs ipreallocated
2012/10/08 20:26:16.285867 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.286403 [11383]: SIGCHLD from 12017
2012/10/08 20:26:16.286680 [eventscript-60.ganesha-ipreallocated:12023]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.ganesha ipreallocated
2012/10/08 20:26:16.287242 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.287937 [eventscript-62.cnfs-ipreallocated:12024]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/62.cnfs ipreallocated
2012/10/08 20:26:16.289949 [11383]: SIGCHLD from 12023
2012/10/08 20:26:16.299728 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.300208 [11383]: SIGCHLD from 12024
2012/10/08 20:26:16.300482 [eventscript-70.iscsi-ipreallocated:12029]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/70.iscsi ipreallocated
2012/10/08 20:26:16.309383 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.309872 [11383]: SIGCHLD from 12029
2012/10/08 20:26:16.310141 [eventscript-91.lvs-ipreallocated:12033]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/91.lvs ipreallocated
2012/10/08 20:26:16.318519 [11383]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/08 20:26:16.318569 [11383]: Monitoring has been enabled
2012/10/08 20:26:16.318585 [11383]: Added timed event "queue_next_trigger": 0x25b9960
2012/10/08 20:26:16.318633 [11383]: Destroying timer event 0x25ba3e0 "ctdb_event_script_timeout"
2012/10/08 20:26:16.318646 [11383]: server/ctdb_server.c:331 ctdb request 2147483603 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.318719 [11383]: Destroying timer event 0x25b9ea0 "ctdb_control_timeout"
2012/10/08 20:26:16.318731 [11383]: Ending timer event 0x25b9960 "queue_next_trigger"
2012/10/08 20:26:16.318788 [11383]: server/ctdb_daemon.c:628 client request 2147483597 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.318803 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:16.318814 [11383]: Added timed event "ctdb_control_timeout": 0x25a9fe0
2012/10/08 20:26:16.318824 [11383]: server/ctdb_server.c:331 ctdb request 2147483604 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:16.318834 [11383]: Recovery has finished
2012/10/08 20:26:16.318843 [11383]: Monitoring has been disabled
2012/10/08 20:26:16.318853 [11383]: server/eventscript.c:754 Starting eventscript recovered
2012/10/08 20:26:16.318959 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:16.318996 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:16.319010 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:16.319029 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:16.319507 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.319603 [11383]: Added timed event "ctdb_event_script_timeout": 0x25aeed0
2012/10/08 20:26:16.319660 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:16.320043 [11383]: SIGCHLD from 12033
2012/10/08 20:26:16.320313 [eventscript-00.ctdb-recovered:12037]: Executing event script /etc/ctdb/events.d/00.ctdb recovered
2012/10/08 20:26:16.331949 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.332408 [11383]: SIGCHLD from 12037
2012/10/08 20:26:16.332695 [eventscript-01.reclock-recovered:12042]: Executing event script /etc/ctdb/events.d/01.reclock recovered
2012/10/08 20:26:16.341696 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.342152 [11383]: SIGCHLD from 12042
2012/10/08 20:26:16.342466 [eventscript-10.interface-recovered:12046]: Executing event script /etc/ctdb/events.d/10.interface recovered
2012/10/08 20:26:16.351708 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.352177 [11383]: SIGCHLD from 12046
2012/10/08 20:26:16.352444 [eventscript-11.natgw-recovered:12050]: Executing event script /etc/ctdb/events.d/11.natgw recovered
2012/10/08 20:26:16.361374 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.361847 [11383]: SIGCHLD from 12050
2012/10/08 20:26:16.362143 [eventscript-11.routing-recovered:12054]: Executing event script /etc/ctdb/events.d/11.routing recovered
2012/10/08 20:26:16.371085 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.371570 [11383]: SIGCHLD from 12054
2012/10/08 20:26:16.371838 [eventscript-13.per_ip_routing-recovered:12058]: Executing event script /etc/ctdb/events.d/13.per_ip_routing recovered
2012/10/08 20:26:16.380816 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.381177 [11383]: SIGCHLD from 12058
2012/10/08 20:26:16.381394 [eventscript-20.multipathd-recovered:12063]: Executing event script /etc/ctdb/events.d/20.multipathd recovered
2012/10/08 20:26:16.381814 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.382349 [eventscript-31.clamd-recovered:12064]: Executing event script /etc/ctdb/events.d/31.clamd recovered
2012/10/08 20:26:16.382794 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.383002 [11383]: SIGCHLD from 12063
2012/10/08 20:26:16.383205 [eventscript-40.vsftpd-recovered:12065]: Executing event script /etc/ctdb/events.d/40.vsftpd recovered
2012/10/08 20:26:16.387411 [11383]: SIGCHLD from 12064
2012/10/08 20:26:16.390199 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.390573 [11383]: SIGCHLD from 12065
2012/10/08 20:26:16.390778 [eventscript-40.fs_use-recovered:12069]: Executing event script /etc/ctdb/events.d/40.fs_use recovered
2012/10/08 20:26:16.391226 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.391722 [eventscript-41.httpd-recovered:12070]: Executing event script /etc/ctdb/events.d/41.httpd recovered
2012/10/08 20:26:16.395805 [11383]: SIGCHLD from 12069
2012/10/08 20:26:16.398681 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.399041 [11383]: SIGCHLD from 12070
2012/10/08 20:26:16.399242 [eventscript-50.samba-recovered:12074]: Executing event script /etc/ctdb/events.d/50.samba recovered
2012/10/08 20:26:16.408225 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.408585 [11383]: SIGCHLD from 12074
2012/10/08 20:26:16.408799 [eventscript-60.nfs-recovered:12079]: Executing event script /etc/ctdb/events.d/60.nfs recovered
2012/10/08 20:26:16.419543 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.419887 [11383]: SIGCHLD from 12079
2012/10/08 20:26:16.420101 [eventscript-60.ganesha-recovered:12085]: Executing event script /etc/ctdb/events.d/60.ganesha recovered
2012/10/08 20:26:16.420536 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.421034 [eventscript-62.cnfs-recovered:12086]: Executing event script /etc/ctdb/events.d/62.cnfs recovered
2012/10/08 20:26:16.425099 [11383]: SIGCHLD from 12085
2012/10/08 20:26:16.429723 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.430083 [11383]: SIGCHLD from 12086
2012/10/08 20:26:16.430311 [eventscript-70.iscsi-recovered:12091]: Executing event script /etc/ctdb/events.d/70.iscsi recovered
2012/10/08 20:26:16.436928 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:16.437290 [11383]: SIGCHLD from 12091
2012/10/08 20:26:16.437498 [eventscript-91.lvs-recovered:12095]: Executing event script /etc/ctdb/events.d/91.lvs recovered
2012/10/08 20:26:16.443811 [11383]: server/eventscript.c:485 Eventscript recovered finished with state 0
2012/10/08 20:26:16.443851 [11383]: Monitoring has been enabled
2012/10/08 20:26:16.443863 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:16.443874 [11383]: Destroying timer event 0x25aeed0 "ctdb_event_script_timeout"
2012/10/08 20:26:16.443883 [11383]: server/ctdb_server.c:331 ctdb request 2147483604 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:16.443922 [11383]: Destroying timer event 0x25a9fe0 "ctdb_control_timeout"
2012/10/08 20:26:16.443931 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:16.443960 [recoverd:11454]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:26:16.443984 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 48 from node 1 to 4026531844
2012/10/08 20:26:16.443995 [11383]: Added timed event "queue_next_trigger": 0x25adb70
2012/10/08 20:26:16.444003 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 5 length 48 from node 1 to 1
2012/10/08 20:26:16.444012 [11383]: Ending timer event 0x25adb70 "queue_next_trigger"
2012/10/08 20:26:16.444025 [recoverd:11454]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:26:16.444033 [recoverd:11454]: Resetting ban count to 0 for all nodes
2012/10/08 20:26:16.444040 [recoverd:11454]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:26:16.444177 [11383]: SIGCHLD from 12095
2012/10/08 20:26:16.953296 [11383]: Added timed event "ctdb_statistics_update": 0x25ada10
2012/10/08 20:26:16.953312 [11383]: Ending timer event 0x25b9a40 "ctdb_statistics_update"
2012/10/08 20:26:17.095360 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25adb70
2012/10/08 20:26:17.095376 [11383]: Ending timer event 0x25adff0 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:17.110427 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:17.110443 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:17.110455 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b4e40
2012/10/08 20:26:17.110463 [11383]: Ending timer event 0x25ad7c0 "ctdb_wait_until_recovered"
2012/10/08 20:26:17.110473 [11383]: Added timed event "ctdb_time_tick": 0x25ad7c0
2012/10/08 20:26:17.110482 [11383]: Ending timer event 0x25b6530 "ctdb_time_tick"
2012/10/08 20:26:17.117559 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:17.117611 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6360
2012/10/08 20:26:17.117622 [11383]: Ending timer event 0x25b98b0 "ctdb_tcp_node_connect"
2012/10/08 20:26:17.117921 [11383]: Destroying timer event 0x25b6360 "ctdb_tcp_node_connect"
2012/10/08 20:26:17.117945 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6360
2012/10/08 20:26:17.954079 [11383]: Added timed event "ctdb_statistics_update": 0x25aef40
2012/10/08 20:26:17.954141 [11383]: Ending timer event 0x25ada10 "ctdb_statistics_update"
2012/10/08 20:26:18.111196 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:18.111212 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:18.111245 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b4320
2012/10/08 20:26:18.111255 [11383]: Ending timer event 0x25b4e40 "ctdb_wait_until_recovered"
2012/10/08 20:26:18.111265 [11383]: Added timed event "ctdb_time_tick": 0x25b4e40
2012/10/08 20:26:18.111273 [11383]: Ending timer event 0x25ad7c0 "ctdb_time_tick"
2012/10/08 20:26:18.118347 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:18.118421 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1a30
2012/10/08 20:26:18.118433 [11383]: Ending timer event 0x25b6360 "ctdb_tcp_node_connect"
2012/10/08 20:26:18.118777 [11383]: Destroying timer event 0x25b1a30 "ctdb_tcp_node_connect"
2012/10/08 20:26:18.118808 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1a30
2012/10/08 20:26:18.954874 [11383]: Added timed event "ctdb_statistics_update": 0x25b1b50
2012/10/08 20:26:18.954892 [11383]: Ending timer event 0x25aef40 "ctdb_statistics_update"
2012/10/08 20:26:19.111969 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:19.111984 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:19.112023 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b1e00
2012/10/08 20:26:19.112033 [11383]: Ending timer event 0x25b4320 "ctdb_wait_until_recovered"
2012/10/08 20:26:19.112043 [11383]: Added timed event "ctdb_time_tick": 0x25b4320
2012/10/08 20:26:19.112052 [11383]: Ending timer event 0x25b4e40 "ctdb_time_tick"
2012/10/08 20:26:19.119109 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:19.119146 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5c00
2012/10/08 20:26:19.119158 [11383]: Ending timer event 0x25b1a30 "ctdb_tcp_node_connect"
2012/10/08 20:26:19.119483 [11383]: Destroying timer event 0x25b5c00 "ctdb_tcp_node_connect"
2012/10/08 20:26:19.119504 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5c00
2012/10/08 20:26:19.955595 [11383]: Added timed event "ctdb_statistics_update": 0x25b5d90
2012/10/08 20:26:19.955614 [11383]: Ending timer event 0x25b1b50 "ctdb_statistics_update"
2012/10/08 20:26:20.112665 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:20.112680 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:20.112692 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b57e0
2012/10/08 20:26:20.112701 [11383]: Ending timer event 0x25b1e00 "ctdb_wait_until_recovered"
2012/10/08 20:26:20.112711 [11383]: Added timed event "ctdb_time_tick": 0x25b1e00
2012/10/08 20:26:20.112719 [11383]: Ending timer event 0x25b4320 "ctdb_time_tick"
2012/10/08 20:26:20.119777 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:20.119813 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5900
2012/10/08 20:26:20.119825 [11383]: Ending timer event 0x25b5c00 "ctdb_tcp_node_connect"
2012/10/08 20:26:20.120121 [11383]: Destroying timer event 0x25b5900 "ctdb_tcp_node_connect"
2012/10/08 20:26:20.120143 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5c00
2012/10/08 20:26:20.956264 [11383]: Added timed event "ctdb_statistics_update": 0x25b4320
2012/10/08 20:26:20.956322 [11383]: Ending timer event 0x25b5d90 "ctdb_statistics_update"
2012/10/08 20:26:21.113378 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:21.113398 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:21.113412 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b6ba0
2012/10/08 20:26:21.113421 [11383]: Ending timer event 0x25b57e0 "ctdb_wait_until_recovered"
2012/10/08 20:26:21.113431 [11383]: Added timed event "ctdb_time_tick": 0x25b6cc0
2012/10/08 20:26:21.113439 [11383]: Ending timer event 0x25b1e00 "ctdb_time_tick"
2012/10/08 20:26:21.120511 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:21.120577 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1e00
2012/10/08 20:26:21.120589 [11383]: Ending timer event 0x25b5c00 "ctdb_tcp_node_connect"
2012/10/08 20:26:21.120964 [11383]: Destroying timer event 0x25b1e00 "ctdb_tcp_node_connect"
2012/10/08 20:26:21.120995 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b57e0
2012/10/08 20:26:21.937114 [11383]: Added timed event "ctdb_check_interfaces_event": 0x25b6780
2012/10/08 20:26:21.937162 [11383]: Ending timer event 0x25aa8e0 "ctdb_check_interfaces_event"
2012/10/08 20:26:21.957222 [11383]: Added timed event "ctdb_statistics_update": 0x25b6910
2012/10/08 20:26:21.957239 [11383]: Ending timer event 0x25b4320 "ctdb_statistics_update"
2012/10/08 20:26:22.092291 [11383]: Added timed event "ctdb_check_recd": 0x25b4320
2012/10/08 20:26:22.092308 [11383]: Ending timer event 0x25adac0 "ctdb_check_recd"
2012/10/08 20:26:22.096363 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25ada80
2012/10/08 20:26:22.096380 [11383]: Ending timer event 0x25adb70 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:22.114434 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:22.114453 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:22.114466 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b6f50
2012/10/08 20:26:22.114475 [11383]: Ending timer event 0x25b6ba0 "ctdb_wait_until_recovered"
2012/10/08 20:26:22.114485 [11383]: Added timed event "ctdb_time_tick": 0x25b6ba0
2012/10/08 20:26:22.114494 [11383]: Ending timer event 0x25b6cc0 "ctdb_time_tick"
2012/10/08 20:26:22.121561 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:22.121613 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5f40
2012/10/08 20:26:22.121625 [11383]: Ending timer event 0x25b57e0 "ctdb_tcp_node_connect"
2012/10/08 20:26:22.121918 [11383]: Destroying timer event 0x25b5f40 "ctdb_tcp_node_connect"
2012/10/08 20:26:22.121943 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5f40
2012/10/08 20:26:22.958013 [11383]: Added timed event "ctdb_statistics_update": 0x25b60d0
2012/10/08 20:26:22.958052 [11383]: Ending timer event 0x25b6910 "ctdb_statistics_update"
2012/10/08 20:26:23.115106 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:23.115126 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:23.115139 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b4ff0
2012/10/08 20:26:23.115148 [11383]: Ending timer event 0x25b6f50 "ctdb_wait_until_recovered"
2012/10/08 20:26:23.115158 [11383]: Added timed event "ctdb_time_tick": 0x25b6f50
2012/10/08 20:26:23.115166 [11383]: Ending timer event 0x25b6ba0 "ctdb_time_tick"
2012/10/08 20:26:23.122295 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:23.122339 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5110
2012/10/08 20:26:23.122351 [11383]: Ending timer event 0x25b5f40 "ctdb_tcp_node_connect"
2012/10/08 20:26:23.122645 [11383]: Destroying timer event 0x25b5110 "ctdb_tcp_node_connect"
2012/10/08 20:26:23.122673 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5110
2012/10/08 20:26:23.456201 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:23.456405 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:23.456427 [11383]: Added timed event "queue_next_trigger": 0x25b25b0
2012/10/08 20:26:23.456439 [11383]: Added timed event "ctdb_control_timeout": 0x25b26d0
2012/10/08 20:26:23.456450 [11383]: server/ctdb_server.c:331 ctdb request 2147483606 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.456463 [11383]: Added timed event "queue_next_trigger": 0x25b2910
2012/10/08 20:26:23.456473 [11383]: Ending timer event 0x25b25b0 "queue_next_trigger"
2012/10/08 20:26:23.456483 [11383]: server/ctdb_server.c:331 ctdb request 2147483606 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:23.456498 [11383]: Destroying timer event 0x25b26d0 "ctdb_control_timeout"
2012/10/08 20:26:23.456508 [11383]: Ending timer event 0x25b2910 "queue_next_trigger"
2012/10/08 20:26:23.456584 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:23.456686 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:23.456727 [11383]: Added timed event "queue_next_trigger": 0x25c3bc0
2012/10/08 20:26:23.456738 [11383]: Added timed event "ctdb_control_timeout": 0x25c3ce0
2012/10/08 20:26:23.456748 [11383]: server/ctdb_server.c:331 ctdb request 2147483608 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.456759 [11383]: Added timed event "queue_next_trigger": 0x25c3f00
2012/10/08 20:26:23.456782 [11383]: Ending timer event 0x25c3bc0 "queue_next_trigger"
2012/10/08 20:26:23.456793 [11383]: server/ctdb_server.c:331 ctdb request 2147483608 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:23.456807 [11383]: Destroying timer event 0x25c3ce0 "ctdb_control_timeout"
2012/10/08 20:26:23.456817 [11383]: Ending timer event 0x25c3f00 "queue_next_trigger"
2012/10/08 20:26:23.456866 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.456881 [11383]: Added timed event "queue_next_trigger": 0x25c3a90
2012/10/08 20:26:23.456891 [11383]: Added timed event "ctdb_control_timeout": 0x25c3bb0
2012/10/08 20:26:23.456901 [11383]: server/ctdb_server.c:331 ctdb request 2147483609 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.456912 [11383]: Added timed event "queue_next_trigger": 0x25c4220
2012/10/08 20:26:23.456922 [11383]: Ending timer event 0x25c3a90 "queue_next_trigger"
2012/10/08 20:26:23.456931 [11383]: server/ctdb_server.c:331 ctdb request 2147483609 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:23.456944 [11383]: Destroying timer event 0x25c3bb0 "ctdb_control_timeout"
2012/10/08 20:26:23.456954 [11383]: Ending timer event 0x25c4220 "queue_next_trigger"
2012/10/08 20:26:23.457005 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.457020 [11383]: Added timed event "queue_next_trigger": 0x25c4100
2012/10/08 20:26:23.457030 [11383]: Added timed event "ctdb_control_timeout": 0x25c4220
2012/10/08 20:26:23.457040 [11383]: server/ctdb_server.c:331 ctdb request 2147483610 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.457054 [11383]: Added timed event "queue_next_trigger": 0x25c4670
2012/10/08 20:26:23.457063 [11383]: Ending timer event 0x25c4100 "queue_next_trigger"
2012/10/08 20:26:23.457073 [11383]: server/ctdb_server.c:331 ctdb request 2147483610 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:23.457086 [11383]: Destroying timer event 0x25c4220 "ctdb_control_timeout"
2012/10/08 20:26:23.457095 [11383]: Ending timer event 0x25c4670 "queue_next_trigger"
2012/10/08 20:26:23.457599 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.457616 [11383]: Added timed event "queue_next_trigger": 0x25c4420
2012/10/08 20:26:23.457627 [11383]: Added timed event "ctdb_control_timeout": 0x25c4540
2012/10/08 20:26:23.457637 [11383]: server/ctdb_server.c:331 ctdb request 2147483611 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.457649 [11383]: Added timed event "queue_next_trigger": 0x25c4900
2012/10/08 20:26:23.457667 [11383]: Ending timer event 0x25c4420 "queue_next_trigger"
2012/10/08 20:26:23.457680 [11383]: server/ctdb_server.c:331 ctdb request 2147483611 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:23.457694 [11383]: Destroying timer event 0x25c4540 "ctdb_control_timeout"
2012/10/08 20:26:23.457704 [11383]: Ending timer event 0x25c4900 "queue_next_trigger"
2012/10/08 20:26:23.458044 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:23.458061 [11383]: Added timed event "queue_next_trigger": 0x25c3bb0
2012/10/08 20:26:23.458071 [11383]: Added timed event "ctdb_control_timeout": 0x25c4870
2012/10/08 20:26:23.458081 [11383]: server/ctdb_server.c:331 ctdb request 2147483612 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.458099 [11383]: Added timed event "queue_next_trigger": 0x25c4b20
2012/10/08 20:26:23.458110 [11383]: Ending timer event 0x25c3bb0 "queue_next_trigger"
2012/10/08 20:26:23.458119 [11383]: server/ctdb_server.c:331 ctdb request 2147483612 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:23.458133 [11383]: Destroying timer event 0x25c4870 "ctdb_control_timeout"
2012/10/08 20:26:23.458142 [11383]: Ending timer event 0x25c4b20 "queue_next_trigger"
2012/10/08 20:26:23.458174 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:23.458188 [11383]: Added timed event "queue_next_trigger": 0x25c3bb0
2012/10/08 20:26:23.458198 [11383]: Added timed event "ctdb_control_timeout": 0x25c4b00
2012/10/08 20:26:23.458208 [11383]: server/ctdb_server.c:331 ctdb request 2147483613 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.458218 [11383]: Added timed event "queue_next_trigger": 0x25c4e40
2012/10/08 20:26:23.458294 [11383]: Ending timer event 0x25c3bb0 "queue_next_trigger"
2012/10/08 20:26:23.458305 [11383]: server/ctdb_server.c:331 ctdb request 2147483613 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:23.458319 [11383]: Destroying timer event 0x25c4b00 "ctdb_control_timeout"
2012/10/08 20:26:23.458329 [11383]: Ending timer event 0x25c4e40 "queue_next_trigger"
2012/10/08 20:26:23.458469 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:23.458485 [11383]: Added timed event "queue_next_trigger": 0x25c4d20
2012/10/08 20:26:23.458495 [11383]: Added timed event "ctdb_control_timeout": 0x25c4e40
2012/10/08 20:26:23.458505 [11383]: server/ctdb_server.c:331 ctdb request 2147483614 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:23.458516 [11383]: Added timed event "queue_next_trigger": 0x25c5160
2012/10/08 20:26:23.458526 [11383]: Ending timer event 0x25c4d20 "queue_next_trigger"
2012/10/08 20:26:23.458584 [11383]: server/ctdb_server.c:331 ctdb request 2147483614 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:23.458601 [11383]: Destroying timer event 0x25c4e40 "ctdb_control_timeout"
2012/10/08 20:26:23.458611 [11383]: Ending timer event 0x25c5160 "queue_next_trigger"
2012/10/08 20:26:23.958982 [11383]: Added timed event "ctdb_statistics_update": 0x25c4d20
2012/10/08 20:26:23.959015 [11383]: Ending timer event 0x25b60d0 "ctdb_statistics_update"
2012/10/08 20:26:24.116067 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:24.116086 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:24.116098 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b26d0
2012/10/08 20:26:24.116107 [11383]: Ending timer event 0x25b4ff0 "ctdb_wait_until_recovered"
2012/10/08 20:26:24.116117 [11383]: Added timed event "ctdb_time_tick": 0x25c4e40
2012/10/08 20:26:24.116125 [11383]: Ending timer event 0x25b6f50 "ctdb_time_tick"
2012/10/08 20:26:24.123189 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:24.123235 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6f50
2012/10/08 20:26:24.123247 [11383]: Ending timer event 0x25b5110 "ctdb_tcp_node_connect"
2012/10/08 20:26:24.123518 [11383]: Destroying timer event 0x25b6f50 "ctdb_tcp_node_connect"
2012/10/08 20:26:24.123544 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b6f50
2012/10/08 20:26:24.959689 [11383]: Added timed event "ctdb_statistics_update": 0x25b2550
2012/10/08 20:26:24.959739 [11383]: Ending timer event 0x25c4d20 "ctdb_statistics_update"
2012/10/08 20:26:25.116795 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:25.116816 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:25.116829 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b27f0
2012/10/08 20:26:25.116838 [11383]: Ending timer event 0x25b26d0 "ctdb_wait_until_recovered"
2012/10/08 20:26:25.116848 [11383]: Added timed event "ctdb_time_tick": 0x25b2910
2012/10/08 20:26:25.116856 [11383]: Ending timer event 0x25c4e40 "ctdb_time_tick"
2012/10/08 20:26:25.123949 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:25.124002 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25c4e40
2012/10/08 20:26:25.124015 [11383]: Ending timer event 0x25b6f50 "ctdb_tcp_node_connect"
2012/10/08 20:26:25.124301 [11383]: Destroying timer event 0x25c4e40 "ctdb_tcp_node_connect"
2012/10/08 20:26:25.124332 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25c4e40
2012/10/08 20:26:25.248043 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:25.248183 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:25.248204 [11383]: Added timed event "queue_next_trigger": 0x25b7460
2012/10/08 20:26:25.248215 [11383]: Added timed event "ctdb_control_timeout": 0x25b7580
2012/10/08 20:26:25.248226 [11383]: server/ctdb_server.c:331 ctdb request 2147483616 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.248239 [11383]: Added timed event "queue_next_trigger": 0x25b7730
2012/10/08 20:26:25.248249 [11383]: Ending timer event 0x25b7460 "queue_next_trigger"
2012/10/08 20:26:25.248259 [11383]: server/ctdb_server.c:331 ctdb request 2147483616 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.248274 [11383]: Destroying timer event 0x25b7580 "ctdb_control_timeout"
2012/10/08 20:26:25.248284 [11383]: Ending timer event 0x25b7730 "queue_next_trigger"
2012/10/08 20:26:25.248346 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:25.248404 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:25.248419 [11383]: Added timed event "queue_next_trigger": 0x25b7ad0
2012/10/08 20:26:25.248429 [11383]: Added timed event "ctdb_control_timeout": 0x25b7c60
2012/10/08 20:26:25.248439 [11383]: server/ctdb_server.c:331 ctdb request 2147483618 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.248450 [11383]: Added timed event "queue_next_trigger": 0x25b7ea0
2012/10/08 20:26:25.248459 [11383]: Ending timer event 0x25b7ad0 "queue_next_trigger"
2012/10/08 20:26:25.248468 [11383]: server/ctdb_server.c:331 ctdb request 2147483618 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.248481 [11383]: Destroying timer event 0x25b7c60 "ctdb_control_timeout"
2012/10/08 20:26:25.248491 [11383]: Ending timer event 0x25b7ea0 "queue_next_trigger"
2012/10/08 20:26:25.248539 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.248554 [11383]: Added timed event "queue_next_trigger": 0x25b7a00
2012/10/08 20:26:25.248579 [11383]: Added timed event "ctdb_control_timeout": 0x25b8030
2012/10/08 20:26:25.248592 [11383]: server/ctdb_server.c:331 ctdb request 2147483619 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.248603 [11383]: Added timed event "queue_next_trigger": 0x25b8270
2012/10/08 20:26:25.248613 [11383]: Ending timer event 0x25b7a00 "queue_next_trigger"
2012/10/08 20:26:25.248622 [11383]: server/ctdb_server.c:331 ctdb request 2147483619 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.248637 [11383]: Destroying timer event 0x25b8030 "ctdb_control_timeout"
2012/10/08 20:26:25.248646 [11383]: Ending timer event 0x25b8270 "queue_next_trigger"
2012/10/08 20:26:25.248699 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.248713 [11383]: Added timed event "queue_next_trigger": 0x25b8150
2012/10/08 20:26:25.248723 [11383]: Added timed event "ctdb_control_timeout": 0x25b8270
2012/10/08 20:26:25.248733 [11383]: server/ctdb_server.c:331 ctdb request 2147483620 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.248746 [11383]: Added timed event "queue_next_trigger": 0x25b8650
2012/10/08 20:26:25.248756 [11383]: Ending timer event 0x25b8150 "queue_next_trigger"
2012/10/08 20:26:25.248765 [11383]: server/ctdb_server.c:331 ctdb request 2147483620 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:25.248778 [11383]: Destroying timer event 0x25b8270 "ctdb_control_timeout"
2012/10/08 20:26:25.248796 [11383]: Ending timer event 0x25b8650 "queue_next_trigger"
2012/10/08 20:26:25.249214 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.249231 [11383]: Added timed event "queue_next_trigger": 0x25b8400
2012/10/08 20:26:25.249289 [11383]: Added timed event "ctdb_control_timeout": 0x25b8520
2012/10/08 20:26:25.249304 [11383]: server/ctdb_server.c:331 ctdb request 2147483621 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.249316 [11383]: Added timed event "queue_next_trigger": 0x25b88e0
2012/10/08 20:26:25.249326 [11383]: Ending timer event 0x25b8400 "queue_next_trigger"
2012/10/08 20:26:25.249336 [11383]: server/ctdb_server.c:331 ctdb request 2147483621 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:25.249350 [11383]: Destroying timer event 0x25b8520 "ctdb_control_timeout"
2012/10/08 20:26:25.249367 [11383]: Ending timer event 0x25b88e0 "queue_next_trigger"
2012/10/08 20:26:25.249775 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:25.249791 [11383]: Added timed event "queue_next_trigger": 0x25b7df0
2012/10/08 20:26:25.249802 [11383]: Added timed event "ctdb_control_timeout": 0x25b8850
2012/10/08 20:26:25.249812 [11383]: server/ctdb_server.c:331 ctdb request 2147483622 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.249823 [11383]: Added timed event "queue_next_trigger": 0x25b8b90
2012/10/08 20:26:25.249833 [11383]: Ending timer event 0x25b7df0 "queue_next_trigger"
2012/10/08 20:26:25.249843 [11383]: server/ctdb_server.c:331 ctdb request 2147483622 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.249856 [11383]: Destroying timer event 0x25b8850 "ctdb_control_timeout"
2012/10/08 20:26:25.249865 [11383]: Ending timer event 0x25b8b90 "queue_next_trigger"
2012/10/08 20:26:25.249897 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:25.249911 [11383]: Added timed event "queue_next_trigger": 0x25b8a70
2012/10/08 20:26:25.249921 [11383]: Added timed event "ctdb_control_timeout": 0x25b8b90
2012/10/08 20:26:25.250000 [11383]: server/ctdb_server.c:331 ctdb request 2147483623 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.250015 [11383]: Added timed event "queue_next_trigger": 0x25b8eb0
2012/10/08 20:26:25.250025 [11383]: Ending timer event 0x25b8a70 "queue_next_trigger"
2012/10/08 20:26:25.250034 [11383]: server/ctdb_server.c:331 ctdb request 2147483623 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.250048 [11383]: Destroying timer event 0x25b8b90 "ctdb_control_timeout"
2012/10/08 20:26:25.250067 [11383]: Ending timer event 0x25b8eb0 "queue_next_trigger"
2012/10/08 20:26:25.250202 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:25.250218 [11383]: Added timed event "queue_next_trigger": 0x25b8d90
2012/10/08 20:26:25.250282 [11383]: Added timed event "ctdb_control_timeout": 0x25b8eb0
2012/10/08 20:26:25.250294 [11383]: server/ctdb_server.c:331 ctdb request 2147483624 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.250306 [11383]: Added timed event "queue_next_trigger": 0x25b91d0
2012/10/08 20:26:25.250316 [11383]: Ending timer event 0x25b8d90 "queue_next_trigger"
2012/10/08 20:26:25.250334 [11383]: server/ctdb_server.c:331 ctdb request 2147483624 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.250350 [11383]: Destroying timer event 0x25b8eb0 "ctdb_control_timeout"
2012/10/08 20:26:25.250360 [11383]: Ending timer event 0x25b91d0 "queue_next_trigger"
2012/10/08 20:26:25.720089 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:25.720221 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:25.720240 [11383]: Added timed event "queue_next_trigger": 0x25b7930
2012/10/08 20:26:25.720250 [11383]: Added timed event "ctdb_control_timeout": 0x25b90b0
2012/10/08 20:26:25.720280 [11383]: server/ctdb_server.c:331 ctdb request 2147483626 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.720293 [11383]: Added timed event "queue_next_trigger": 0x25b55a0
2012/10/08 20:26:25.720303 [11383]: Ending timer event 0x25b7930 "queue_next_trigger"
2012/10/08 20:26:25.720313 [11383]: server/ctdb_server.c:331 ctdb request 2147483626 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.720328 [11383]: Destroying timer event 0x25b90b0 "ctdb_control_timeout"
2012/10/08 20:26:25.720338 [11383]: Ending timer event 0x25b55a0 "queue_next_trigger"
2012/10/08 20:26:25.720394 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:25.720451 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:25.720466 [11383]: Added timed event "queue_next_trigger": 0x25b94a0
2012/10/08 20:26:25.720475 [11383]: Added timed event "ctdb_control_timeout": 0x25b95c0
2012/10/08 20:26:25.720485 [11383]: server/ctdb_server.c:331 ctdb request 2147483628 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.720497 [11383]: Added timed event "queue_next_trigger": 0x25bc4f0
2012/10/08 20:26:25.720506 [11383]: Ending timer event 0x25b94a0 "queue_next_trigger"
2012/10/08 20:26:25.720516 [11383]: server/ctdb_server.c:331 ctdb request 2147483628 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.720529 [11383]: Destroying timer event 0x25b95c0 "ctdb_control_timeout"
2012/10/08 20:26:25.720539 [11383]: Ending timer event 0x25bc4f0 "queue_next_trigger"
2012/10/08 20:26:25.720587 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.720602 [11383]: Added timed event "queue_next_trigger": 0x25b93d0
2012/10/08 20:26:25.720612 [11383]: Added timed event "ctdb_control_timeout": 0x25bc680
2012/10/08 20:26:25.720622 [11383]: server/ctdb_server.c:331 ctdb request 2147483629 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.720632 [11383]: Added timed event "queue_next_trigger": 0x25bc830
2012/10/08 20:26:25.720642 [11383]: Ending timer event 0x25b93d0 "queue_next_trigger"
2012/10/08 20:26:25.720651 [11383]: server/ctdb_server.c:331 ctdb request 2147483629 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.720664 [11383]: Destroying timer event 0x25bc680 "ctdb_control_timeout"
2012/10/08 20:26:25.720674 [11383]: Ending timer event 0x25bc830 "queue_next_trigger"
2012/10/08 20:26:25.720726 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.720740 [11383]: Added timed event "queue_next_trigger": 0x25b5580
2012/10/08 20:26:25.720750 [11383]: Added timed event "ctdb_control_timeout": 0x25bc810
2012/10/08 20:26:25.720760 [11383]: server/ctdb_server.c:331 ctdb request 2147483630 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.720773 [11383]: Added timed event "queue_next_trigger": 0x25bcc80
2012/10/08 20:26:25.720783 [11383]: Ending timer event 0x25b5580 "queue_next_trigger"
2012/10/08 20:26:25.720792 [11383]: server/ctdb_server.c:331 ctdb request 2147483630 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:25.720805 [11383]: Destroying timer event 0x25bc810 "ctdb_control_timeout"
2012/10/08 20:26:25.720815 [11383]: Ending timer event 0x25bcc80 "queue_next_trigger"
2012/10/08 20:26:25.721220 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.721237 [11383]: Added timed event "queue_next_trigger": 0x25bca30
2012/10/08 20:26:25.721247 [11383]: Added timed event "ctdb_control_timeout": 0x25bcb50
2012/10/08 20:26:25.721257 [11383]: server/ctdb_server.c:331 ctdb request 2147483631 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.721269 [11383]: Added timed event "queue_next_trigger": 0x25bcf10
2012/10/08 20:26:25.721295 [11383]: Ending timer event 0x25bca30 "queue_next_trigger"
2012/10/08 20:26:25.721306 [11383]: server/ctdb_server.c:331 ctdb request 2147483631 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:25.721327 [11383]: Destroying timer event 0x25bcb50 "ctdb_control_timeout"
2012/10/08 20:26:25.721337 [11383]: Ending timer event 0x25bcf10 "queue_next_trigger"
2012/10/08 20:26:25.721666 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:25.721683 [11383]: Added timed event "queue_next_trigger": 0x25b5580
2012/10/08 20:26:25.721693 [11383]: Added timed event "ctdb_control_timeout": 0x25bce80
2012/10/08 20:26:25.721703 [11383]: server/ctdb_server.c:331 ctdb request 2147483632 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.721725 [11383]: Added timed event "queue_next_trigger": 0x25bd1c0
2012/10/08 20:26:25.721745 [11383]: Ending timer event 0x25b5580 "queue_next_trigger"
2012/10/08 20:26:25.721755 [11383]: server/ctdb_server.c:331 ctdb request 2147483632 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.721769 [11383]: Destroying timer event 0x25bce80 "ctdb_control_timeout"
2012/10/08 20:26:25.721778 [11383]: Ending timer event 0x25bd1c0 "queue_next_trigger"
2012/10/08 20:26:25.721810 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:25.721824 [11383]: Added timed event "queue_next_trigger": 0x25bd0a0
2012/10/08 20:26:25.721834 [11383]: Added timed event "ctdb_control_timeout": 0x25bd1c0
2012/10/08 20:26:25.721844 [11383]: server/ctdb_server.c:331 ctdb request 2147483633 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.721855 [11383]: Added timed event "queue_next_trigger": 0x25bd4e0
2012/10/08 20:26:25.721864 [11383]: Ending timer event 0x25bd0a0 "queue_next_trigger"
2012/10/08 20:26:25.721874 [11383]: server/ctdb_server.c:331 ctdb request 2147483633 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.721886 [11383]: Destroying timer event 0x25bd1c0 "ctdb_control_timeout"
2012/10/08 20:26:25.721896 [11383]: Ending timer event 0x25bd4e0 "queue_next_trigger"
2012/10/08 20:26:25.722075 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:25.722091 [11383]: Added timed event "queue_next_trigger": 0x25bd3c0
2012/10/08 20:26:25.722101 [11383]: Added timed event "ctdb_control_timeout": 0x25bd4e0
2012/10/08 20:26:25.722111 [11383]: server/ctdb_server.c:331 ctdb request 2147483634 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.722122 [11383]: Added timed event "queue_next_trigger": 0x25bd800
2012/10/08 20:26:25.722132 [11383]: Ending timer event 0x25bd3c0 "queue_next_trigger"
2012/10/08 20:26:25.722141 [11383]: server/ctdb_server.c:331 ctdb request 2147483634 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.722154 [11383]: Destroying timer event 0x25bd4e0 "ctdb_control_timeout"
2012/10/08 20:26:25.722164 [11383]: Ending timer event 0x25bd800 "queue_next_trigger"
2012/10/08 20:26:25.903476 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:25.903588 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:25.903606 [11383]: Added timed event "queue_next_trigger": 0x25b8a70
2012/10/08 20:26:25.903616 [11383]: Added timed event "ctdb_control_timeout": 0x25b7df0
2012/10/08 20:26:25.903627 [11383]: server/ctdb_server.c:331 ctdb request 2147483636 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.903639 [11383]: Added timed event "queue_next_trigger": 0x25bd990
2012/10/08 20:26:25.903648 [11383]: Ending timer event 0x25b8a70 "queue_next_trigger"
2012/10/08 20:26:25.903658 [11383]: server/ctdb_server.c:331 ctdb request 2147483636 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.903671 [11383]: Destroying timer event 0x25b7df0 "ctdb_control_timeout"
2012/10/08 20:26:25.903681 [11383]: Ending timer event 0x25bd990 "queue_next_trigger"
2012/10/08 20:26:25.903737 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:25.903797 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:25.903822 [11383]: Added timed event "queue_next_trigger": 0x25bdc60
2012/10/08 20:26:25.903832 [11383]: Added timed event "ctdb_control_timeout": 0x25bdd80
2012/10/08 20:26:25.903842 [11383]: server/ctdb_server.c:331 ctdb request 2147483638 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.903868 [11383]: Added timed event "queue_next_trigger": 0x25bdfc0
2012/10/08 20:26:25.903878 [11383]: Ending timer event 0x25bdc60 "queue_next_trigger"
2012/10/08 20:26:25.903888 [11383]: server/ctdb_server.c:331 ctdb request 2147483638 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.903902 [11383]: Destroying timer event 0x25bdd80 "ctdb_control_timeout"
2012/10/08 20:26:25.903911 [11383]: Ending timer event 0x25bdfc0 "queue_next_trigger"
2012/10/08 20:26:25.903960 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.903974 [11383]: Added timed event "queue_next_trigger": 0x25bdea0
2012/10/08 20:26:25.903984 [11383]: Added timed event "ctdb_control_timeout": 0x25bdfc0
2012/10/08 20:26:25.903995 [11383]: server/ctdb_server.c:331 ctdb request 2147483639 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.904006 [11383]: Added timed event "queue_next_trigger": 0x25be2e0
2012/10/08 20:26:25.904015 [11383]: Ending timer event 0x25bdea0 "queue_next_trigger"
2012/10/08 20:26:25.904024 [11383]: server/ctdb_server.c:331 ctdb request 2147483639 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.904037 [11383]: Destroying timer event 0x25bdfc0 "ctdb_control_timeout"
2012/10/08 20:26:25.904047 [11383]: Ending timer event 0x25be2e0 "queue_next_trigger"
2012/10/08 20:26:25.904098 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.904112 [11383]: Added timed event "queue_next_trigger": 0x25be1c0
2012/10/08 20:26:25.904122 [11383]: Added timed event "ctdb_control_timeout": 0x25be270
2012/10/08 20:26:25.904131 [11383]: server/ctdb_server.c:331 ctdb request 2147483640 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.904144 [11383]: Added timed event "queue_next_trigger": 0x25be6c0
2012/10/08 20:26:25.904154 [11383]: Ending timer event 0x25be1c0 "queue_next_trigger"
2012/10/08 20:26:25.904163 [11383]: server/ctdb_server.c:331 ctdb request 2147483640 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:25.904176 [11383]: Destroying timer event 0x25be270 "ctdb_control_timeout"
2012/10/08 20:26:25.904186 [11383]: Ending timer event 0x25be6c0 "queue_next_trigger"
2012/10/08 20:26:25.904658 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.904675 [11383]: Added timed event "queue_next_trigger": 0x25be260
2012/10/08 20:26:25.904686 [11383]: Added timed event "ctdb_control_timeout": 0x25be550
2012/10/08 20:26:25.904696 [11383]: server/ctdb_server.c:331 ctdb request 2147483641 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.904708 [11383]: Added timed event "queue_next_trigger": 0x25be8e0
2012/10/08 20:26:25.904718 [11383]: Ending timer event 0x25be260 "queue_next_trigger"
2012/10/08 20:26:25.904727 [11383]: server/ctdb_server.c:331 ctdb request 2147483641 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:25.904740 [11383]: Destroying timer event 0x25be550 "ctdb_control_timeout"
2012/10/08 20:26:25.904750 [11383]: Ending timer event 0x25be8e0 "queue_next_trigger"
2012/10/08 20:26:25.905103 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:25.905119 [11383]: Added timed event "queue_next_trigger": 0x25be850
2012/10/08 20:26:25.905129 [11383]: Added timed event "ctdb_control_timeout": 0x25be1c0
2012/10/08 20:26:25.905140 [11383]: server/ctdb_server.c:331 ctdb request 2147483642 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.905151 [11383]: Added timed event "queue_next_trigger": 0x25beb90
2012/10/08 20:26:25.905161 [11383]: Ending timer event 0x25be850 "queue_next_trigger"
2012/10/08 20:26:25.905170 [11383]: server/ctdb_server.c:331 ctdb request 2147483642 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.905191 [11383]: Destroying timer event 0x25be1c0 "ctdb_control_timeout"
2012/10/08 20:26:25.905201 [11383]: Ending timer event 0x25beb90 "queue_next_trigger"
2012/10/08 20:26:25.905232 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:25.905246 [11383]: Added timed event "queue_next_trigger": 0x25bea70
2012/10/08 20:26:25.905256 [11383]: Added timed event "ctdb_control_timeout": 0x25beb90
2012/10/08 20:26:25.905266 [11383]: server/ctdb_server.c:331 ctdb request 2147483643 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.905277 [11383]: Added timed event "queue_next_trigger": 0x25beeb0
2012/10/08 20:26:25.905286 [11383]: Ending timer event 0x25bea70 "queue_next_trigger"
2012/10/08 20:26:25.905296 [11383]: server/ctdb_server.c:331 ctdb request 2147483643 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.905308 [11383]: Destroying timer event 0x25beb90 "ctdb_control_timeout"
2012/10/08 20:26:25.905318 [11383]: Ending timer event 0x25beeb0 "queue_next_trigger"
2012/10/08 20:26:25.905480 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:25.905496 [11383]: Added timed event "queue_next_trigger": 0x25bed90
2012/10/08 20:26:25.905506 [11383]: Added timed event "ctdb_control_timeout": 0x25beeb0
2012/10/08 20:26:25.905516 [11383]: server/ctdb_server.c:331 ctdb request 2147483644 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:25.905527 [11383]: Added timed event "queue_next_trigger": 0x25bf1d0
2012/10/08 20:26:25.905537 [11383]: Ending timer event 0x25bed90 "queue_next_trigger"
2012/10/08 20:26:25.905546 [11383]: server/ctdb_server.c:331 ctdb request 2147483644 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:25.905559 [11383]: Destroying timer event 0x25beeb0 "ctdb_control_timeout"
2012/10/08 20:26:25.905569 [11383]: Ending timer event 0x25bf1d0 "queue_next_trigger"
2012/10/08 20:26:25.959910 [11383]: Added timed event "ctdb_statistics_update": 0x25beb90
2012/10/08 20:26:25.959929 [11383]: Ending timer event 0x25b2550 "ctdb_statistics_update"
2012/10/08 20:26:26.116989 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:26.117029 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:26.117042 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25bea70
2012/10/08 20:26:26.117051 [11383]: Ending timer event 0x25b27f0 "ctdb_wait_until_recovered"
2012/10/08 20:26:26.117063 [11383]: Added timed event "ctdb_time_tick": 0x25bd3c0
2012/10/08 20:26:26.117071 [11383]: Ending timer event 0x25b2910 "ctdb_time_tick"
2012/10/08 20:26:26.125139 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:26.125189 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b8a70
2012/10/08 20:26:26.125201 [11383]: Ending timer event 0x25c4e40 "ctdb_tcp_node_connect"
2012/10/08 20:26:26.125474 [11383]: Destroying timer event 0x25b8a70 "ctdb_tcp_node_connect"
2012/10/08 20:26:26.125502 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25c4e40
2012/10/08 20:26:26.454099 [recoverd:11454]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:26:26.454155 [11383]: server/ctdb_daemon.c:628 client request 2147483598 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.454174 [11383]: Added timed event "queue_next_trigger": 0x25b7250
2012/10/08 20:26:26.454184 [11383]: Added timed event "ctdb_control_timeout": 0x25bf0b0
2012/10/08 20:26:26.454195 [11383]: server/ctdb_server.c:331 ctdb request 2147483645 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454206 [11383]: Destroying timer event 0x25ae1e0 "ctdb_recd_ping_timeout"
2012/10/08 20:26:26.454216 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:26:26.454226 [11383]: Added timed event "queue_next_trigger": 0x25bf4f0
2012/10/08 20:26:26.454235 [11383]: Ending timer event 0x25b7250 "queue_next_trigger"
2012/10/08 20:26:26.454245 [11383]: server/ctdb_server.c:331 ctdb request 2147483645 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.454274 [11383]: Destroying timer event 0x25bf0b0 "ctdb_control_timeout"
2012/10/08 20:26:26.454307 [11383]: Ending timer event 0x25bf4f0 "queue_next_trigger"
2012/10/08 20:26:26.454353 [11383]: server/ctdb_daemon.c:628 client request 2147483599 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.454368 [11383]: Added timed event "queue_next_trigger": 0x25bf3d0
2012/10/08 20:26:26.454378 [11383]: Added timed event "ctdb_control_timeout": 0x25bf4f0
2012/10/08 20:26:26.454388 [11383]: server/ctdb_server.c:331 ctdb request 2147483646 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454401 [11383]: Added timed event "queue_next_trigger": 0x25bf810
2012/10/08 20:26:26.454411 [11383]: Ending timer event 0x25bf3d0 "queue_next_trigger"
2012/10/08 20:26:26.454420 [11383]: server/ctdb_server.c:331 ctdb request 2147483646 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.454433 [11383]: Destroying timer event 0x25bf4f0 "ctdb_control_timeout"
2012/10/08 20:26:26.454443 [11383]: Ending timer event 0x25bf810 "queue_next_trigger"
2012/10/08 20:26:26.454483 [11383]: server/ctdb_daemon.c:628 client request 2147483600 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.454498 [11383]: Added timed event "queue_next_trigger": 0x25bf6f0
2012/10/08 20:26:26.454507 [11383]: Added timed event "ctdb_control_timeout": 0x25bf810
2012/10/08 20:26:26.454517 [11383]: server/ctdb_server.c:331 ctdb request 2147483647 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454530 [11383]: Added timed event "queue_next_trigger": 0x25bfc90
2012/10/08 20:26:26.454539 [11383]: Ending timer event 0x25bf6f0 "queue_next_trigger"
2012/10/08 20:26:26.454548 [11383]: server/ctdb_server.c:331 ctdb request 2147483647 of type 8 length 228 from node 1 to 1
2012/10/08 20:26:26.454561 [11383]: Destroying timer event 0x25bf810 "ctdb_control_timeout"
2012/10/08 20:26:26.454571 [11383]: Ending timer event 0x25bfc90 "queue_next_trigger"
2012/10/08 20:26:26.454612 [11383]: server/ctdb_daemon.c:628 client request 2147483601 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.454625 [11383]: Reqid wrap!
2012/10/08 20:26:26.454636 [11383]: Added timed event "queue_next_trigger": 0x25bfa10
2012/10/08 20:26:26.454645 [11383]: Added timed event "ctdb_control_timeout": 0x25bfb30
2012/10/08 20:26:26.454655 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454666 [11383]: Added timed event "queue_next_trigger": 0x25bfec0
2012/10/08 20:26:26.454703 [11383]: Ending timer event 0x25bfa10 "queue_next_trigger"
2012/10/08 20:26:26.454714 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 8 length 67 from node 1 to 1
2012/10/08 20:26:26.454728 [11383]: Destroying timer event 0x25bfb30 "ctdb_control_timeout"
2012/10/08 20:26:26.454737 [11383]: Ending timer event 0x25bfec0 "queue_next_trigger"
2012/10/08 20:26:26.454779 [11383]: server/ctdb_daemon.c:628 client request 2147483602 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.454794 [11383]: Added timed event "queue_next_trigger": 0x25b7220
2012/10/08 20:26:26.454804 [11383]: Added timed event "ctdb_control_timeout": 0x25bfe90
2012/10/08 20:26:26.454814 [11383]: server/ctdb_server.c:331 ctdb request 1 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454824 [11383]: Added timed event "queue_next_trigger": 0x25c01e0
2012/10/08 20:26:26.454834 [11383]: Ending timer event 0x25b7220 "queue_next_trigger"
2012/10/08 20:26:26.454843 [11383]: server/ctdb_server.c:331 ctdb request 1 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.454856 [11383]: Destroying timer event 0x25bfe90 "ctdb_control_timeout"
2012/10/08 20:26:26.454866 [11383]: Ending timer event 0x25c01e0 "queue_next_trigger"
2012/10/08 20:26:26.454905 [11383]: server/ctdb_daemon.c:628 client request 2147483603 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454919 [11383]: Added timed event "queue_next_trigger": 0x25c00c0
2012/10/08 20:26:26.454935 [11383]: Added timed event "ctdb_control_timeout": 0x25c01e0
2012/10/08 20:26:26.454946 [11383]: server/ctdb_server.c:331 ctdb request 2 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.454957 [11383]: Added timed event "queue_next_trigger": 0x25c0570
2012/10/08 20:26:26.454967 [11383]: Ending timer event 0x25c00c0 "queue_next_trigger"
2012/10/08 20:26:26.454976 [11383]: server/ctdb_server.c:331 ctdb request 2 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:26.454989 [11383]: Destroying timer event 0x25c01e0 "ctdb_control_timeout"
2012/10/08 20:26:26.454999 [11383]: Ending timer event 0x25c0570 "queue_next_trigger"
2012/10/08 20:26:26.455040 [11383]: server/ctdb_daemon.c:628 client request 2147483604 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455054 [11383]: Added timed event "queue_next_trigger": 0x25bfe90
2012/10/08 20:26:26.455064 [11383]: Added timed event "ctdb_control_timeout": 0x25b7220
2012/10/08 20:26:26.455074 [11383]: server/ctdb_server.c:331 ctdb request 3 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455086 [11383]: Added timed event "queue_next_trigger": 0x25c0950
2012/10/08 20:26:26.455096 [11383]: Ending timer event 0x25bfe90 "queue_next_trigger"
2012/10/08 20:26:26.455105 [11383]: server/ctdb_server.c:331 ctdb request 3 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:26.455118 [11383]: Destroying timer event 0x25b7220 "ctdb_control_timeout"
2012/10/08 20:26:26.455127 [11383]: Ending timer event 0x25c0950 "queue_next_trigger"
2012/10/08 20:26:26.455168 [11383]: server/ctdb_daemon.c:628 client request 2147483605 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455182 [11383]: Added timed event "queue_next_trigger": 0x25c0700
2012/10/08 20:26:26.455191 [11383]: Added timed event "ctdb_control_timeout": 0x25c0820
2012/10/08 20:26:26.455201 [11383]: server/ctdb_server.c:331 ctdb request 4 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455211 [11383]: Added timed event "queue_next_trigger": 0x25c0b70
2012/10/08 20:26:26.455221 [11383]: Ending timer event 0x25c0700 "queue_next_trigger"
2012/10/08 20:26:26.455230 [11383]: server/ctdb_server.c:331 ctdb request 4 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.455243 [11383]: Destroying timer event 0x25c0820 "ctdb_control_timeout"
2012/10/08 20:26:26.455252 [11383]: Ending timer event 0x25c0b70 "queue_next_trigger"
2012/10/08 20:26:26.455291 [11383]: server/ctdb_daemon.c:628 client request 2147483606 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455305 [11383]: Added timed event "queue_next_trigger": 0x25b7220
2012/10/08 20:26:26.455315 [11383]: Added timed event "ctdb_control_timeout": 0x25c0b50
2012/10/08 20:26:26.455325 [11383]: server/ctdb_server.c:331 ctdb request 5 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455349 [11383]: Added timed event "queue_next_trigger": 0x25c0fc0
2012/10/08 20:26:26.455361 [11383]: Ending timer event 0x25b7220 "queue_next_trigger"
2012/10/08 20:26:26.455370 [11383]: server/ctdb_server.c:331 ctdb request 5 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:26.455383 [11383]: Destroying timer event 0x25c0b50 "ctdb_control_timeout"
2012/10/08 20:26:26.455393 [11383]: Ending timer event 0x25c0fc0 "queue_next_trigger"
2012/10/08 20:26:26.455434 [11383]: server/ctdb_daemon.c:628 client request 2147483607 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.455448 [11383]: Added timed event "queue_next_trigger": 0x25c0d70
2012/10/08 20:26:26.455458 [11383]: Added timed event "ctdb_control_timeout": 0x25c0e90
2012/10/08 20:26:26.455467 [11383]: server/ctdb_server.c:331 ctdb request 6 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455510 [11383]: Added timed event "queue_next_trigger": 0x25aa8e0
2012/10/08 20:26:26.455520 [11383]: Ending timer event 0x25c0d70 "queue_next_trigger"
2012/10/08 20:26:26.455530 [11383]: server/ctdb_server.c:331 ctdb request 6 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:26.455543 [11383]: Destroying timer event 0x25c0e90 "ctdb_control_timeout"
2012/10/08 20:26:26.455559 [11383]: Ending timer event 0x25aa8e0 "queue_next_trigger"
2012/10/08 20:26:26.455600 [11383]: server/ctdb_daemon.c:628 client request 2147483608 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.455614 [11383]: Added timed event "queue_next_trigger": 0x25ade00
2012/10/08 20:26:26.455624 [11383]: Added timed event "ctdb_control_timeout": 0x25c0e90
2012/10/08 20:26:26.455633 [11383]: server/ctdb_server.c:331 ctdb request 7 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455645 [11383]: Added timed event "queue_next_trigger": 0x25c0fb0
2012/10/08 20:26:26.455655 [11383]: Ending timer event 0x25ade00 "queue_next_trigger"
2012/10/08 20:26:26.455664 [11383]: server/ctdb_server.c:331 ctdb request 7 of type 8 length 72 from node 1 to 1
2012/10/08 20:26:26.455677 [11383]: Destroying timer event 0x25c0e90 "ctdb_control_timeout"
2012/10/08 20:26:26.455686 [11383]: Ending timer event 0x25c0fb0 "queue_next_trigger"
2012/10/08 20:26:26.455727 [11383]: server/ctdb_daemon.c:628 client request 2147483609 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.455741 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:26.455751 [11383]: Added timed event "ctdb_control_timeout": 0x25b6530
2012/10/08 20:26:26.455761 [11383]: server/ctdb_server.c:331 ctdb request 8 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455772 [11383]: Added timed event "queue_next_trigger": 0x25adb30
2012/10/08 20:26:26.455782 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:26.455791 [11383]: server/ctdb_server.c:331 ctdb request 8 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:26.455804 [11383]: Destroying timer event 0x25b6530 "ctdb_control_timeout"
2012/10/08 20:26:26.455813 [11383]: Ending timer event 0x25adb30 "queue_next_trigger"
2012/10/08 20:26:26.455853 [11383]: server/ctdb_daemon.c:628 client request 2147483610 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.455867 [11383]: Added timed event "queue_next_trigger": 0x25b6530
2012/10/08 20:26:26.455877 [11383]: Added timed event "ctdb_control_timeout": 0x25adb30
2012/10/08 20:26:26.455886 [11383]: server/ctdb_server.c:331 ctdb request 9 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.455897 [11383]: Added timed event "queue_next_trigger": 0x25aa0b0
2012/10/08 20:26:26.455907 [11383]: Ending timer event 0x25b6530 "queue_next_trigger"
2012/10/08 20:26:26.455916 [11383]: server/ctdb_server.c:331 ctdb request 9 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:26.455929 [11383]: Destroying timer event 0x25adb30 "ctdb_control_timeout"
2012/10/08 20:26:26.455938 [11383]: Ending timer event 0x25aa0b0 "queue_next_trigger"
2012/10/08 20:26:26.455991 [11383]: server/ctdb_daemon.c:628 client request 2147483611 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.456005 [11383]: Added timed event "queue_next_trigger": 0x25b6530
2012/10/08 20:26:26.456015 [11383]: Added timed event "ctdb_control_timeout": 0x25b62f0
2012/10/08 20:26:26.456025 [11383]: server/ctdb_server.c:331 ctdb request 10 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.456035 [11383]: Added timed event "queue_next_trigger": 0x25aef60
2012/10/08 20:26:26.456045 [11383]: Ending timer event 0x25b6530 "queue_next_trigger"
2012/10/08 20:26:26.456054 [11383]: server/ctdb_server.c:331 ctdb request 10 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.456067 [11383]: Destroying timer event 0x25b62f0 "ctdb_control_timeout"
2012/10/08 20:26:26.456077 [11383]: Ending timer event 0x25aef60 "queue_next_trigger"
2012/10/08 20:26:26.456117 [11383]: server/ctdb_daemon.c:628 client request 2147483612 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.456131 [11383]: Added timed event "queue_next_trigger": 0x25aeed0
2012/10/08 20:26:26.456141 [11383]: Added timed event "ctdb_control_timeout": 0x25b1a30
2012/10/08 20:26:26.456151 [11383]: server/ctdb_server.c:331 ctdb request 11 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.456162 [11383]: Added timed event "queue_next_trigger": 0x25b1d40
2012/10/08 20:26:26.456178 [11383]: Ending timer event 0x25aeed0 "queue_next_trigger"
2012/10/08 20:26:26.456188 [11383]: server/ctdb_server.c:331 ctdb request 11 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.456201 [11383]: Destroying timer event 0x25b1a30 "ctdb_control_timeout"
2012/10/08 20:26:26.456211 [11383]: Ending timer event 0x25b1d40 "queue_next_trigger"
2012/10/08 20:26:26.456603 [recoverd:11454]: server/ctdb_recoverd.c:2810 Created PIPE FD:17 for check_recovery_lock
2012/10/08 20:26:26.459084 [recoverd:11454]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:26:26.459099 [recoverd:11454]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:26:26.460391 [11383]: server/ctdb_daemon.c:628 client request 2147483613 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:26.460408 [11383]: Added timed event "queue_next_trigger": 0x25b6530
2012/10/08 20:26:26.460418 [11383]: Added timed event "ctdb_control_timeout": 0x25b1d20
2012/10/08 20:26:26.460429 [11383]: server/ctdb_server.c:331 ctdb request 12 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:26.460440 [11383]: Added timed event "queue_next_trigger": 0x25b9840
2012/10/08 20:26:26.460450 [11383]: Ending timer event 0x25b6530 "queue_next_trigger"
2012/10/08 20:26:26.460459 [11383]: server/ctdb_server.c:331 ctdb request 12 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.460473 [11383]: Destroying timer event 0x25b1d20 "ctdb_control_timeout"
2012/10/08 20:26:26.460483 [11383]: Ending timer event 0x25b9840 "queue_next_trigger"
2012/10/08 20:26:26.460549 [recoverd:11454]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:26:26.460562 [recoverd:11454]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:26:26.460571 [recoverd:11454]: Taking out recovery lock from recovery daemon
2012/10/08 20:26:26.460579 [recoverd:11454]: Take the recovery lock
2012/10/08 20:26:26.466494 [recoverd:11454]: Recovery lock taken successfully
2012/10/08 20:26:26.466509 [recoverd:11454]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:26:26.466538 [11383]: server/ctdb_daemon.c:628 client request 2147483614 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:26.466553 [11383]: Added timed event "queue_next_trigger": 0x25b6530
2012/10/08 20:26:26.466563 [11383]: Added timed event "ctdb_control_timeout": 0x25b9a40
2012/10/08 20:26:26.466574 [11383]: server/ctdb_server.c:331 ctdb request 13 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:26.466585 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:26:26.466595 [11383]: Ending timer event 0x25b6530 "queue_next_trigger"
2012/10/08 20:26:26.466604 [11383]: server/ctdb_server.c:331 ctdb request 13 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.466617 [11383]: Destroying timer event 0x25b9a40 "ctdb_control_timeout"
2012/10/08 20:26:26.466627 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:26:26.466656 [recoverd:11454]: RECD SIGCHLD from 12103
2012/10/08 20:26:26.466675 [recoverd:11454]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:26:26.466685 [recoverd:11454]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 1
2012/10/08 20:26:26.466710 [11383]: server/ctdb_daemon.c:628 client request 2147483615 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.466724 [11383]: Added timed event "queue_next_trigger": 0x25b6530
2012/10/08 20:26:26.466734 [11383]: Added timed event "ctdb_control_timeout": 0x25c50b0
2012/10/08 20:26:26.466744 [11383]: server/ctdb_server.c:331 ctdb request 14 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.466755 [11383]: Added timed event "queue_next_trigger": 0x25b57e0
2012/10/08 20:26:26.466765 [11383]: Ending timer event 0x25b6530 "queue_next_trigger"
2012/10/08 20:26:26.466775 [11383]: server/ctdb_server.c:331 ctdb request 14 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.466787 [11383]: Destroying timer event 0x25c50b0 "ctdb_control_timeout"
2012/10/08 20:26:26.466803 [11383]: Ending timer event 0x25b57e0 "queue_next_trigger"
2012/10/08 20:26:26.466832 [recoverd:11454]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:26:26.466844 [recoverd:11454]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:26:26.466870 [11383]: server/ctdb_daemon.c:628 client request 2147483616 of type 7 length 88 from node 1 to 1
2012/10/08 20:26:26.466884 [11383]: Added timed event "queue_next_trigger": 0x25c50b0
2012/10/08 20:26:26.466894 [11383]: Added timed event "ctdb_control_timeout": 0x25aa0c0
2012/10/08 20:26:26.466904 [11383]: server/ctdb_server.c:331 ctdb request 15 of type 7 length 83 from node 1 to 1
2012/10/08 20:26:26.466915 [11383]: Added timed event "queue_next_trigger": 0x25b5f40
2012/10/08 20:26:26.466924 [11383]: Ending timer event 0x25c50b0 "queue_next_trigger"
2012/10/08 20:26:26.466934 [11383]: server/ctdb_server.c:331 ctdb request 15 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.466947 [11383]: Destroying timer event 0x25aa0c0 "ctdb_control_timeout"
2012/10/08 20:26:26.466957 [11383]: Ending timer event 0x25b5f40 "queue_next_trigger"
2012/10/08 20:26:26.466999 [11383]: server/ctdb_daemon.c:628 client request 2147483617 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.467013 [11383]: Added timed event "queue_next_trigger": 0x25c50b0
2012/10/08 20:26:26.467023 [11383]: Added timed event "ctdb_control_timeout": 0x25b5a50
2012/10/08 20:26:26.467033 [11383]: server/ctdb_server.c:331 ctdb request 16 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.467042 [11383]: Freeze priority 1
2012/10/08 20:26:26.467557 [11383]: Ending timer event 0x25c50b0 "queue_next_trigger"
2012/10/08 20:26:26.467936 [11383]: Added timed event "queue_next_trigger": 0x25b6ae0
2012/10/08 20:26:26.467967 [11383]: server/ctdb_server.c:331 ctdb request 16 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.468026 [11383]: Destroying timer event 0x25b5a50 "ctdb_control_timeout"
2012/10/08 20:26:26.468038 [11383]: Ending timer event 0x25b6ae0 "queue_next_trigger"
2012/10/08 20:26:26.468082 [11383]: server/ctdb_daemon.c:628 client request 2147483618 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.468114 [11383]: Added timed event "queue_next_trigger": 0x25b6a50
2012/10/08 20:26:26.468126 [11383]: Added timed event "ctdb_control_timeout": 0x25b6f90
2012/10/08 20:26:26.468136 [11383]: server/ctdb_server.c:331 ctdb request 17 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.468145 [11383]: Freeze priority 2
2012/10/08 20:26:26.468599 [11383]: Ending timer event 0x25b6a50 "queue_next_trigger"
2012/10/08 20:26:26.469026 [11383]: Added timed event "queue_next_trigger": 0x25b5d10
2012/10/08 20:26:26.469058 [11383]: server/ctdb_server.c:331 ctdb request 17 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.469120 [11383]: Destroying timer event 0x25b6f90 "ctdb_control_timeout"
2012/10/08 20:26:26.469132 [11383]: Ending timer event 0x25b5d10 "queue_next_trigger"
2012/10/08 20:26:26.469176 [11383]: server/ctdb_daemon.c:628 client request 2147483619 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.469190 [11383]: Added timed event "queue_next_trigger": 0x25b5350
2012/10/08 20:26:26.469200 [11383]: Added timed event "ctdb_control_timeout": 0x25b5470
2012/10/08 20:26:26.469210 [11383]: server/ctdb_server.c:331 ctdb request 18 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.469220 [11383]: Freeze priority 3
2012/10/08 20:26:26.469650 [11383]: Ending timer event 0x25b5350 "queue_next_trigger"
2012/10/08 20:26:26.470090 [11383]: Added timed event "queue_next_trigger": 0x25b4dd0
2012/10/08 20:26:26.470105 [11383]: server/ctdb_server.c:331 ctdb request 18 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.470178 [11383]: Destroying timer event 0x25b5470 "ctdb_control_timeout"
2012/10/08 20:26:26.470206 [11383]: Ending timer event 0x25b4dd0 "queue_next_trigger"
2012/10/08 20:26:26.470277 [11383]: server/ctdb_daemon.c:628 client request 2147483620 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.470300 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:26:26.470311 [11383]: Added timed event "ctdb_control_timeout": 0x25b4f60
2012/10/08 20:26:26.470321 [11383]: server/ctdb_server.c:331 ctdb request 19 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.470331 [11383]: Added timed event "ctdb_drop_all_ips_event": 0x25b5080
2012/10/08 20:26:26.470340 [11383]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:26:26.470350 [11383]: Added timed event "queue_next_trigger": 0x25c52a0
2012/10/08 20:26:26.470359 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:26:26.470368 [11383]: server/ctdb_server.c:331 ctdb request 19 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.470381 [11383]: Destroying timer event 0x25b4f60 "ctdb_control_timeout"
2012/10/08 20:26:26.470391 [11383]: Ending timer event 0x25c52a0 "queue_next_trigger"
2012/10/08 20:26:26.470433 [11383]: server/ctdb_daemon.c:628 client request 2147483621 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.470468 [11383]: Added timed event "queue_next_trigger": 0x25b6e00
2012/10/08 20:26:26.470480 [11383]: Added timed event "ctdb_control_timeout": 0x25c5280
2012/10/08 20:26:26.470490 [11383]: server/ctdb_server.c:331 ctdb request 20 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.470499 [11383]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:26:26.470508 [11383]: Monitoring has been disabled
2012/10/08 20:26:26.470520 [11383]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/08 20:26:26.470711 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:26.470747 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:26.470762 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:26.470780 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:26.471250 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.471337 [11383]: Added timed event "ctdb_event_script_timeout": 0x25adff0
2012/10/08 20:26:26.471390 [11383]: Ending timer event 0x25b6e00 "queue_next_trigger"
2012/10/08 20:26:26.471873 [eventscript-00.ctdb-startrecovery:12107]: Executing event script /etc/ctdb/events.d/00.ctdb startrecovery
2012/10/08 20:26:26.484558 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.485020 [11383]: SIGCHLD from 12107
2012/10/08 20:26:26.485314 [eventscript-01.reclock-startrecovery:12112]: Executing event script /etc/ctdb/events.d/01.reclock startrecovery
2012/10/08 20:26:26.494668 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.495142 [11383]: SIGCHLD from 12112
2012/10/08 20:26:26.495442 [eventscript-10.interface-startrecovery:12116]: Executing event script /etc/ctdb/events.d/10.interface startrecovery
2012/10/08 20:26:26.505280 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.505760 [11383]: SIGCHLD from 12116
2012/10/08 20:26:26.506041 [eventscript-11.natgw-startrecovery:12120]: Executing event script /etc/ctdb/events.d/11.natgw startrecovery
2012/10/08 20:26:26.515232 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.515706 [11383]: SIGCHLD from 12120
2012/10/08 20:26:26.515998 [eventscript-11.routing-startrecovery:12124]: Executing event script /etc/ctdb/events.d/11.routing startrecovery
2012/10/08 20:26:26.525099 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.525537 [11383]: SIGCHLD from 12124
2012/10/08 20:26:26.525833 [eventscript-13.per_ip_routing-startrecovery:12128]: Executing event script /etc/ctdb/events.d/13.per_ip_routing startrecovery
2012/10/08 20:26:26.537632 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.538091 [11383]: SIGCHLD from 12128
2012/10/08 20:26:26.538406 [eventscript-20.multipathd-startrecovery:12133]: Executing event script /etc/ctdb/events.d/20.multipathd startrecovery
2012/10/08 20:26:26.538944 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.539635 [eventscript-31.clamd-startrecovery:12134]: Executing event script /etc/ctdb/events.d/31.clamd startrecovery
2012/10/08 20:26:26.540188 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.540517 [11383]: SIGCHLD from 12133
2012/10/08 20:26:26.540803 [eventscript-40.vsftpd-startrecovery:12135]: Executing event script /etc/ctdb/events.d/40.vsftpd startrecovery
2012/10/08 20:26:26.546389 [11383]: SIGCHLD from 12134
2012/10/08 20:26:26.550307 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.550740 [11383]: SIGCHLD from 12135
2012/10/08 20:26:26.551060 [eventscript-40.fs_use-startrecovery:12139]: Executing event script /etc/ctdb/events.d/40.fs_use startrecovery
2012/10/08 20:26:26.551638 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.552349 [eventscript-41.httpd-startrecovery:12140]: Executing event script /etc/ctdb/events.d/41.httpd startrecovery
2012/10/08 20:26:26.554287 [11383]: SIGCHLD from 12139
2012/10/08 20:26:26.561687 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.562156 [11383]: SIGCHLD from 12140
2012/10/08 20:26:26.562471 [eventscript-50.samba-startrecovery:12144]: Executing event script /etc/ctdb/events.d/50.samba startrecovery
2012/10/08 20:26:26.574883 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.575352 [11383]: SIGCHLD from 12144
2012/10/08 20:26:26.575702 [eventscript-60.nfs-startrecovery:12149]: Executing event script /etc/ctdb/events.d/60.nfs startrecovery
2012/10/08 20:26:26.590310 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.590777 [11383]: SIGCHLD from 12149
2012/10/08 20:26:26.591087 [eventscript-60.ganesha-startrecovery:12155]: Executing event script /etc/ctdb/events.d/60.ganesha startrecovery
2012/10/08 20:26:26.591684 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.592426 [eventscript-62.cnfs-startrecovery:12156]: Executing event script /etc/ctdb/events.d/62.cnfs startrecovery
2012/10/08 20:26:26.594725 [11383]: SIGCHLD from 12155
2012/10/08 20:26:26.605684 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.606170 [11383]: SIGCHLD from 12156
2012/10/08 20:26:26.606498 [eventscript-70.iscsi-startrecovery:12161]: Executing event script /etc/ctdb/events.d/70.iscsi startrecovery
2012/10/08 20:26:26.615450 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:26.615922 [11383]: SIGCHLD from 12161
2012/10/08 20:26:26.616224 [eventscript-91.lvs-startrecovery:12165]: Executing event script /etc/ctdb/events.d/91.lvs startrecovery
2012/10/08 20:26:26.624928 [11383]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/08 20:26:26.624993 [11383]: Added timed event "queue_next_trigger": 0x25b66a0
2012/10/08 20:26:26.625012 [11383]: Destroying timer event 0x25adff0 "ctdb_event_script_timeout"
2012/10/08 20:26:26.625024 [11383]: server/ctdb_server.c:331 ctdb request 20 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.625135 [11383]: Destroying timer event 0x25c5280 "ctdb_control_timeout"
2012/10/08 20:26:26.625149 [11383]: Ending timer event 0x25b66a0 "queue_next_trigger"
2012/10/08 20:26:26.625213 [11383]: server/ctdb_daemon.c:628 client request 2147483622 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.625230 [11383]: Added timed event "queue_next_trigger": 0x25aad80
2012/10/08 20:26:26.625252 [11383]: Added timed event "ctdb_control_timeout": 0x25b43d0
2012/10/08 20:26:26.625264 [11383]: server/ctdb_server.c:331 ctdb request 21 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.625275 [11383]: Added timed event "queue_next_trigger": 0x25b1f40
2012/10/08 20:26:26.625285 [11383]: Ending timer event 0x25aad80 "queue_next_trigger"
2012/10/08 20:26:26.625294 [11383]: server/ctdb_server.c:331 ctdb request 21 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.625307 [11383]: Destroying timer event 0x25b43d0 "ctdb_control_timeout"
2012/10/08 20:26:26.625317 [11383]: Ending timer event 0x25b1f40 "queue_next_trigger"
2012/10/08 20:26:26.625358 [11383]: server/ctdb_daemon.c:628 client request 2147483623 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.625372 [11383]: Added timed event "queue_next_trigger": 0x25ad830
2012/10/08 20:26:26.625381 [11383]: Added timed event "ctdb_control_timeout": 0x25b43d0
2012/10/08 20:26:26.625391 [11383]: server/ctdb_server.c:331 ctdb request 22 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.625405 [11383]: Added timed event "queue_next_trigger": 0x25b6900
2012/10/08 20:26:26.625415 [11383]: Ending timer event 0x25ad830 "queue_next_trigger"
2012/10/08 20:26:26.625424 [11383]: server/ctdb_server.c:331 ctdb request 22 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:26.625437 [11383]: Destroying timer event 0x25b43d0 "ctdb_control_timeout"
2012/10/08 20:26:26.625446 [11383]: Ending timer event 0x25b6900 "queue_next_trigger"
2012/10/08 20:26:26.625488 [11383]: server/ctdb_daemon.c:628 client request 2147483624 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.625502 [11383]: Added timed event "queue_next_trigger": 0x25ae290
2012/10/08 20:26:26.625512 [11383]: Added timed event "ctdb_control_timeout": 0x25b5fb0
2012/10/08 20:26:26.625522 [11383]: server/ctdb_server.c:331 ctdb request 23 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.625532 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:26.625542 [11383]: Added timed event "queue_next_trigger": 0x25b6830
2012/10/08 20:26:26.625552 [11383]: Ending timer event 0x25ae290 "queue_next_trigger"
2012/10/08 20:26:26.625561 [11383]: server/ctdb_server.c:331 ctdb request 23 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.625573 [11383]: Destroying timer event 0x25b5fb0 "ctdb_control_timeout"
2012/10/08 20:26:26.625583 [11383]: Ending timer event 0x25b6830 "queue_next_trigger"
2012/10/08 20:26:26.625610 [recoverd:11454]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:26:26.625641 [11383]: server/ctdb_daemon.c:628 client request 2147483625 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.625654 [11383]: Added timed event "queue_next_trigger": 0x25ad830
2012/10/08 20:26:26.625664 [11383]: Added timed event "ctdb_control_timeout": 0x25a9fe0
2012/10/08 20:26:26.625674 [11383]: server/ctdb_server.c:331 ctdb request 24 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.625739 [11383]: Added timed event "queue_next_trigger": 0x25c4ef0
2012/10/08 20:26:26.625751 [11383]: Ending timer event 0x25ad830 "queue_next_trigger"
2012/10/08 20:26:26.625761 [11383]: server/ctdb_server.c:331 ctdb request 24 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.625812 [11383]: Destroying timer event 0x25a9fe0 "ctdb_control_timeout"
2012/10/08 20:26:26.625823 [11383]: Ending timer event 0x25c4ef0 "queue_next_trigger"
2012/10/08 20:26:26.625865 [11383]: server/ctdb_daemon.c:628 client request 2147483626 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.625879 [11383]: Added timed event "queue_next_trigger": 0x25b1f40
2012/10/08 20:26:26.625889 [11383]: Added timed event "ctdb_control_timeout": 0x25b62f0
2012/10/08 20:26:26.625899 [11383]: server/ctdb_server.c:331 ctdb request 25 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.625910 [11383]: Added timed event "queue_next_trigger": 0x25aeff0
2012/10/08 20:26:26.625919 [11383]: Ending timer event 0x25b1f40 "queue_next_trigger"
2012/10/08 20:26:26.625942 [11383]: server/ctdb_server.c:331 ctdb request 25 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.625956 [11383]: Destroying timer event 0x25b62f0 "ctdb_control_timeout"
2012/10/08 20:26:26.625966 [11383]: Ending timer event 0x25aeff0 "queue_next_trigger"
2012/10/08 20:26:26.626154 [11383]: SIGCHLD from 12165
2012/10/08 20:26:26.626188 [recoverd:11454]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:26:26.626201 [recoverd:11454]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:26:26.626253 [11383]: server/ctdb_daemon.c:628 client request 2147483627 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626269 [11383]: Added timed event "queue_next_trigger": 0x25aeed0
2012/10/08 20:26:26.626280 [11383]: Added timed event "ctdb_control_timeout": 0x25aeff0
2012/10/08 20:26:26.626290 [11383]: server/ctdb_server.c:331 ctdb request 26 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626300 [11383]: server/ctdb_freeze.c:523 num_nodes[2]
2012/10/08 20:26:26.626308 [11383]: server/ctdb_freeze.c:526 node[0].flags[0x3]
2012/10/08 20:26:26.626316 [11383]: server/ctdb_freeze.c:526 node[1].flags[0x2]
2012/10/08 20:26:26.626324 [11383]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/08 20:26:26.626333 [11383]: Added timed event "queue_next_trigger": 0x25c5210
2012/10/08 20:26:26.626343 [11383]: Ending timer event 0x25aeed0 "queue_next_trigger"
2012/10/08 20:26:26.626352 [11383]: server/ctdb_server.c:331 ctdb request 26 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.626365 [11383]: Destroying timer event 0x25aeff0 "ctdb_control_timeout"
2012/10/08 20:26:26.626375 [11383]: Ending timer event 0x25c5210 "queue_next_trigger"
2012/10/08 20:26:26.626401 [recoverd:11454]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:26:26.626429 [11383]: server/ctdb_daemon.c:628 client request 2147483628 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626443 [11383]: Added timed event "queue_next_trigger": 0x25c5210
2012/10/08 20:26:26.626453 [11383]: Added timed event "ctdb_control_timeout": 0x25b1d20
2012/10/08 20:26:26.626463 [11383]: server/ctdb_server.c:331 ctdb request 27 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626474 [11383]: Added timed event "queue_next_trigger": 0x25b5d80
2012/10/08 20:26:26.626484 [11383]: Ending timer event 0x25c5210 "queue_next_trigger"
2012/10/08 20:26:26.626493 [11383]: server/ctdb_server.c:331 ctdb request 27 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.626506 [11383]: Destroying timer event 0x25b1d20 "ctdb_control_timeout"
2012/10/08 20:26:26.626516 [11383]: Ending timer event 0x25b5d80 "queue_next_trigger"
2012/10/08 20:26:26.626558 [11383]: server/ctdb_daemon.c:628 client request 2147483629 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.626574 [11383]: Added timed event "queue_next_trigger": 0x25b62f0
2012/10/08 20:26:26.626584 [11383]: Added timed event "ctdb_control_timeout": 0x25b5d60
2012/10/08 20:26:26.626593 [11383]: server/ctdb_server.c:331 ctdb request 28 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.626605 [11383]: Added timed event "queue_next_trigger": 0x25b9a40
2012/10/08 20:26:26.626614 [11383]: Ending timer event 0x25b62f0 "queue_next_trigger"
2012/10/08 20:26:26.626623 [11383]: server/ctdb_server.c:331 ctdb request 28 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.626636 [11383]: Destroying timer event 0x25b5d60 "ctdb_control_timeout"
2012/10/08 20:26:26.626646 [11383]: Ending timer event 0x25b9a40 "queue_next_trigger"
2012/10/08 20:26:26.626670 [recoverd:11454]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:26:26.626697 [11383]: server/ctdb_daemon.c:628 client request 2147483630 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626711 [11383]: Added timed event "queue_next_trigger": 0x25b4eb0
2012/10/08 20:26:26.626720 [11383]: Added timed event "ctdb_control_timeout": 0x25b4fd0
2012/10/08 20:26:26.626730 [11383]: server/ctdb_server.c:331 ctdb request 29 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626759 [11383]: Added timed event "queue_next_trigger": 0x25b57e0
2012/10/08 20:26:26.626770 [11383]: Ending timer event 0x25b4eb0 "queue_next_trigger"
2012/10/08 20:26:26.626780 [11383]: server/ctdb_server.c:331 ctdb request 29 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.626793 [11383]: Destroying timer event 0x25b4fd0 "ctdb_control_timeout"
2012/10/08 20:26:26.626802 [11383]: Ending timer event 0x25b57e0 "queue_next_trigger"
2012/10/08 20:26:26.626829 [recoverd:11454]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:26:26.626857 [11383]: server/ctdb_daemon.c:628 client request 2147483631 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:26.626871 [11383]: Added timed event "queue_next_trigger": 0x25b5d60
2012/10/08 20:26:26.626880 [11383]: Added timed event "ctdb_control_timeout": 0x25b9a00
2012/10/08 20:26:26.626890 [11383]: server/ctdb_server.c:331 ctdb request 30 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626901 [11383]: Added timed event "queue_next_trigger": 0x25b6a50
2012/10/08 20:26:26.626910 [11383]: Ending timer event 0x25b5d60 "queue_next_trigger"
2012/10/08 20:26:26.626920 [11383]: server/ctdb_server.c:331 ctdb request 30 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.626932 [11383]: Destroying timer event 0x25b9a00 "ctdb_control_timeout"
2012/10/08 20:26:26.626942 [11383]: Ending timer event 0x25b6a50 "queue_next_trigger"
2012/10/08 20:26:26.626980 [11383]: server/ctdb_daemon.c:628 client request 2147483632 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.626994 [11383]: Added timed event "queue_next_trigger": 0x25b9a00
2012/10/08 20:26:26.627004 [11383]: Added timed event "ctdb_control_timeout": 0x25b5a50
2012/10/08 20:26:26.627014 [11383]: server/ctdb_server.c:331 ctdb request 31 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.627026 [11383]: Added timed event "queue_next_trigger": 0x25b6fa0
2012/10/08 20:26:26.627035 [11383]: Ending timer event 0x25b9a00 "queue_next_trigger"
2012/10/08 20:26:26.627045 [11383]: server/ctdb_server.c:331 ctdb request 31 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:26.627057 [11383]: Destroying timer event 0x25b5a50 "ctdb_control_timeout"
2012/10/08 20:26:26.627067 [11383]: Ending timer event 0x25b6fa0 "queue_next_trigger"
2012/10/08 20:26:26.627107 [11383]: server/ctdb_daemon.c:628 client request 2147483633 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.627121 [11383]: Added timed event "queue_next_trigger": 0x25b9a00
2012/10/08 20:26:26.627131 [11383]: Added timed event "ctdb_control_timeout": 0x25b6e70
2012/10/08 20:26:26.627140 [11383]: server/ctdb_server.c:331 ctdb request 32 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:26.627150 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:26.627160 [11383]: Added timed event "queue_next_trigger": 0x25b5470
2012/10/08 20:26:26.627169 [11383]: Ending timer event 0x25b9a00 "queue_next_trigger"
2012/10/08 20:26:26.627178 [11383]: server/ctdb_server.c:331 ctdb request 32 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.627191 [11383]: Destroying timer event 0x25b6e70 "ctdb_control_timeout"
2012/10/08 20:26:26.627201 [11383]: Ending timer event 0x25b5470 "queue_next_trigger"
2012/10/08 20:26:26.627226 [recoverd:11454]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:26:26.627254 [11383]: server/ctdb_daemon.c:628 client request 2147483634 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.627267 [11383]: Added timed event "queue_next_trigger": 0x25b5600
2012/10/08 20:26:26.627277 [11383]: Added timed event "ctdb_control_timeout": 0x25b5350
2012/10/08 20:26:26.627287 [11383]: server/ctdb_server.c:331 ctdb request 33 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.627297 [11383]: Destroying timer event 0x25b5080 "ctdb_drop_all_ips_event"
2012/10/08 20:26:26.627306 [11383]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:26:26.627316 [11383]: Thawing priority 1
2012/10/08 20:26:26.627324 [11383]: Release freeze handler for prio 1
2012/10/08 20:26:26.627358 [11383]: Thawing priority 2
2012/10/08 20:26:26.627368 [11383]: Release freeze handler for prio 2
2012/10/08 20:26:26.627382 [11383]: Thawing priority 3
2012/10/08 20:26:26.627391 [11383]: Release freeze handler for prio 3
2012/10/08 20:26:26.627761 [11383]: server/ctdb_recover.c:808 Created PIPE FD:11 for setrecmode
2012/10/08 20:26:26.627855 [11383]: Added timed event "ctdb_set_recmode_timeout": 0x25b2240
2012/10/08 20:26:26.627947 [11383]: Ending timer event 0x25b5600 "queue_next_trigger"
2012/10/08 20:26:26.628833 [11383]: SIGCHLD from 12104
2012/10/08 20:26:26.628849 [11383]: SIGCHLD from 12105
2012/10/08 20:26:26.628862 [11383]: SIGCHLD from 12106
2012/10/08 20:26:26.633138 [11383]: Destroying timer event 0x25b2240 "ctdb_set_recmode_timeout"
2012/10/08 20:26:26.633168 [11383]: Added timed event "queue_next_trigger": 0x25b25f0
2012/10/08 20:26:26.633184 [11383]: server/ctdb_server.c:331 ctdb request 33 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.633231 [11383]: Destroying timer event 0x25b5350 "ctdb_control_timeout"
2012/10/08 20:26:26.633240 [11383]: Ending timer event 0x25b25f0 "queue_next_trigger"
2012/10/08 20:26:26.633366 [11383]: SIGCHLD from 12169
2012/10/08 20:26:26.633394 [recoverd:11454]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:26:26.633418 [11383]: server/ctdb_daemon.c:628 client request 2147483635 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.633429 [11383]: Added timed event "queue_next_trigger": 0x25b9a00
2012/10/08 20:26:26.633436 [11383]: Added timed event "ctdb_control_timeout": 0x25b25d0
2012/10/08 20:26:26.633444 [11383]: server/ctdb_server.c:331 ctdb request 34 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.633453 [11383]: Added timed event "queue_next_trigger": 0x25b2950
2012/10/08 20:26:26.633460 [11383]: Ending timer event 0x25b9a00 "queue_next_trigger"
2012/10/08 20:26:26.633467 [11383]: server/ctdb_server.c:331 ctdb request 34 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:26.633493 [11383]: Destroying timer event 0x25b25d0 "ctdb_control_timeout"
2012/10/08 20:26:26.633501 [11383]: Ending timer event 0x25b2950 "queue_next_trigger"
2012/10/08 20:26:26.633534 [11383]: server/ctdb_daemon.c:628 client request 2147483636 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.633545 [11383]: Added timed event "queue_next_trigger": 0x25b27f0
2012/10/08 20:26:26.633552 [11383]: Added timed event "ctdb_control_timeout": 0x25b2910
2012/10/08 20:26:26.633559 [11383]: server/ctdb_server.c:331 ctdb request 35 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.633568 [11383]: Added timed event "queue_next_trigger": 0x25bc5f0
2012/10/08 20:26:26.633575 [11383]: Ending timer event 0x25b27f0 "queue_next_trigger"
2012/10/08 20:26:26.633582 [11383]: server/ctdb_server.c:331 ctdb request 35 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.633592 [11383]: Destroying timer event 0x25b2910 "ctdb_control_timeout"
2012/10/08 20:26:26.633599 [11383]: Ending timer event 0x25bc5f0 "queue_next_trigger"
2012/10/08 20:26:26.633624 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:26.633632 [recoverd:11454]: CONSIDERING MOVES (UNASSIGNED)
2012/10/08 20:26:26.633638 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:26.633647 [recoverd:11454]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:26:26.633666 [11383]: server/ctdb_daemon.c:628 client request 2147483637 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:26.633677 [11383]: Added timed event "queue_next_trigger": 0x25bc4d0
2012/10/08 20:26:26.633684 [11383]: Added timed event "ctdb_control_timeout": 0x25bc5f0
2012/10/08 20:26:26.633692 [11383]: server/ctdb_server.c:331 ctdb request 36 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:26.633729 [11383]: Redundant release of IP 10.100.55.198/24 on interface __none__ (ip not held)
2012/10/08 20:26:26.633739 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/08 20:26:26.633762 [11383]: Added timed event "queue_next_trigger": 0x25bc910
2012/10/08 20:26:26.633770 [11383]: Ending timer event 0x25bc4d0 "queue_next_trigger"
2012/10/08 20:26:26.633777 [11383]: server/ctdb_server.c:331 ctdb request 36 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.633787 [11383]: Destroying timer event 0x25bc5f0 "ctdb_control_timeout"
2012/10/08 20:26:26.633795 [11383]: Ending timer event 0x25bc910 "queue_next_trigger"
2012/10/08 20:26:26.633828 [11383]: server/ctdb_daemon.c:628 client request 2147483638 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:26.633838 [11383]: Added timed event "queue_next_trigger": 0x25bc7f0
2012/10/08 20:26:26.633846 [11383]: Added timed event "ctdb_control_timeout": 0x25bc910
2012/10/08 20:26:26.633853 [11383]: server/ctdb_server.c:331 ctdb request 37 of type 7 length 74 from node 1 to 1
2012/10/08 20:26:26.633863 [11383]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:26:26.633878 [11383]: Monitoring has been disabled
2012/10/08 20:26:26.633888 [11383]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/08 20:26:26.633978 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:26.634006 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:26.634017 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:26.634031 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:26.634443 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.634508 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ade00
2012/10/08 20:26:26.634528 [11383]: Ending timer event 0x25bc7f0 "queue_next_trigger"
2012/10/08 20:26:26.634871 [eventscript-00.ctdb-ipreallocated:12170]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/00.ctdb ipreallocated
2012/10/08 20:26:26.643874 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.644214 [11383]: SIGCHLD from 12170
2012/10/08 20:26:26.644427 [eventscript-01.reclock-ipreallocated:12175]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/01.reclock ipreallocated
2012/10/08 20:26:26.651208 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.651557 [11383]: SIGCHLD from 12175
2012/10/08 20:26:26.651767 [eventscript-10.interface-ipreallocated:12179]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/10.interface ipreallocated
2012/10/08 20:26:26.658700 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.659073 [11383]: SIGCHLD from 12179
2012/10/08 20:26:26.659276 [eventscript-11.natgw-ipreallocated:12183]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.natgw ipreallocated
2012/10/08 20:26:26.665925 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.666303 [11383]: SIGCHLD from 12183
2012/10/08 20:26:26.666513 [eventscript-11.routing-ipreallocated:12187]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.routing ipreallocated
2012/10/08 20:26:26.673198 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.673544 [11383]: SIGCHLD from 12187
2012/10/08 20:26:26.673758 [eventscript-13.per_ip_routing-ipreallocated:12191]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/13.per_ip_routing ipreallocated
2012/10/08 20:26:26.682513 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.682867 [11383]: SIGCHLD from 12191
2012/10/08 20:26:26.683083 [eventscript-20.multipathd-ipreallocated:12196]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/20.multipathd ipreallocated
2012/10/08 20:26:26.683507 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.684028 [eventscript-31.clamd-ipreallocated:12197]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/31.clamd ipreallocated
2012/10/08 20:26:26.684450 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.684677 [11383]: SIGCHLD from 12196
2012/10/08 20:26:26.684892 [eventscript-40.vsftpd-ipreallocated:12198]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.vsftpd ipreallocated
2012/10/08 20:26:26.689052 [11383]: SIGCHLD from 12197
2012/10/08 20:26:26.691875 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.692239 [11383]: SIGCHLD from 12198
2012/10/08 20:26:26.692448 [eventscript-40.fs_use-ipreallocated:12202]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.fs_use ipreallocated
2012/10/08 20:26:26.692859 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.693390 [eventscript-41.httpd-ipreallocated:12203]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/41.httpd ipreallocated
2012/10/08 20:26:26.697410 [11383]: SIGCHLD from 12202
2012/10/08 20:26:26.700291 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.700640 [11383]: SIGCHLD from 12203
2012/10/08 20:26:26.700859 [eventscript-50.samba-ipreallocated:12207]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/50.samba ipreallocated
2012/10/08 20:26:26.710007 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.710379 [11383]: SIGCHLD from 12207
2012/10/08 20:26:26.710588 [eventscript-60.nfs-ipreallocated:12212]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.nfs ipreallocated
2012/10/08 20:26:26.721290 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.721656 [11383]: SIGCHLD from 12212
2012/10/08 20:26:26.721868 [eventscript-60.ganesha-ipreallocated:12218]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.ganesha ipreallocated
2012/10/08 20:26:26.722327 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.722845 [eventscript-62.cnfs-ipreallocated:12219]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/62.cnfs ipreallocated
2012/10/08 20:26:26.726889 [11383]: SIGCHLD from 12218
2012/10/08 20:26:26.731653 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.731990 [11383]: SIGCHLD from 12219
2012/10/08 20:26:26.732201 [eventscript-70.iscsi-ipreallocated:12224]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/70.iscsi ipreallocated
2012/10/08 20:26:26.738840 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.739203 [11383]: SIGCHLD from 12224
2012/10/08 20:26:26.739413 [eventscript-91.lvs-ipreallocated:12228]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/91.lvs ipreallocated
2012/10/08 20:26:26.745699 [11383]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/08 20:26:26.745773 [11383]: Monitoring has been enabled
2012/10/08 20:26:26.745797 [11383]: Added timed event "queue_next_trigger": 0x25b6530
2012/10/08 20:26:26.745810 [11383]: Destroying timer event 0x25ade00 "ctdb_event_script_timeout"
2012/10/08 20:26:26.745819 [11383]: server/ctdb_server.c:331 ctdb request 37 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.745860 [11383]: Destroying timer event 0x25bc910 "ctdb_control_timeout"
2012/10/08 20:26:26.745869 [11383]: Ending timer event 0x25b6530 "queue_next_trigger"
2012/10/08 20:26:26.745916 [11383]: server/ctdb_daemon.c:628 client request 2147483639 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.745928 [11383]: Added timed event "queue_next_trigger": 0x25aa8e0
2012/10/08 20:26:26.745935 [11383]: Added timed event "ctdb_control_timeout": 0x25adff0
2012/10/08 20:26:26.745949 [11383]: server/ctdb_server.c:331 ctdb request 38 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:26.745958 [11383]: Recovery has finished
2012/10/08 20:26:26.745964 [11383]: Monitoring has been disabled
2012/10/08 20:26:26.745973 [11383]: server/eventscript.c:754 Starting eventscript recovered
2012/10/08 20:26:26.746042 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:26.746082 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:26.746094 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:26.746108 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:26.746500 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.746590 [11383]: Added timed event "ctdb_event_script_timeout": 0x25b1f40
2012/10/08 20:26:26.746621 [11383]: Ending timer event 0x25aa8e0 "queue_next_trigger"
2012/10/08 20:26:26.746895 [11383]: SIGCHLD from 12228
2012/10/08 20:26:26.747110 [eventscript-00.ctdb-recovered:12232]: Executing event script /etc/ctdb/events.d/00.ctdb recovered
2012/10/08 20:26:26.755895 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.756266 [11383]: SIGCHLD from 12232
2012/10/08 20:26:26.756494 [eventscript-01.reclock-recovered:12237]: Executing event script /etc/ctdb/events.d/01.reclock recovered
2012/10/08 20:26:26.763221 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.763591 [11383]: SIGCHLD from 12237
2012/10/08 20:26:26.763816 [eventscript-10.interface-recovered:12241]: Executing event script /etc/ctdb/events.d/10.interface recovered
2012/10/08 20:26:26.771066 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.771426 [11383]: SIGCHLD from 12241
2012/10/08 20:26:26.771650 [eventscript-11.natgw-recovered:12245]: Executing event script /etc/ctdb/events.d/11.natgw recovered
2012/10/08 20:26:26.778204 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.778598 [11383]: SIGCHLD from 12245
2012/10/08 20:26:26.778828 [eventscript-11.routing-recovered:12249]: Executing event script /etc/ctdb/events.d/11.routing recovered
2012/10/08 20:26:26.785443 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.785806 [11383]: SIGCHLD from 12249
2012/10/08 20:26:26.786024 [eventscript-13.per_ip_routing-recovered:12253]: Executing event script /etc/ctdb/events.d/13.per_ip_routing recovered
2012/10/08 20:26:26.794738 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.795101 [11383]: SIGCHLD from 12253
2012/10/08 20:26:26.795330 [eventscript-20.multipathd-recovered:12258]: Executing event script /etc/ctdb/events.d/20.multipathd recovered
2012/10/08 20:26:26.795767 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.796315 [eventscript-31.clamd-recovered:12259]: Executing event script /etc/ctdb/events.d/31.clamd recovered
2012/10/08 20:26:26.796734 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.796965 [11383]: SIGCHLD from 12258
2012/10/08 20:26:26.797181 [eventscript-40.vsftpd-recovered:12260]: Executing event script /etc/ctdb/events.d/40.vsftpd recovered
2012/10/08 20:26:26.801354 [11383]: SIGCHLD from 12259
2012/10/08 20:26:26.804235 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.804605 [11383]: SIGCHLD from 12260
2012/10/08 20:26:26.804834 [eventscript-40.fs_use-recovered:12264]: Executing event script /etc/ctdb/events.d/40.fs_use recovered
2012/10/08 20:26:26.805249 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.805792 [eventscript-41.httpd-recovered:12265]: Executing event script /etc/ctdb/events.d/41.httpd recovered
2012/10/08 20:26:26.809745 [11383]: SIGCHLD from 12264
2012/10/08 20:26:26.812550 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.812917 [11383]: SIGCHLD from 12265
2012/10/08 20:26:26.813148 [eventscript-50.samba-recovered:12269]: Executing event script /etc/ctdb/events.d/50.samba recovered
2012/10/08 20:26:26.822052 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.822444 [11383]: SIGCHLD from 12269
2012/10/08 20:26:26.822664 [eventscript-60.nfs-recovered:12274]: Executing event script /etc/ctdb/events.d/60.nfs recovered
2012/10/08 20:26:26.833379 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.833748 [11383]: SIGCHLD from 12274
2012/10/08 20:26:26.833980 [eventscript-60.ganesha-recovered:12280]: Executing event script /etc/ctdb/events.d/60.ganesha recovered
2012/10/08 20:26:26.834452 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.835000 [eventscript-62.cnfs-recovered:12281]: Executing event script /etc/ctdb/events.d/62.cnfs recovered
2012/10/08 20:26:26.839018 [11383]: SIGCHLD from 12280
2012/10/08 20:26:26.843888 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.844250 [11383]: SIGCHLD from 12281
2012/10/08 20:26:26.844468 [eventscript-70.iscsi-recovered:12286]: Executing event script /etc/ctdb/events.d/70.iscsi recovered
2012/10/08 20:26:26.851141 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:26.851507 [11383]: SIGCHLD from 12286
2012/10/08 20:26:26.851738 [eventscript-91.lvs-recovered:12290]: Executing event script /etc/ctdb/events.d/91.lvs recovered
2012/10/08 20:26:26.858023 [11383]: server/eventscript.c:485 Eventscript recovered finished with state 0
2012/10/08 20:26:26.858069 [11383]: Monitoring has been enabled
2012/10/08 20:26:26.858082 [11383]: Added timed event "queue_next_trigger": 0x25ae290
2012/10/08 20:26:26.858115 [11383]: Destroying timer event 0x25b1f40 "ctdb_event_script_timeout"
2012/10/08 20:26:26.858125 [11383]: server/ctdb_server.c:331 ctdb request 38 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:26.858163 [11383]: Destroying timer event 0x25adff0 "ctdb_control_timeout"
2012/10/08 20:26:26.858172 [11383]: Ending timer event 0x25ae290 "queue_next_trigger"
2012/10/08 20:26:26.858201 [recoverd:11454]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:26:26.858251 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 48 from node 1 to 4026531844
2012/10/08 20:26:26.858263 [11383]: Added timed event "queue_next_trigger": 0x25ade00
2012/10/08 20:26:26.858272 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 5 length 48 from node 1 to 1
2012/10/08 20:26:26.858280 [11383]: Ending timer event 0x25ade00 "queue_next_trigger"
2012/10/08 20:26:26.858294 [recoverd:11454]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:26:26.858302 [recoverd:11454]: Resetting ban count to 0 for all nodes
2012/10/08 20:26:26.858309 [recoverd:11454]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:26:26.858447 [11383]: SIGCHLD from 12290
2012/10/08 20:26:26.960565 [11383]: Added timed event "ctdb_statistics_update": 0x25adff0
2012/10/08 20:26:26.960581 [11383]: Ending timer event 0x25beb90 "ctdb_statistics_update"
2012/10/08 20:26:27.096636 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25ade00
2012/10/08 20:26:27.096650 [11383]: Ending timer event 0x25ada80 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:27.117725 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:27.117739 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:27.117759 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b1fb0
2012/10/08 20:26:27.117768 [11383]: Ending timer event 0x25bea70 "ctdb_wait_until_recovered"
2012/10/08 20:26:27.117779 [11383]: Added timed event "ctdb_time_tick": 0x25ad830
2012/10/08 20:26:27.117787 [11383]: Ending timer event 0x25bd3c0 "ctdb_time_tick"
2012/10/08 20:26:27.125850 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:27.125903 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b63a0
2012/10/08 20:26:27.125916 [11383]: Ending timer event 0x25c4e40 "ctdb_tcp_node_connect"
2012/10/08 20:26:27.126319 [11383]: Destroying timer event 0x25b63a0 "ctdb_tcp_node_connect"
2012/10/08 20:26:27.126873 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b62f0
2012/10/08 20:26:27.263890 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:27.264020 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:27.264039 [11383]: Added timed event "queue_next_trigger": 0x25b7010
2012/10/08 20:26:27.264050 [11383]: Added timed event "ctdb_control_timeout": 0x25b5b90
2012/10/08 20:26:27.264060 [11383]: server/ctdb_server.c:331 ctdb request 40 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.264072 [11383]: Added timed event "queue_next_trigger": 0x25b57e0
2012/10/08 20:26:27.264082 [11383]: Ending timer event 0x25b7010 "queue_next_trigger"
2012/10/08 20:26:27.264091 [11383]: server/ctdb_server.c:331 ctdb request 40 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:27.264105 [11383]: Destroying timer event 0x25b5b90 "ctdb_control_timeout"
2012/10/08 20:26:27.264115 [11383]: Ending timer event 0x25b57e0 "queue_next_trigger"
2012/10/08 20:26:27.264176 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:27.264233 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:27.264248 [11383]: Added timed event "queue_next_trigger": 0x25b7040
2012/10/08 20:26:27.264258 [11383]: Added timed event "ctdb_control_timeout": 0x25b53c0
2012/10/08 20:26:27.264267 [11383]: server/ctdb_server.c:331 ctdb request 42 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.264278 [11383]: Added timed event "queue_next_trigger": 0x25b5570
2012/10/08 20:26:27.264288 [11383]: Ending timer event 0x25b7040 "queue_next_trigger"
2012/10/08 20:26:27.264297 [11383]: server/ctdb_server.c:331 ctdb request 42 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:27.264310 [11383]: Destroying timer event 0x25b53c0 "ctdb_control_timeout"
2012/10/08 20:26:27.264319 [11383]: Ending timer event 0x25b5570 "queue_next_trigger"
2012/10/08 20:26:27.264367 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.264382 [11383]: Added timed event "queue_next_trigger": 0x25b6f70
2012/10/08 20:26:27.264392 [11383]: Added timed event "ctdb_control_timeout": 0x25b4eb0
2012/10/08 20:26:27.264402 [11383]: server/ctdb_server.c:331 ctdb request 43 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.264412 [11383]: Added timed event "queue_next_trigger": 0x25b4fd0
2012/10/08 20:26:27.264422 [11383]: Ending timer event 0x25b6f70 "queue_next_trigger"
2012/10/08 20:26:27.264431 [11383]: server/ctdb_server.c:331 ctdb request 43 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:27.264443 [11383]: Destroying timer event 0x25b4eb0 "ctdb_control_timeout"
2012/10/08 20:26:27.264453 [11383]: Ending timer event 0x25b4fd0 "queue_next_trigger"
2012/10/08 20:26:27.264504 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.264518 [11383]: Added timed event "queue_next_trigger": 0x25b53c0
2012/10/08 20:26:27.264528 [11383]: Added timed event "ctdb_control_timeout": 0x25b5550
2012/10/08 20:26:27.264538 [11383]: server/ctdb_server.c:331 ctdb request 44 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.264550 [11383]: Added timed event "queue_next_trigger": 0x25c7a00
2012/10/08 20:26:27.264567 [11383]: Ending timer event 0x25b53c0 "queue_next_trigger"
2012/10/08 20:26:27.264586 [11383]: server/ctdb_server.c:331 ctdb request 44 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:27.264600 [11383]: Destroying timer event 0x25b5550 "ctdb_control_timeout"
2012/10/08 20:26:27.264610 [11383]: Ending timer event 0x25c7a00 "queue_next_trigger"
2012/10/08 20:26:27.265017 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.265034 [11383]: Added timed event "queue_next_trigger": 0x25b51d0
2012/10/08 20:26:27.265044 [11383]: Added timed event "ctdb_control_timeout": 0x25c78d0
2012/10/08 20:26:27.265054 [11383]: server/ctdb_server.c:331 ctdb request 45 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.265066 [11383]: Added timed event "queue_next_trigger": 0x25c7c90
2012/10/08 20:26:27.265075 [11383]: Ending timer event 0x25b51d0 "queue_next_trigger"
2012/10/08 20:26:27.265085 [11383]: server/ctdb_server.c:331 ctdb request 45 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:27.265098 [11383]: Destroying timer event 0x25c78d0 "ctdb_control_timeout"
2012/10/08 20:26:27.265108 [11383]: Ending timer event 0x25c7c90 "queue_next_trigger"
2012/10/08 20:26:27.265379 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:27.265395 [11383]: Added timed event "queue_next_trigger": 0x25b5550
2012/10/08 20:26:27.265405 [11383]: Added timed event "ctdb_control_timeout": 0x25c7c00
2012/10/08 20:26:27.265415 [11383]: server/ctdb_server.c:331 ctdb request 46 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.265426 [11383]: Added timed event "queue_next_trigger": 0x25c7f40
2012/10/08 20:26:27.265436 [11383]: Ending timer event 0x25b5550 "queue_next_trigger"
2012/10/08 20:26:27.265445 [11383]: server/ctdb_server.c:331 ctdb request 46 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:27.265458 [11383]: Destroying timer event 0x25c7c00 "ctdb_control_timeout"
2012/10/08 20:26:27.265468 [11383]: Ending timer event 0x25c7f40 "queue_next_trigger"
2012/10/08 20:26:27.265500 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:27.265514 [11383]: Added timed event "queue_next_trigger": 0x25c7e20
2012/10/08 20:26:27.265524 [11383]: Added timed event "ctdb_control_timeout": 0x25c7f40
2012/10/08 20:26:27.265534 [11383]: server/ctdb_server.c:331 ctdb request 47 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.265544 [11383]: Added timed event "queue_next_trigger": 0x25b2280
2012/10/08 20:26:27.265554 [11383]: Ending timer event 0x25c7e20 "queue_next_trigger"
2012/10/08 20:26:27.265563 [11383]: server/ctdb_server.c:331 ctdb request 47 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:27.265575 [11383]: Destroying timer event 0x25c7f40 "ctdb_control_timeout"
2012/10/08 20:26:27.265585 [11383]: Ending timer event 0x25b2280 "queue_next_trigger"
2012/10/08 20:26:27.265694 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:27.265709 [11383]: Added timed event "queue_next_trigger": 0x25b2160
2012/10/08 20:26:27.265719 [11383]: Added timed event "ctdb_control_timeout": 0x25b2280
2012/10/08 20:26:27.265729 [11383]: server/ctdb_server.c:331 ctdb request 48 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:27.265740 [11383]: Added timed event "queue_next_trigger": 0x25b25a0
2012/10/08 20:26:27.265749 [11383]: Ending timer event 0x25b2160 "queue_next_trigger"
2012/10/08 20:26:27.265759 [11383]: server/ctdb_server.c:331 ctdb request 48 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:27.265771 [11383]: Destroying timer event 0x25b2280 "ctdb_control_timeout"
2012/10/08 20:26:27.265781 [11383]: Ending timer event 0x25b25a0 "queue_next_trigger"
2012/10/08 20:26:27.961104 [11383]: Added timed event "ctdb_statistics_update": 0x25b2160
2012/10/08 20:26:27.961122 [11383]: Ending timer event 0x25adff0 "ctdb_statistics_update"
2012/10/08 20:26:28.118175 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:28.118201 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:28.118213 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25c7c00
2012/10/08 20:26:28.118222 [11383]: Ending timer event 0x25b1fb0 "ctdb_wait_until_recovered"
2012/10/08 20:26:28.118231 [11383]: Added timed event "ctdb_time_tick": 0x25b2280
2012/10/08 20:26:28.118240 [11383]: Ending timer event 0x25ad830 "ctdb_time_tick"
2012/10/08 20:26:28.127299 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:28.127334 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25adff0
2012/10/08 20:26:28.127346 [11383]: Ending timer event 0x25b62f0 "ctdb_tcp_node_connect"
2012/10/08 20:26:28.127620 [11383]: Destroying timer event 0x25adff0 "ctdb_tcp_node_connect"
2012/10/08 20:26:28.127643 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25adff0
2012/10/08 20:26:28.961879 [11383]: Added timed event "ctdb_statistics_update": 0x25b5550
2012/10/08 20:26:28.961895 [11383]: Ending timer event 0x25b2160 "ctdb_statistics_update"
2012/10/08 20:26:29.118946 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:29.118961 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:29.118972 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25aef40
2012/10/08 20:26:29.118980 [11383]: Ending timer event 0x25c7c00 "ctdb_wait_until_recovered"
2012/10/08 20:26:29.118990 [11383]: Added timed event "ctdb_time_tick": 0x25c7c00
2012/10/08 20:26:29.118999 [11383]: Ending timer event 0x25b2280 "ctdb_time_tick"
2012/10/08 20:26:29.128056 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:29.128091 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b59e0
2012/10/08 20:26:29.128102 [11383]: Ending timer event 0x25adff0 "ctdb_tcp_node_connect"
2012/10/08 20:26:29.128456 [11383]: Destroying timer event 0x25b59e0 "ctdb_tcp_node_connect"
2012/10/08 20:26:29.128477 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b59e0
2012/10/08 20:26:29.962562 [11383]: Added timed event "ctdb_statistics_update": 0x25adaf0
2012/10/08 20:26:29.962617 [11383]: Ending timer event 0x25b5550 "ctdb_statistics_update"
2012/10/08 20:26:30.119671 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:30.119686 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:30.119699 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b2730
2012/10/08 20:26:30.119708 [11383]: Ending timer event 0x25aef40 "ctdb_wait_until_recovered"
2012/10/08 20:26:30.119718 [11383]: Added timed event "ctdb_time_tick": 0x25aef40
2012/10/08 20:26:30.119726 [11383]: Ending timer event 0x25c7c00 "ctdb_time_tick"
2012/10/08 20:26:30.128799 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:30.128860 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b2850
2012/10/08 20:26:30.128872 [11383]: Ending timer event 0x25b59e0 "ctdb_tcp_node_connect"
2012/10/08 20:26:30.129200 [11383]: Destroying timer event 0x25b2850 "ctdb_tcp_node_connect"
2012/10/08 20:26:30.129232 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b59e0
2012/10/08 20:26:30.963284 [11383]: Added timed event "ctdb_statistics_update": 0x25b29e0
2012/10/08 20:26:30.963311 [11383]: Ending timer event 0x25adaf0 "ctdb_statistics_update"
2012/10/08 20:26:31.120360 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:31.120377 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:31.120389 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b47f0
2012/10/08 20:26:31.120398 [11383]: Ending timer event 0x25b2730 "ctdb_wait_until_recovered"
2012/10/08 20:26:31.120408 [11383]: Added timed event "ctdb_time_tick": 0x25b2730
2012/10/08 20:26:31.120416 [11383]: Ending timer event 0x25aef40 "ctdb_time_tick"
2012/10/08 20:26:31.129477 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:31.129520 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b4910
2012/10/08 20:26:31.129585 [11383]: Ending timer event 0x25b59e0 "ctdb_tcp_node_connect"
2012/10/08 20:26:31.129780 [11383]: Destroying timer event 0x25b4910 "ctdb_tcp_node_connect"
2012/10/08 20:26:31.129805 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b4910
2012/10/08 20:26:31.963862 [11383]: Added timed event "ctdb_statistics_update": 0x25b4aa0
2012/10/08 20:26:31.963885 [11383]: Ending timer event 0x25b29e0 "ctdb_statistics_update"
2012/10/08 20:26:32.093945 [11383]: Added timed event "ctdb_update_tcp_tickles": 0x25b29e0
2012/10/08 20:26:32.093963 [11383]: Ending timer event 0x25b6480 "ctdb_update_tcp_tickles"
2012/10/08 20:26:32.097043 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25b6480
2012/10/08 20:26:32.097058 [11383]: Ending timer event 0x25ade00 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:32.121107 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:32.121122 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:32.121134 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b7220
2012/10/08 20:26:32.121143 [11383]: Ending timer event 0x25b47f0 "ctdb_wait_until_recovered"
2012/10/08 20:26:32.121152 [11383]: Added timed event "ctdb_time_tick": 0x25b47f0
2012/10/08 20:26:32.121161 [11383]: Ending timer event 0x25b2730 "ctdb_time_tick"
2012/10/08 20:26:32.130219 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:32.130258 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b7340
2012/10/08 20:26:32.130270 [11383]: Ending timer event 0x25b4910 "ctdb_tcp_node_connect"
2012/10/08 20:26:32.130621 [11383]: Destroying timer event 0x25b7340 "ctdb_tcp_node_connect"
2012/10/08 20:26:32.130760 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b7340
2012/10/08 20:26:32.964819 [11383]: Added timed event "ctdb_statistics_update": 0x25b74d0
2012/10/08 20:26:32.964841 [11383]: Ending timer event 0x25b4aa0 "ctdb_statistics_update"
2012/10/08 20:26:33.121893 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:33.121909 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:33.121920 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b76d0
2012/10/08 20:26:33.121929 [11383]: Ending timer event 0x25b7220 "ctdb_wait_until_recovered"
2012/10/08 20:26:33.121939 [11383]: Added timed event "ctdb_time_tick": 0x25b7220
2012/10/08 20:26:33.121947 [11383]: Ending timer event 0x25b47f0 "ctdb_time_tick"
2012/10/08 20:26:33.131006 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:33.131044 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b77f0
2012/10/08 20:26:33.131056 [11383]: Ending timer event 0x25b7340 "ctdb_tcp_node_connect"
2012/10/08 20:26:33.131309 [11383]: Destroying timer event 0x25b77f0 "ctdb_tcp_node_connect"
2012/10/08 20:26:33.131333 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b77f0
2012/10/08 20:26:33.965580 [11383]: Added timed event "ctdb_statistics_update": 0x25b7980
2012/10/08 20:26:33.965603 [11383]: Ending timer event 0x25b74d0 "ctdb_statistics_update"
2012/10/08 20:26:34.122656 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:34.122672 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:34.122684 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b7b80
2012/10/08 20:26:34.122693 [11383]: Ending timer event 0x25b76d0 "ctdb_wait_until_recovered"
2012/10/08 20:26:34.122702 [11383]: Added timed event "ctdb_time_tick": 0x25b76d0
2012/10/08 20:26:34.122711 [11383]: Ending timer event 0x25b7220 "ctdb_time_tick"
2012/10/08 20:26:34.131772 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:34.131810 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b7ca0
2012/10/08 20:26:34.131859 [11383]: Ending timer event 0x25b77f0 "ctdb_tcp_node_connect"
2012/10/08 20:26:34.132084 [11383]: Destroying timer event 0x25b7ca0 "ctdb_tcp_node_connect"
2012/10/08 20:26:34.132107 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b7ca0
2012/10/08 20:26:34.966380 [11383]: Added timed event "ctdb_statistics_update": 0x25b7e30
2012/10/08 20:26:34.966407 [11383]: Ending timer event 0x25b7980 "ctdb_statistics_update"
2012/10/08 20:26:35.123459 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:35.123477 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:35.123522 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b8030
2012/10/08 20:26:35.123532 [11383]: Ending timer event 0x25b7b80 "ctdb_wait_until_recovered"
2012/10/08 20:26:35.123542 [11383]: Added timed event "ctdb_time_tick": 0x25b7b80
2012/10/08 20:26:35.123551 [11383]: Ending timer event 0x25b76d0 "ctdb_time_tick"
2012/10/08 20:26:35.132616 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:35.132658 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b8150
2012/10/08 20:26:35.132670 [11383]: Ending timer event 0x25b7ca0 "ctdb_tcp_node_connect"
2012/10/08 20:26:35.132944 [11383]: Destroying timer event 0x25b8150 "ctdb_tcp_node_connect"
2012/10/08 20:26:35.132970 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b8150
2012/10/08 20:26:35.967027 [11383]: Added timed event "ctdb_statistics_update": 0x25b82e0
2012/10/08 20:26:35.967049 [11383]: Ending timer event 0x25b7e30 "ctdb_statistics_update"
2012/10/08 20:26:36.124103 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:36.124120 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:36.124132 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b84e0
2012/10/08 20:26:36.124141 [11383]: Ending timer event 0x25b8030 "ctdb_wait_until_recovered"
2012/10/08 20:26:36.124150 [11383]: Added timed event "ctdb_time_tick": 0x25b8030
2012/10/08 20:26:36.124159 [11383]: Ending timer event 0x25b7b80 "ctdb_time_tick"
2012/10/08 20:26:36.133219 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:36.133257 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b8600
2012/10/08 20:26:36.133269 [11383]: Ending timer event 0x25b8150 "ctdb_tcp_node_connect"
2012/10/08 20:26:36.133549 [11383]: Destroying timer event 0x25b8600 "ctdb_tcp_node_connect"
2012/10/08 20:26:36.133572 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b8600
2012/10/08 20:26:36.868359 [recoverd:11454]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:26:36.868428 [11383]: server/ctdb_daemon.c:628 client request 2147483640 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.868449 [11383]: Added timed event "queue_next_trigger": 0x25b8be0
2012/10/08 20:26:36.868459 [11383]: Added timed event "ctdb_control_timeout": 0x25b8d00
2012/10/08 20:26:36.868470 [11383]: server/ctdb_server.c:331 ctdb request 49 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.868481 [11383]: Destroying timer event 0x25ae1e0 "ctdb_recd_ping_timeout"
2012/10/08 20:26:36.868491 [11383]: Added timed event "ctdb_recd_ping_timeout": 0x25ae1e0
2012/10/08 20:26:36.868502 [11383]: Added timed event "queue_next_trigger": 0x25b9020
2012/10/08 20:26:36.868511 [11383]: Ending timer event 0x25b8be0 "queue_next_trigger"
2012/10/08 20:26:36.868522 [11383]: server/ctdb_server.c:331 ctdb request 49 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.868536 [11383]: Destroying timer event 0x25b8d00 "ctdb_control_timeout"
2012/10/08 20:26:36.868546 [11383]: Ending timer event 0x25b9020 "queue_next_trigger"
2012/10/08 20:26:36.868588 [11383]: server/ctdb_daemon.c:628 client request 2147483641 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.868603 [11383]: Added timed event "queue_next_trigger": 0x25b8940
2012/10/08 20:26:36.868612 [11383]: Added timed event "ctdb_control_timeout": 0x25b91b0
2012/10/08 20:26:36.868622 [11383]: server/ctdb_server.c:331 ctdb request 50 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.868634 [11383]: Added timed event "queue_next_trigger": 0x25b93f0
2012/10/08 20:26:36.868643 [11383]: Ending timer event 0x25b8940 "queue_next_trigger"
2012/10/08 20:26:36.868666 [11383]: server/ctdb_server.c:331 ctdb request 50 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.868679 [11383]: Destroying timer event 0x25b91b0 "ctdb_control_timeout"
2012/10/08 20:26:36.868689 [11383]: Ending timer event 0x25b93f0 "queue_next_trigger"
2012/10/08 20:26:36.868729 [11383]: server/ctdb_daemon.c:628 client request 2147483642 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.868744 [11383]: Added timed event "queue_next_trigger": 0x25b8940
2012/10/08 20:26:36.868753 [11383]: Added timed event "ctdb_control_timeout": 0x25b93b0
2012/10/08 20:26:36.868763 [11383]: server/ctdb_server.c:331 ctdb request 51 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.868776 [11383]: Added timed event "queue_next_trigger": 0x25bc610
2012/10/08 20:26:36.868786 [11383]: Ending timer event 0x25b8940 "queue_next_trigger"
2012/10/08 20:26:36.868795 [11383]: server/ctdb_server.c:331 ctdb request 51 of type 8 length 228 from node 1 to 1
2012/10/08 20:26:36.868808 [11383]: Destroying timer event 0x25b93b0 "ctdb_control_timeout"
2012/10/08 20:26:36.868818 [11383]: Ending timer event 0x25bc610 "queue_next_trigger"
2012/10/08 20:26:36.868858 [11383]: server/ctdb_daemon.c:628 client request 2147483643 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.868872 [11383]: Added timed event "queue_next_trigger": 0x25b8f00
2012/10/08 20:26:36.868882 [11383]: Added timed event "ctdb_control_timeout": 0x25b9020
2012/10/08 20:26:36.868892 [11383]: server/ctdb_server.c:331 ctdb request 52 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.868903 [11383]: Added timed event "queue_next_trigger": 0x25bc7a0
2012/10/08 20:26:36.868912 [11383]: Ending timer event 0x25b8f00 "queue_next_trigger"
2012/10/08 20:26:36.868922 [11383]: server/ctdb_server.c:331 ctdb request 52 of type 8 length 67 from node 1 to 1
2012/10/08 20:26:36.868934 [11383]: Destroying timer event 0x25b9020 "ctdb_control_timeout"
2012/10/08 20:26:36.868944 [11383]: Ending timer event 0x25bc7a0 "queue_next_trigger"
2012/10/08 20:26:36.868985 [11383]: server/ctdb_daemon.c:628 client request 2147483644 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.869000 [11383]: Added timed event "queue_next_trigger": 0x25b93b0
2012/10/08 20:26:36.869010 [11383]: Added timed event "ctdb_control_timeout": 0x25b9660
2012/10/08 20:26:36.869020 [11383]: server/ctdb_server.c:331 ctdb request 53 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869030 [11383]: Added timed event "queue_next_trigger": 0x25bcac0
2012/10/08 20:26:36.869040 [11383]: Ending timer event 0x25b93b0 "queue_next_trigger"
2012/10/08 20:26:36.869049 [11383]: server/ctdb_server.c:331 ctdb request 53 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.869062 [11383]: Destroying timer event 0x25b9660 "ctdb_control_timeout"
2012/10/08 20:26:36.869072 [11383]: Ending timer event 0x25bcac0 "queue_next_trigger"
2012/10/08 20:26:36.869111 [11383]: server/ctdb_daemon.c:628 client request 2147483645 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869125 [11383]: Added timed event "queue_next_trigger": 0x25bc9a0
2012/10/08 20:26:36.869135 [11383]: Added timed event "ctdb_control_timeout": 0x25bcac0
2012/10/08 20:26:36.869144 [11383]: server/ctdb_server.c:331 ctdb request 54 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869156 [11383]: Added timed event "queue_next_trigger": 0x25bcde0
2012/10/08 20:26:36.869165 [11383]: Ending timer event 0x25bc9a0 "queue_next_trigger"
2012/10/08 20:26:36.869174 [11383]: server/ctdb_server.c:331 ctdb request 54 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:36.869187 [11383]: Destroying timer event 0x25bcac0 "ctdb_control_timeout"
2012/10/08 20:26:36.869197 [11383]: Ending timer event 0x25bcde0 "queue_next_trigger"
2012/10/08 20:26:36.869237 [11383]: server/ctdb_daemon.c:628 client request 2147483646 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869251 [11383]: Added timed event "queue_next_trigger": 0x25b93b0
2012/10/08 20:26:36.869268 [11383]: Added timed event "ctdb_control_timeout": 0x25bcd30
2012/10/08 20:26:36.869278 [11383]: server/ctdb_server.c:331 ctdb request 55 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869291 [11383]: Added timed event "queue_next_trigger": 0x25bd1c0
2012/10/08 20:26:36.869301 [11383]: Ending timer event 0x25b93b0 "queue_next_trigger"
2012/10/08 20:26:36.869310 [11383]: server/ctdb_server.c:331 ctdb request 55 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:36.869323 [11383]: Destroying timer event 0x25bcd30 "ctdb_control_timeout"
2012/10/08 20:26:36.869333 [11383]: Ending timer event 0x25bd1c0 "queue_next_trigger"
2012/10/08 20:26:36.869373 [11383]: server/ctdb_daemon.c:628 client request 2147483647 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869387 [11383]: Added timed event "queue_next_trigger": 0x25bcf70
2012/10/08 20:26:36.869397 [11383]: Added timed event "ctdb_control_timeout": 0x25bd090
2012/10/08 20:26:36.869407 [11383]: server/ctdb_server.c:331 ctdb request 56 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869418 [11383]: Added timed event "queue_next_trigger": 0x25bd3e0
2012/10/08 20:26:36.869427 [11383]: Ending timer event 0x25bcf70 "queue_next_trigger"
2012/10/08 20:26:36.869436 [11383]: server/ctdb_server.c:331 ctdb request 56 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.869449 [11383]: Destroying timer event 0x25bd090 "ctdb_control_timeout"
2012/10/08 20:26:36.869459 [11383]: Ending timer event 0x25bd3e0 "queue_next_trigger"
2012/10/08 20:26:36.869500 [recoverd:11454]: Reqid wrap!
2012/10/08 20:26:36.869536 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869550 [11383]: Added timed event "queue_next_trigger": 0x25b93b0
2012/10/08 20:26:36.869560 [11383]: Added timed event "ctdb_control_timeout": 0x25bd3c0
2012/10/08 20:26:36.869569 [11383]: server/ctdb_server.c:331 ctdb request 57 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869581 [11383]: Added timed event "queue_next_trigger": 0x25bd830
2012/10/08 20:26:36.869590 [11383]: Ending timer event 0x25b93b0 "queue_next_trigger"
2012/10/08 20:26:36.869600 [11383]: server/ctdb_server.c:331 ctdb request 57 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:36.869612 [11383]: Destroying timer event 0x25bd3c0 "ctdb_control_timeout"
2012/10/08 20:26:36.869622 [11383]: Ending timer event 0x25bd830 "queue_next_trigger"
2012/10/08 20:26:36.869662 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.869676 [11383]: Added timed event "queue_next_trigger": 0x25bd5e0
2012/10/08 20:26:36.869686 [11383]: Added timed event "ctdb_control_timeout": 0x25bd700
2012/10/08 20:26:36.869695 [11383]: server/ctdb_server.c:331 ctdb request 58 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869707 [11383]: Added timed event "queue_next_trigger": 0x25bdb60
2012/10/08 20:26:36.869717 [11383]: Ending timer event 0x25bd5e0 "queue_next_trigger"
2012/10/08 20:26:36.869726 [11383]: server/ctdb_server.c:331 ctdb request 58 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:36.869739 [11383]: Destroying timer event 0x25bd700 "ctdb_control_timeout"
2012/10/08 20:26:36.869749 [11383]: Ending timer event 0x25bdb60 "queue_next_trigger"
2012/10/08 20:26:36.869789 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.869803 [11383]: Added timed event "queue_next_trigger": 0x25b93b0
2012/10/08 20:26:36.869812 [11383]: Added timed event "ctdb_control_timeout": 0x25bda30
2012/10/08 20:26:36.869822 [11383]: server/ctdb_server.c:331 ctdb request 59 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869834 [11383]: Added timed event "queue_next_trigger": 0x25bdd90
2012/10/08 20:26:36.869843 [11383]: Ending timer event 0x25b93b0 "queue_next_trigger"
2012/10/08 20:26:36.869852 [11383]: server/ctdb_server.c:331 ctdb request 59 of type 8 length 72 from node 1 to 1
2012/10/08 20:26:36.869865 [11383]: Destroying timer event 0x25bda30 "ctdb_control_timeout"
2012/10/08 20:26:36.869881 [11383]: Ending timer event 0x25bdd90 "queue_next_trigger"
2012/10/08 20:26:36.869923 [11383]: server/ctdb_daemon.c:628 client request 3 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.869937 [11383]: Added timed event "queue_next_trigger": 0x25b93b0
2012/10/08 20:26:36.869947 [11383]: Added timed event "ctdb_control_timeout": 0x25bdd60
2012/10/08 20:26:36.869956 [11383]: server/ctdb_server.c:331 ctdb request 60 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.869968 [11383]: Added timed event "queue_next_trigger": 0x25be0f0
2012/10/08 20:26:36.869978 [11383]: Ending timer event 0x25b93b0 "queue_next_trigger"
2012/10/08 20:26:36.869987 [11383]: server/ctdb_server.c:331 ctdb request 60 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:36.870000 [11383]: Destroying timer event 0x25bdd60 "ctdb_control_timeout"
2012/10/08 20:26:36.870009 [11383]: Ending timer event 0x25be0f0 "queue_next_trigger"
2012/10/08 20:26:36.870049 [11383]: server/ctdb_daemon.c:628 client request 4 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:36.870063 [11383]: Added timed event "queue_next_trigger": 0x25bdf90
2012/10/08 20:26:36.870073 [11383]: Added timed event "ctdb_control_timeout": 0x25be0b0
2012/10/08 20:26:36.870083 [11383]: server/ctdb_server.c:331 ctdb request 61 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.870094 [11383]: Added timed event "queue_next_trigger": 0x25be530
2012/10/08 20:26:36.870103 [11383]: Ending timer event 0x25bdf90 "queue_next_trigger"
2012/10/08 20:26:36.870112 [11383]: server/ctdb_server.c:331 ctdb request 61 of type 8 length 108 from node 1 to 1
2012/10/08 20:26:36.870125 [11383]: Destroying timer event 0x25be0b0 "ctdb_control_timeout"
2012/10/08 20:26:36.870135 [11383]: Ending timer event 0x25be530 "queue_next_trigger"
2012/10/08 20:26:36.870189 [11383]: server/ctdb_daemon.c:628 client request 5 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.870204 [11383]: Added timed event "queue_next_trigger": 0x25be2f0
2012/10/08 20:26:36.870213 [11383]: Added timed event "ctdb_control_timeout": 0x25be410
2012/10/08 20:26:36.870248 [11383]: server/ctdb_server.c:331 ctdb request 62 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.870263 [11383]: Added timed event "queue_next_trigger": 0x25be6e0
2012/10/08 20:26:36.870273 [11383]: Ending timer event 0x25be2f0 "queue_next_trigger"
2012/10/08 20:26:36.870282 [11383]: server/ctdb_server.c:331 ctdb request 62 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.870295 [11383]: Destroying timer event 0x25be410 "ctdb_control_timeout"
2012/10/08 20:26:36.870305 [11383]: Ending timer event 0x25be6e0 "queue_next_trigger"
2012/10/08 20:26:36.870346 [11383]: server/ctdb_daemon.c:628 client request 6 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.870360 [11383]: Added timed event "queue_next_trigger": 0x25be0b0
2012/10/08 20:26:36.870370 [11383]: Added timed event "ctdb_control_timeout": 0x25be6c0
2012/10/08 20:26:36.870380 [11383]: server/ctdb_server.c:331 ctdb request 63 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.870391 [11383]: Added timed event "queue_next_trigger": 0x25bea00
2012/10/08 20:26:36.870401 [11383]: Ending timer event 0x25be0b0 "queue_next_trigger"
2012/10/08 20:26:36.870410 [11383]: server/ctdb_server.c:331 ctdb request 63 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.870423 [11383]: Destroying timer event 0x25be6c0 "ctdb_control_timeout"
2012/10/08 20:26:36.870433 [11383]: Ending timer event 0x25bea00 "queue_next_trigger"
2012/10/08 20:26:36.870831 [recoverd:11454]: server/ctdb_recoverd.c:2810 Created PIPE FD:17 for check_recovery_lock
2012/10/08 20:26:36.873181 [recoverd:11454]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:26:36.873197 [recoverd:11454]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:26:36.874393 [11383]: server/ctdb_daemon.c:628 client request 7 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:36.874411 [11383]: Added timed event "queue_next_trigger": 0x25be8e0
2012/10/08 20:26:36.874429 [11383]: Added timed event "ctdb_control_timeout": 0x25bea00
2012/10/08 20:26:36.874441 [11383]: server/ctdb_server.c:331 ctdb request 64 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:36.874452 [11383]: Added timed event "queue_next_trigger": 0x25b9a40
2012/10/08 20:26:36.874462 [11383]: Ending timer event 0x25be8e0 "queue_next_trigger"
2012/10/08 20:26:36.874471 [11383]: server/ctdb_server.c:331 ctdb request 64 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.874485 [11383]: Destroying timer event 0x25bea00 "ctdb_control_timeout"
2012/10/08 20:26:36.874495 [11383]: Ending timer event 0x25b9a40 "queue_next_trigger"
2012/10/08 20:26:36.874579 [recoverd:11454]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:26:36.874591 [recoverd:11454]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:26:36.874600 [recoverd:11454]: Taking out recovery lock from recovery daemon
2012/10/08 20:26:36.874609 [recoverd:11454]: Take the recovery lock
2012/10/08 20:26:36.880001 [recoverd:11454]: Recovery lock taken successfully
2012/10/08 20:26:36.880017 [recoverd:11454]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:26:36.880046 [11383]: server/ctdb_daemon.c:628 client request 8 of type 7 length 72 from node 1 to 4026531841
2012/10/08 20:26:36.880061 [11383]: Added timed event "queue_next_trigger": 0x25be0b0
2012/10/08 20:26:36.880071 [11383]: Added timed event "ctdb_control_timeout": 0x25b9a00
2012/10/08 20:26:36.880081 [11383]: server/ctdb_server.c:331 ctdb request 65 of type 7 length 68 from node 1 to 1
2012/10/08 20:26:36.880092 [11383]: Added timed event "queue_next_trigger": 0x25b9d60
2012/10/08 20:26:36.880102 [11383]: Ending timer event 0x25be0b0 "queue_next_trigger"
2012/10/08 20:26:36.880112 [11383]: server/ctdb_server.c:331 ctdb request 65 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.880125 [11383]: Destroying timer event 0x25b9a00 "ctdb_control_timeout"
2012/10/08 20:26:36.880135 [11383]: Ending timer event 0x25b9d60 "queue_next_trigger"
2012/10/08 20:26:36.880166 [recoverd:11454]: RECD SIGCHLD from 12295
2012/10/08 20:26:36.880185 [recoverd:11454]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:26:36.880195 [recoverd:11454]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 1
2012/10/08 20:26:36.880232 [11383]: server/ctdb_daemon.c:628 client request 9 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.880247 [11383]: Added timed event "queue_next_trigger": 0x25b9c40
2012/10/08 20:26:36.880256 [11383]: Added timed event "ctdb_control_timeout": 0x25b9d60
2012/10/08 20:26:36.880266 [11383]: server/ctdb_server.c:331 ctdb request 66 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.880292 [11383]: Added timed event "queue_next_trigger": 0x25ba080
2012/10/08 20:26:36.880303 [11383]: Ending timer event 0x25b9c40 "queue_next_trigger"
2012/10/08 20:26:36.880313 [11383]: server/ctdb_server.c:331 ctdb request 66 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.880326 [11383]: Destroying timer event 0x25b9d60 "ctdb_control_timeout"
2012/10/08 20:26:36.880336 [11383]: Ending timer event 0x25ba080 "queue_next_trigger"
2012/10/08 20:26:36.880364 [recoverd:11454]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:26:36.880376 [recoverd:11454]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:26:36.880402 [11383]: server/ctdb_daemon.c:628 client request 10 of type 7 length 88 from node 1 to 1
2012/10/08 20:26:36.880416 [11383]: Added timed event "queue_next_trigger": 0x25b9a00
2012/10/08 20:26:36.880427 [11383]: Added timed event "ctdb_control_timeout": 0x25ba040
2012/10/08 20:26:36.880437 [11383]: server/ctdb_server.c:331 ctdb request 67 of type 7 length 83 from node 1 to 1
2012/10/08 20:26:36.880448 [11383]: Added timed event "queue_next_trigger": 0x25ba3a0
2012/10/08 20:26:36.880458 [11383]: Ending timer event 0x25b9a00 "queue_next_trigger"
2012/10/08 20:26:36.880480 [11383]: server/ctdb_server.c:331 ctdb request 67 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.880495 [11383]: Destroying timer event 0x25ba040 "ctdb_control_timeout"
2012/10/08 20:26:36.880505 [11383]: Ending timer event 0x25ba3a0 "queue_next_trigger"
2012/10/08 20:26:36.880548 [11383]: server/ctdb_daemon.c:628 client request 11 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.880562 [11383]: Added timed event "queue_next_trigger": 0x25ba280
2012/10/08 20:26:36.880572 [11383]: Added timed event "ctdb_control_timeout": 0x25ba3a0
2012/10/08 20:26:36.880582 [11383]: server/ctdb_server.c:331 ctdb request 68 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.880591 [11383]: Freeze priority 1
2012/10/08 20:26:36.881099 [11383]: Ending timer event 0x25ba280 "queue_next_trigger"
2012/10/08 20:26:36.881461 [11383]: Added timed event "queue_next_trigger": 0x25ba750
2012/10/08 20:26:36.881493 [11383]: server/ctdb_server.c:331 ctdb request 68 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.881609 [11383]: Destroying timer event 0x25ba3a0 "ctdb_control_timeout"
2012/10/08 20:26:36.881621 [11383]: Ending timer event 0x25ba750 "queue_next_trigger"
2012/10/08 20:26:36.881664 [11383]: server/ctdb_daemon.c:628 client request 12 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.881679 [11383]: Added timed event "queue_next_trigger": 0x25ba040
2012/10/08 20:26:36.881689 [11383]: Added timed event "ctdb_control_timeout": 0x25ba8e0
2012/10/08 20:26:36.881699 [11383]: server/ctdb_server.c:331 ctdb request 69 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.881708 [11383]: Freeze priority 2
2012/10/08 20:26:36.882180 [11383]: Ending timer event 0x25ba040 "queue_next_trigger"
2012/10/08 20:26:36.882595 [11383]: Added timed event "queue_next_trigger": 0x25babb0
2012/10/08 20:26:36.882628 [11383]: server/ctdb_server.c:331 ctdb request 69 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.882724 [11383]: Destroying timer event 0x25ba8e0 "ctdb_control_timeout"
2012/10/08 20:26:36.882736 [11383]: Ending timer event 0x25babb0 "queue_next_trigger"
2012/10/08 20:26:36.882795 [11383]: server/ctdb_daemon.c:628 client request 13 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.882826 [11383]: Added timed event "queue_next_trigger": 0x25bab20
2012/10/08 20:26:36.882839 [11383]: Added timed event "ctdb_control_timeout": 0x25bad40
2012/10/08 20:26:36.882849 [11383]: server/ctdb_server.c:331 ctdb request 70 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.882858 [11383]: Freeze priority 3
2012/10/08 20:26:36.883334 [11383]: Ending timer event 0x25bab20 "queue_next_trigger"
2012/10/08 20:26:36.883736 [11383]: Added timed event "queue_next_trigger": 0x25bb010
2012/10/08 20:26:36.883766 [11383]: server/ctdb_server.c:331 ctdb request 70 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.883865 [11383]: Destroying timer event 0x25bad40 "ctdb_control_timeout"
2012/10/08 20:26:36.883877 [11383]: Ending timer event 0x25bb010 "queue_next_trigger"
2012/10/08 20:26:36.883937 [11383]: server/ctdb_daemon.c:628 client request 14 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.883952 [11383]: Added timed event "queue_next_trigger": 0x25baf80
2012/10/08 20:26:36.883962 [11383]: Added timed event "ctdb_control_timeout": 0x25bb1a0
2012/10/08 20:26:36.883972 [11383]: server/ctdb_server.c:331 ctdb request 71 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.883983 [11383]: Added timed event "ctdb_drop_all_ips_event": 0x25bb2c0
2012/10/08 20:26:36.883991 [11383]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:26:36.884001 [11383]: Added timed event "queue_next_trigger": 0x25bb570
2012/10/08 20:26:36.884010 [11383]: Ending timer event 0x25baf80 "queue_next_trigger"
2012/10/08 20:26:36.884019 [11383]: server/ctdb_server.c:331 ctdb request 71 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:36.884032 [11383]: Destroying timer event 0x25bb1a0 "ctdb_control_timeout"
2012/10/08 20:26:36.884042 [11383]: Ending timer event 0x25bb570 "queue_next_trigger"
2012/10/08 20:26:36.884091 [11383]: server/ctdb_daemon.c:628 client request 15 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.884124 [11383]: Added timed event "queue_next_trigger": 0x25bb450
2012/10/08 20:26:36.884136 [11383]: Added timed event "ctdb_control_timeout": 0x25bb570
2012/10/08 20:26:36.884146 [11383]: server/ctdb_server.c:331 ctdb request 72 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:36.884156 [11383]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:26:36.884182 [11383]: Monitoring has been disabled
2012/10/08 20:26:36.884214 [11383]: server/eventscript.c:754 Starting eventscript startrecovery
2012/10/08 20:26:36.884381 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:36.884418 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:36.884433 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:36.884452 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:36.884979 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.885093 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ade00
2012/10/08 20:26:36.885121 [11383]: Ending timer event 0x25bb450 "queue_next_trigger"
2012/10/08 20:26:36.885640 [eventscript-00.ctdb-startrecovery:12299]: Executing event script /etc/ctdb/events.d/00.ctdb startrecovery
2012/10/08 20:26:36.898677 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.899151 [11383]: SIGCHLD from 12299
2012/10/08 20:26:36.899445 [eventscript-01.reclock-startrecovery:12304]: Executing event script /etc/ctdb/events.d/01.reclock startrecovery
2012/10/08 20:26:36.908874 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.909371 [11383]: SIGCHLD from 12304
2012/10/08 20:26:36.909661 [eventscript-10.interface-startrecovery:12308]: Executing event script /etc/ctdb/events.d/10.interface startrecovery
2012/10/08 20:26:36.919114 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.919713 [11383]: SIGCHLD from 12308
2012/10/08 20:26:36.920007 [eventscript-11.natgw-startrecovery:12312]: Executing event script /etc/ctdb/events.d/11.natgw startrecovery
2012/10/08 20:26:36.929223 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.929697 [11383]: SIGCHLD from 12312
2012/10/08 20:26:36.929987 [eventscript-11.routing-startrecovery:12316]: Executing event script /etc/ctdb/events.d/11.routing startrecovery
2012/10/08 20:26:36.939246 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.939741 [11383]: SIGCHLD from 12316
2012/10/08 20:26:36.940059 [eventscript-13.per_ip_routing-startrecovery:12320]: Executing event script /etc/ctdb/events.d/13.per_ip_routing startrecovery
2012/10/08 20:26:36.952051 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.952554 [11383]: SIGCHLD from 12320
2012/10/08 20:26:36.952889 [eventscript-20.multipathd-startrecovery:12325]: Executing event script /etc/ctdb/events.d/20.multipathd startrecovery
2012/10/08 20:26:36.953514 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.954197 [eventscript-31.clamd-startrecovery:12326]: Executing event script /etc/ctdb/events.d/31.clamd startrecovery
2012/10/08 20:26:36.954775 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.955108 [11383]: SIGCHLD from 12325
2012/10/08 20:26:36.955382 [eventscript-40.vsftpd-startrecovery:12327]: Executing event script /etc/ctdb/events.d/40.vsftpd startrecovery
2012/10/08 20:26:36.961013 [11383]: SIGCHLD from 12326
2012/10/08 20:26:36.964913 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.965413 [11383]: SIGCHLD from 12327
2012/10/08 20:26:36.965724 [eventscript-40.fs_use-startrecovery:12331]: Executing event script /etc/ctdb/events.d/40.fs_use startrecovery
2012/10/08 20:26:36.966299 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.967036 [eventscript-41.httpd-startrecovery:12332]: Executing event script /etc/ctdb/events.d/41.httpd startrecovery
2012/10/08 20:26:36.967948 [11383]: Added timed event "ctdb_statistics_update": 0x25adff0
2012/10/08 20:26:36.967963 [11383]: Ending timer event 0x25b82e0 "ctdb_statistics_update"
2012/10/08 20:26:36.969491 [11383]: SIGCHLD from 12331
2012/10/08 20:26:36.976764 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.977284 [11383]: SIGCHLD from 12332
2012/10/08 20:26:36.977609 [eventscript-50.samba-startrecovery:12336]: Executing event script /etc/ctdb/events.d/50.samba startrecovery
2012/10/08 20:26:36.990123 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:36.990685 [11383]: SIGCHLD from 12336
2012/10/08 20:26:36.990974 [eventscript-60.nfs-startrecovery:12341]: Executing event script /etc/ctdb/events.d/60.nfs startrecovery
2012/10/08 20:26:37.005785 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:37.006328 [11383]: SIGCHLD from 12341
2012/10/08 20:26:37.006617 [eventscript-60.ganesha-startrecovery:12347]: Executing event script /etc/ctdb/events.d/60.ganesha startrecovery
2012/10/08 20:26:37.007253 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:37.007963 [eventscript-62.cnfs-startrecovery:12348]: Executing event script /etc/ctdb/events.d/62.cnfs startrecovery
2012/10/08 20:26:37.013382 [11383]: SIGCHLD from 12347
2012/10/08 20:26:37.019826 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:37.020362 [11383]: SIGCHLD from 12348
2012/10/08 20:26:37.020653 [eventscript-70.iscsi-startrecovery:12353]: Executing event script /etc/ctdb/events.d/70.iscsi startrecovery
2012/10/08 20:26:37.029449 [11383]: server/eventscript.c:414 Created PIPE FD:17 to child eventscript process
2012/10/08 20:26:37.029888 [11383]: SIGCHLD from 12353
2012/10/08 20:26:37.030104 [eventscript-91.lvs-startrecovery:12357]: Executing event script /etc/ctdb/events.d/91.lvs startrecovery
2012/10/08 20:26:37.036719 [11383]: server/eventscript.c:485 Eventscript startrecovery finished with state 0
2012/10/08 20:26:37.036807 [11383]: Added timed event "queue_next_trigger": 0x25b66a0
2012/10/08 20:26:37.036821 [11383]: Destroying timer event 0x25ade00 "ctdb_event_script_timeout"
2012/10/08 20:26:37.036848 [11383]: server/ctdb_server.c:331 ctdb request 72 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.036927 [11383]: Destroying timer event 0x25bb570 "ctdb_control_timeout"
2012/10/08 20:26:37.036937 [11383]: Ending timer event 0x25b66a0 "queue_next_trigger"
2012/10/08 20:26:37.036984 [11383]: server/ctdb_daemon.c:628 client request 16 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:37.036997 [11383]: Added timed event "queue_next_trigger": 0x25bb7e0
2012/10/08 20:26:37.037004 [11383]: Added timed event "ctdb_control_timeout": 0x25b6530
2012/10/08 20:26:37.037012 [11383]: server/ctdb_server.c:331 ctdb request 73 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037021 [11383]: Added timed event "queue_next_trigger": 0x25ad7c0
2012/10/08 20:26:37.037028 [11383]: Ending timer event 0x25bb7e0 "queue_next_trigger"
2012/10/08 20:26:37.037035 [11383]: server/ctdb_server.c:331 ctdb request 73 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037045 [11383]: Destroying timer event 0x25b6530 "ctdb_control_timeout"
2012/10/08 20:26:37.037052 [11383]: Ending timer event 0x25ad7c0 "queue_next_trigger"
2012/10/08 20:26:37.037082 [11383]: server/ctdb_daemon.c:628 client request 17 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037109 [11383]: Added timed event "queue_next_trigger": 0x25ad7c0
2012/10/08 20:26:37.037117 [11383]: Added timed event "ctdb_control_timeout": 0x25b6530
2012/10/08 20:26:37.037125 [11383]: server/ctdb_server.c:331 ctdb request 74 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037135 [11383]: Added timed event "queue_next_trigger": 0x25b6900
2012/10/08 20:26:37.037142 [11383]: Ending timer event 0x25ad7c0 "queue_next_trigger"
2012/10/08 20:26:37.037149 [11383]: server/ctdb_server.c:331 ctdb request 74 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:37.037159 [11383]: Destroying timer event 0x25b6530 "ctdb_control_timeout"
2012/10/08 20:26:37.037167 [11383]: Ending timer event 0x25b6900 "queue_next_trigger"
2012/10/08 20:26:37.037198 [11383]: server/ctdb_daemon.c:628 client request 18 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.037209 [11383]: Added timed event "queue_next_trigger": 0x25b62f0
2012/10/08 20:26:37.037216 [11383]: Added timed event "ctdb_control_timeout": 0x25a9fe0
2012/10/08 20:26:37.037223 [11383]: server/ctdb_server.c:331 ctdb request 75 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.037231 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:37.037239 [11383]: Added timed event "queue_next_trigger": 0x25ada10
2012/10/08 20:26:37.037246 [11383]: Ending timer event 0x25b62f0 "queue_next_trigger"
2012/10/08 20:26:37.037253 [11383]: server/ctdb_server.c:331 ctdb request 75 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037263 [11383]: Destroying timer event 0x25a9fe0 "ctdb_control_timeout"
2012/10/08 20:26:37.037270 [11383]: Ending timer event 0x25ada10 "queue_next_trigger"
2012/10/08 20:26:37.037290 [recoverd:11454]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:26:37.037313 [11383]: server/ctdb_daemon.c:628 client request 19 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.037323 [11383]: Added timed event "queue_next_trigger": 0x25b6830
2012/10/08 20:26:37.037331 [11383]: Added timed event "ctdb_control_timeout": 0x25ada10
2012/10/08 20:26:37.037338 [11383]: server/ctdb_server.c:331 ctdb request 76 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.037363 [11383]: Added timed event "queue_next_trigger": 0x25b98d0
2012/10/08 20:26:37.037372 [11383]: Ending timer event 0x25b6830 "queue_next_trigger"
2012/10/08 20:26:37.037380 [11383]: server/ctdb_server.c:331 ctdb request 76 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037390 [11383]: Destroying timer event 0x25ada10 "ctdb_control_timeout"
2012/10/08 20:26:37.037397 [11383]: Ending timer event 0x25b98d0 "queue_next_trigger"
2012/10/08 20:26:37.037427 [11383]: server/ctdb_daemon.c:628 client request 20 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037438 [11383]: Added timed event "queue_next_trigger": 0x25b9840
2012/10/08 20:26:37.037445 [11383]: Added timed event "ctdb_control_timeout": 0x25b1a30
2012/10/08 20:26:37.037453 [11383]: server/ctdb_server.c:331 ctdb request 77 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037461 [11383]: Added timed event "queue_next_trigger": 0x25b94a0
2012/10/08 20:26:37.037468 [11383]: Ending timer event 0x25b9840 "queue_next_trigger"
2012/10/08 20:26:37.037475 [11383]: server/ctdb_server.c:331 ctdb request 77 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037485 [11383]: Destroying timer event 0x25b1a30 "ctdb_control_timeout"
2012/10/08 20:26:37.037492 [11383]: Ending timer event 0x25b94a0 "queue_next_trigger"
2012/10/08 20:26:37.037639 [11383]: SIGCHLD from 12357
2012/10/08 20:26:37.037664 [recoverd:11454]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:26:37.037674 [recoverd:11454]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:26:37.037693 [11383]: server/ctdb_daemon.c:628 client request 21 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037703 [11383]: Added timed event "queue_next_trigger": 0x25b1b50
2012/10/08 20:26:37.037711 [11383]: Added timed event "ctdb_control_timeout": 0x25b9630
2012/10/08 20:26:37.037732 [11383]: server/ctdb_server.c:331 ctdb request 78 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037741 [11383]: server/ctdb_freeze.c:523 num_nodes[2]
2012/10/08 20:26:37.037747 [11383]: server/ctdb_freeze.c:526 node[0].flags[0x3]
2012/10/08 20:26:37.037753 [11383]: server/ctdb_freeze.c:526 node[1].flags[0x2]
2012/10/08 20:26:37.037759 [11383]: server/ctdb_freeze.c:531 healthy_nodes[0]
2012/10/08 20:26:37.037766 [11383]: Added timed event "queue_next_trigger": 0x25b5900
2012/10/08 20:26:37.037773 [11383]: Ending timer event 0x25b1b50 "queue_next_trigger"
2012/10/08 20:26:37.037780 [11383]: server/ctdb_server.c:331 ctdb request 78 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037790 [11383]: Destroying timer event 0x25b9630 "ctdb_control_timeout"
2012/10/08 20:26:37.037798 [11383]: Ending timer event 0x25b5900 "queue_next_trigger"
2012/10/08 20:26:37.037817 [recoverd:11454]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:26:37.037838 [11383]: server/ctdb_daemon.c:628 client request 22 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037848 [11383]: Added timed event "queue_next_trigger": 0x25adb30
2012/10/08 20:26:37.037856 [11383]: Added timed event "ctdb_control_timeout": 0x25b5850
2012/10/08 20:26:37.037863 [11383]: server/ctdb_server.c:331 ctdb request 79 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.037871 [11383]: Added timed event "queue_next_trigger": 0x25b5cb0
2012/10/08 20:26:37.037878 [11383]: Ending timer event 0x25adb30 "queue_next_trigger"
2012/10/08 20:26:37.037885 [11383]: server/ctdb_server.c:331 ctdb request 79 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037895 [11383]: Destroying timer event 0x25b5850 "ctdb_control_timeout"
2012/10/08 20:26:37.037902 [11383]: Ending timer event 0x25b5cb0 "queue_next_trigger"
2012/10/08 20:26:37.037934 [11383]: server/ctdb_daemon.c:628 client request 23 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.037944 [11383]: Added timed event "queue_next_trigger": 0x25adb30
2012/10/08 20:26:37.037952 [11383]: Added timed event "ctdb_control_timeout": 0x25b5c00
2012/10/08 20:26:37.037959 [11383]: server/ctdb_server.c:331 ctdb request 80 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.037967 [11383]: Added timed event "queue_next_trigger": 0x25b6f20
2012/10/08 20:26:37.037974 [11383]: Ending timer event 0x25adb30 "queue_next_trigger"
2012/10/08 20:26:37.037981 [11383]: server/ctdb_server.c:331 ctdb request 80 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.037991 [11383]: Destroying timer event 0x25b5c00 "ctdb_control_timeout"
2012/10/08 20:26:37.037998 [11383]: Ending timer event 0x25b6f20 "queue_next_trigger"
2012/10/08 20:26:37.038016 [recoverd:11454]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:26:37.038037 [11383]: server/ctdb_daemon.c:628 client request 24 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038047 [11383]: Added timed event "queue_next_trigger": 0x25adb30
2012/10/08 20:26:37.038054 [11383]: Added timed event "ctdb_control_timeout": 0x25b6e70
2012/10/08 20:26:37.038061 [11383]: server/ctdb_server.c:331 ctdb request 81 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038070 [11383]: Added timed event "queue_next_trigger": 0x25bb490
2012/10/08 20:26:37.038077 [11383]: Ending timer event 0x25adb30 "queue_next_trigger"
2012/10/08 20:26:37.038084 [11383]: server/ctdb_server.c:331 ctdb request 81 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.038093 [11383]: Destroying timer event 0x25b6e70 "ctdb_control_timeout"
2012/10/08 20:26:37.038100 [11383]: Ending timer event 0x25bb490 "queue_next_trigger"
2012/10/08 20:26:37.038119 [recoverd:11454]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:26:37.038140 [11383]: server/ctdb_daemon.c:628 client request 25 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:37.038150 [11383]: Added timed event "queue_next_trigger": 0x25adb30
2012/10/08 20:26:37.038158 [11383]: Added timed event "ctdb_control_timeout": 0x25bb3e0
2012/10/08 20:26:37.038178 [11383]: server/ctdb_server.c:331 ctdb request 82 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038187 [11383]: Added timed event "queue_next_trigger": 0x25ba770
2012/10/08 20:26:37.038194 [11383]: Ending timer event 0x25adb30 "queue_next_trigger"
2012/10/08 20:26:37.038201 [11383]: server/ctdb_server.c:331 ctdb request 82 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.038211 [11383]: Destroying timer event 0x25bb3e0 "ctdb_control_timeout"
2012/10/08 20:26:37.038218 [11383]: Ending timer event 0x25ba770 "queue_next_trigger"
2012/10/08 20:26:37.038271 [11383]: server/ctdb_daemon.c:628 client request 26 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038282 [11383]: Added timed event "queue_next_trigger": 0x25ba900
2012/10/08 20:26:37.038289 [11383]: Added timed event "ctdb_control_timeout": 0x25ba6c0
2012/10/08 20:26:37.038296 [11383]: server/ctdb_server.c:331 ctdb request 83 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038306 [11383]: Added timed event "queue_next_trigger": 0x25bb160
2012/10/08 20:26:37.038313 [11383]: Ending timer event 0x25ba900 "queue_next_trigger"
2012/10/08 20:26:37.038320 [11383]: server/ctdb_server.c:331 ctdb request 83 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:37.038329 [11383]: Destroying timer event 0x25ba6c0 "ctdb_control_timeout"
2012/10/08 20:26:37.038337 [11383]: Ending timer event 0x25bb160 "queue_next_trigger"
2012/10/08 20:26:37.038367 [11383]: server/ctdb_daemon.c:628 client request 27 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.038377 [11383]: Added timed event "queue_next_trigger": 0x25baf10
2012/10/08 20:26:37.038385 [11383]: Added timed event "ctdb_control_timeout": 0x25bb030
2012/10/08 20:26:37.038392 [11383]: server/ctdb_server.c:331 ctdb request 84 of type 7 length 72 from node 1 to 1
2012/10/08 20:26:37.038399 [11383]: Control modflags on node 1 - Unchanged - flags 0x2
2012/10/08 20:26:37.038406 [11383]: Added timed event "queue_next_trigger": 0x25babd0
2012/10/08 20:26:37.038413 [11383]: Ending timer event 0x25baf10 "queue_next_trigger"
2012/10/08 20:26:37.038420 [11383]: server/ctdb_server.c:331 ctdb request 84 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.038430 [11383]: Destroying timer event 0x25bb030 "ctdb_control_timeout"
2012/10/08 20:26:37.038438 [11383]: Ending timer event 0x25babd0 "queue_next_trigger"
2012/10/08 20:26:37.038457 [recoverd:11454]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:26:37.038478 [11383]: server/ctdb_daemon.c:628 client request 28 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038489 [11383]: Added timed event "queue_next_trigger": 0x25bad60
2012/10/08 20:26:37.038496 [11383]: Added timed event "ctdb_control_timeout": 0x25baab0
2012/10/08 20:26:37.038503 [11383]: server/ctdb_server.c:331 ctdb request 85 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.038511 [11383]: Destroying timer event 0x25bb2c0 "ctdb_drop_all_ips_event"
2012/10/08 20:26:37.038518 [11383]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:26:37.038525 [11383]: Thawing priority 1
2012/10/08 20:26:37.038531 [11383]: Release freeze handler for prio 1
2012/10/08 20:26:37.038544 [11383]: Thawing priority 2
2012/10/08 20:26:37.038551 [11383]: Release freeze handler for prio 2
2012/10/08 20:26:37.038561 [11383]: Thawing priority 3
2012/10/08 20:26:37.038568 [11383]: Release freeze handler for prio 3
2012/10/08 20:26:37.038869 [11383]: server/ctdb_recover.c:808 Created PIPE FD:11 for setrecmode
2012/10/08 20:26:37.038934 [11383]: Added timed event "ctdb_set_recmode_timeout": 0x25b8380
2012/10/08 20:26:37.038994 [11383]: Ending timer event 0x25bad60 "queue_next_trigger"
2012/10/08 20:26:37.039709 [11383]: SIGCHLD from 12296
2012/10/08 20:26:37.039722 [11383]: SIGCHLD from 12297
2012/10/08 20:26:37.039731 [11383]: SIGCHLD from 12298
2012/10/08 20:26:37.043433 [11383]: Destroying timer event 0x25b8380 "ctdb_set_recmode_timeout"
2012/10/08 20:26:37.043448 [11383]: Added timed event "queue_next_trigger": 0x25b55e0
2012/10/08 20:26:37.043483 [11383]: server/ctdb_server.c:331 ctdb request 85 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.043550 [11383]: Destroying timer event 0x25baab0 "ctdb_control_timeout"
2012/10/08 20:26:37.043559 [11383]: Ending timer event 0x25b55e0 "queue_next_trigger"
2012/10/08 20:26:37.043690 [11383]: SIGCHLD from 12361
2012/10/08 20:26:37.043719 [recoverd:11454]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:26:37.043743 [11383]: server/ctdb_daemon.c:628 client request 29 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.043754 [11383]: Added timed event "queue_next_trigger": 0x25badd0
2012/10/08 20:26:37.043761 [11383]: Added timed event "ctdb_control_timeout": 0x25bae80
2012/10/08 20:26:37.043769 [11383]: server/ctdb_server.c:331 ctdb request 86 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.043777 [11383]: Added timed event "queue_next_trigger": 0x25b8880
2012/10/08 20:26:37.043785 [11383]: Ending timer event 0x25badd0 "queue_next_trigger"
2012/10/08 20:26:37.043792 [11383]: server/ctdb_server.c:331 ctdb request 86 of type 8 length 80 from node 1 to 1
2012/10/08 20:26:37.043802 [11383]: Destroying timer event 0x25bae80 "ctdb_control_timeout"
2012/10/08 20:26:37.043809 [11383]: Ending timer event 0x25b8880 "queue_next_trigger"
2012/10/08 20:26:37.043841 [11383]: server/ctdb_daemon.c:628 client request 30 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.043851 [11383]: Added timed event "queue_next_trigger": 0x25bae70
2012/10/08 20:26:37.043859 [11383]: Added timed event "ctdb_control_timeout": 0x25b8a80
2012/10/08 20:26:37.043866 [11383]: server/ctdb_server.c:331 ctdb request 87 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.043875 [11383]: Added timed event "queue_next_trigger": 0x25b87b0
2012/10/08 20:26:37.043882 [11383]: Ending timer event 0x25bae70 "queue_next_trigger"
2012/10/08 20:26:37.043889 [11383]: server/ctdb_server.c:331 ctdb request 87 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.043899 [11383]: Destroying timer event 0x25b8a80 "ctdb_control_timeout"
2012/10/08 20:26:37.043906 [11383]: Ending timer event 0x25b87b0 "queue_next_trigger"
2012/10/08 20:26:37.043930 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:37.043938 [recoverd:11454]: CONSIDERING MOVES (UNASSIGNED)
2012/10/08 20:26:37.043944 [recoverd:11454]: ----------------------------------------
2012/10/08 20:26:37.043953 [recoverd:11454]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:26:37.043972 [11383]: server/ctdb_daemon.c:628 client request 31 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:37.043982 [11383]: Added timed event "queue_next_trigger": 0x25badd0
2012/10/08 20:26:37.043990 [11383]: Added timed event "ctdb_control_timeout": 0x25b8d00
2012/10/08 20:26:37.043998 [11383]: server/ctdb_server.c:331 ctdb request 88 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:37.044025 [11383]: Redundant release of IP 10.100.55.198/24 on interface __none__ (ip not held)
2012/10/08 20:26:37.044045 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/08 20:26:37.044054 [11383]: Added timed event "queue_next_trigger": 0x25b8f40
2012/10/08 20:26:37.044061 [11383]: Ending timer event 0x25badd0 "queue_next_trigger"
2012/10/08 20:26:37.044068 [11383]: server/ctdb_server.c:331 ctdb request 88 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.044079 [11383]: Destroying timer event 0x25b8d00 "ctdb_control_timeout"
2012/10/08 20:26:37.044086 [11383]: Ending timer event 0x25b8f40 "queue_next_trigger"
2012/10/08 20:26:37.044119 [11383]: server/ctdb_daemon.c:628 client request 32 of type 7 length 80 from node 1 to 1
2012/10/08 20:26:37.044129 [11383]: Added timed event "queue_next_trigger": 0x25b8e20
2012/10/08 20:26:37.044137 [11383]: Added timed event "ctdb_control_timeout": 0x25b8f40
2012/10/08 20:26:37.044144 [11383]: server/ctdb_server.c:331 ctdb request 89 of type 7 length 74 from node 1 to 1
2012/10/08 20:26:37.044171 [11383]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:26:37.044179 [11383]: Monitoring has been disabled
2012/10/08 20:26:37.044187 [11383]: server/eventscript.c:754 Starting eventscript ipreallocated
2012/10/08 20:26:37.044272 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:37.044299 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:37.044311 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:37.044325 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:37.044714 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.044805 [11383]: Added timed event "ctdb_event_script_timeout": 0x25b66a0
2012/10/08 20:26:37.044825 [11383]: Ending timer event 0x25b8e20 "queue_next_trigger"
2012/10/08 20:26:37.045166 [eventscript-00.ctdb-ipreallocated:12362]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/00.ctdb ipreallocated
2012/10/08 20:26:37.054175 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.054575 [11383]: SIGCHLD from 12362
2012/10/08 20:26:37.054803 [eventscript-01.reclock-ipreallocated:12367]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/01.reclock ipreallocated
2012/10/08 20:26:37.061660 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.062023 [11383]: SIGCHLD from 12367
2012/10/08 20:26:37.062263 [eventscript-10.interface-ipreallocated:12371]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/10.interface ipreallocated
2012/10/08 20:26:37.069093 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.069469 [11383]: SIGCHLD from 12371
2012/10/08 20:26:37.069694 [eventscript-11.natgw-ipreallocated:12375]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.natgw ipreallocated
2012/10/08 20:26:37.076296 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.076644 [11383]: SIGCHLD from 12375
2012/10/08 20:26:37.076857 [eventscript-11.routing-ipreallocated:12379]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/11.routing ipreallocated
2012/10/08 20:26:37.083657 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.084024 [11383]: SIGCHLD from 12379
2012/10/08 20:26:37.084237 [eventscript-13.per_ip_routing-ipreallocated:12383]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/13.per_ip_routing ipreallocated
2012/10/08 20:26:37.092978 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.093334 [11383]: SIGCHLD from 12383
2012/10/08 20:26:37.093542 [eventscript-20.multipathd-ipreallocated:12388]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/20.multipathd ipreallocated
2012/10/08 20:26:37.093949 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.094537 [eventscript-31.clamd-ipreallocated:12389]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/31.clamd ipreallocated
2012/10/08 20:26:37.094973 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.095197 [11383]: SIGCHLD from 12388
2012/10/08 20:26:37.095412 [eventscript-40.vsftpd-ipreallocated:12390]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.vsftpd ipreallocated
2012/10/08 20:26:37.097246 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25b43d0
2012/10/08 20:26:37.097257 [11383]: Ending timer event 0x25b6480 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:37.097411 [11383]: SIGCHLD from 12389
2012/10/08 20:26:37.102520 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.102893 [11383]: SIGCHLD from 12390
2012/10/08 20:26:37.103112 [eventscript-40.fs_use-ipreallocated:12394]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/40.fs_use ipreallocated
2012/10/08 20:26:37.103522 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.104042 [eventscript-41.httpd-ipreallocated:12395]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/41.httpd ipreallocated
2012/10/08 20:26:37.106279 [11383]: SIGCHLD from 12394
2012/10/08 20:26:37.110848 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.111218 [11383]: SIGCHLD from 12395
2012/10/08 20:26:37.111436 [eventscript-50.samba-ipreallocated:12399]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/50.samba ipreallocated
2012/10/08 20:26:37.120679 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.121039 [11383]: SIGCHLD from 12399
2012/10/08 20:26:37.121253 [eventscript-60.nfs-ipreallocated:12404]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.nfs ipreallocated
2012/10/08 20:26:37.125073 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:37.125094 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:37.125105 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25aa8e0
2012/10/08 20:26:37.125111 [11383]: Ending timer event 0x25b84e0 "ctdb_wait_until_recovered"
2012/10/08 20:26:37.125137 [11383]: Added timed event "ctdb_time_tick": 0x25ad7c0
2012/10/08 20:26:37.125146 [11383]: Ending timer event 0x25b8030 "ctdb_time_tick"
2012/10/08 20:26:37.132162 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.132569 [11383]: SIGCHLD from 12404
2012/10/08 20:26:37.132819 [eventscript-60.ganesha-ipreallocated:12410]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/60.ganesha ipreallocated
2012/10/08 20:26:37.133255 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.133613 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:12
2012/10/08 20:26:37.133656 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25ae290
2012/10/08 20:26:37.133665 [11383]: Ending timer event 0x25b8600 "ctdb_tcp_node_connect"
2012/10/08 20:26:37.133884 [eventscript-62.cnfs-ipreallocated:12411]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/62.cnfs ipreallocated
2012/10/08 20:26:37.134468 [11383]: Destroying timer event 0x25ae290 "ctdb_tcp_node_connect"
2012/10/08 20:26:37.134501 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25aad80
2012/10/08 20:26:37.138001 [11383]: SIGCHLD from 12410
2012/10/08 20:26:37.142922 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.143287 [11383]: SIGCHLD from 12411
2012/10/08 20:26:37.143489 [eventscript-70.iscsi-ipreallocated:12416]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/70.iscsi ipreallocated
2012/10/08 20:26:37.150155 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.150542 [11383]: SIGCHLD from 12416
2012/10/08 20:26:37.150757 [eventscript-91.lvs-ipreallocated:12420]: Executing event script CTDB_CALLED_BY_USER=1 /etc/ctdb/events.d/91.lvs ipreallocated
2012/10/08 20:26:37.157025 [11383]: server/eventscript.c:485 Eventscript ipreallocated finished with state 0
2012/10/08 20:26:37.157084 [11383]: Monitoring has been enabled
2012/10/08 20:26:37.157096 [11383]: Added timed event "queue_next_trigger": 0x25b1f40
2012/10/08 20:26:37.157108 [11383]: Destroying timer event 0x25b66a0 "ctdb_event_script_timeout"
2012/10/08 20:26:37.157117 [11383]: server/ctdb_server.c:331 ctdb request 89 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.157177 [11383]: Destroying timer event 0x25b8f40 "ctdb_control_timeout"
2012/10/08 20:26:37.157187 [11383]: Ending timer event 0x25b1f40 "queue_next_trigger"
2012/10/08 20:26:37.157239 [11383]: server/ctdb_daemon.c:628 client request 33 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.157251 [11383]: Added timed event "queue_next_trigger": 0x25ae290
2012/10/08 20:26:37.157258 [11383]: Added timed event "ctdb_control_timeout": 0x25b6830
2012/10/08 20:26:37.157266 [11383]: server/ctdb_server.c:331 ctdb request 90 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:37.157275 [11383]: Recovery has finished
2012/10/08 20:26:37.157282 [11383]: Monitoring has been disabled
2012/10/08 20:26:37.157290 [11383]: server/eventscript.c:754 Starting eventscript recovered
2012/10/08 20:26:37.157344 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:37.157370 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:37.157381 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:37.157396 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:37.157769 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.157852 [11383]: Added timed event "ctdb_event_script_timeout": 0x25b9660
2012/10/08 20:26:37.157872 [11383]: Ending timer event 0x25ae290 "queue_next_trigger"
2012/10/08 20:26:37.158147 [11383]: SIGCHLD from 12420
2012/10/08 20:26:37.158381 [eventscript-00.ctdb-recovered:12424]: Executing event script /etc/ctdb/events.d/00.ctdb recovered
2012/10/08 20:26:37.169345 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.169806 [11383]: SIGCHLD from 12424
2012/10/08 20:26:37.170108 [eventscript-01.reclock-recovered:12429]: Executing event script /etc/ctdb/events.d/01.reclock recovered
2012/10/08 20:26:37.178986 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.179470 [11383]: SIGCHLD from 12429
2012/10/08 20:26:37.179784 [eventscript-10.interface-recovered:12433]: Executing event script /etc/ctdb/events.d/10.interface recovered
2012/10/08 20:26:37.189023 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.189475 [11383]: SIGCHLD from 12433
2012/10/08 20:26:37.189782 [eventscript-11.natgw-recovered:12437]: Executing event script /etc/ctdb/events.d/11.natgw recovered
2012/10/08 20:26:37.198668 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.199140 [11383]: SIGCHLD from 12437
2012/10/08 20:26:37.199449 [eventscript-11.routing-recovered:12441]: Executing event script /etc/ctdb/events.d/11.routing recovered
2012/10/08 20:26:37.208423 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.208886 [11383]: SIGCHLD from 12441
2012/10/08 20:26:37.209177 [eventscript-13.per_ip_routing-recovered:12445]: Executing event script /etc/ctdb/events.d/13.per_ip_routing recovered
2012/10/08 20:26:37.220659 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.221149 [11383]: SIGCHLD from 12445
2012/10/08 20:26:37.221455 [eventscript-20.multipathd-recovered:12450]: Executing event script /etc/ctdb/events.d/20.multipathd recovered
2012/10/08 20:26:37.222016 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.222744 [eventscript-31.clamd-recovered:12451]: Executing event script /etc/ctdb/events.d/31.clamd recovered
2012/10/08 20:26:37.223314 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.223612 [11383]: SIGCHLD from 12450
2012/10/08 20:26:37.223896 [eventscript-40.vsftpd-recovered:12452]: Executing event script /etc/ctdb/events.d/40.vsftpd recovered
2012/10/08 20:26:37.225877 [11383]: SIGCHLD from 12451
2012/10/08 20:26:37.233482 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.233959 [11383]: SIGCHLD from 12452
2012/10/08 20:26:37.234298 [eventscript-40.fs_use-recovered:12456]: Executing event script /etc/ctdb/events.d/40.fs_use recovered
2012/10/08 20:26:37.234862 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.235601 [eventscript-41.httpd-recovered:12457]: Executing event script /etc/ctdb/events.d/41.httpd recovered
2012/10/08 20:26:37.240958 [11383]: SIGCHLD from 12456
2012/10/08 20:26:37.244742 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.245199 [11383]: SIGCHLD from 12457
2012/10/08 20:26:37.245508 [eventscript-50.samba-recovered:12461]: Executing event script /etc/ctdb/events.d/50.samba recovered
2012/10/08 20:26:37.258180 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.258685 [11383]: SIGCHLD from 12461
2012/10/08 20:26:37.258972 [eventscript-60.nfs-recovered:12466]: Executing event script /etc/ctdb/events.d/60.nfs recovered
2012/10/08 20:26:37.273425 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.273894 [11383]: SIGCHLD from 12466
2012/10/08 20:26:37.274195 [eventscript-60.ganesha-recovered:12472]: Executing event script /etc/ctdb/events.d/60.ganesha recovered
2012/10/08 20:26:37.274774 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.275519 [eventscript-62.cnfs-recovered:12473]: Executing event script /etc/ctdb/events.d/62.cnfs recovered
2012/10/08 20:26:37.280915 [11383]: SIGCHLD from 12472
2012/10/08 20:26:37.287286 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.287755 [11383]: SIGCHLD from 12473
2012/10/08 20:26:37.288057 [eventscript-70.iscsi-recovered:12478]: Executing event script /etc/ctdb/events.d/70.iscsi recovered
2012/10/08 20:26:37.297301 [11383]: server/eventscript.c:414 Created PIPE FD:11 to child eventscript process
2012/10/08 20:26:37.297652 [11383]: SIGCHLD from 12478
2012/10/08 20:26:37.297864 [eventscript-91.lvs-recovered:12482]: Executing event script /etc/ctdb/events.d/91.lvs recovered
2012/10/08 20:26:37.304086 [11383]: server/eventscript.c:485 Eventscript recovered finished with state 0
2012/10/08 20:26:37.304142 [11383]: Monitoring has been enabled
2012/10/08 20:26:37.304166 [11383]: Added timed event "queue_next_trigger": 0x25b1f40
2012/10/08 20:26:37.304179 [11383]: Destroying timer event 0x25b9660 "ctdb_event_script_timeout"
2012/10/08 20:26:37.304188 [11383]: server/ctdb_server.c:331 ctdb request 90 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:37.304236 [11383]: Destroying timer event 0x25b6830 "ctdb_control_timeout"
2012/10/08 20:26:37.304246 [11383]: Ending timer event 0x25b1f40 "queue_next_trigger"
2012/10/08 20:26:37.304275 [recoverd:11454]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:26:37.304300 [11383]: server/ctdb_daemon.c:628 client request 0 of type 5 length 48 from node 1 to 4026531844
2012/10/08 20:26:37.304311 [11383]: Added timed event "queue_next_trigger": 0x25b66a0
2012/10/08 20:26:37.304319 [11383]: server/ctdb_server.c:331 ctdb request 0 of type 5 length 48 from node 1 to 1
2012/10/08 20:26:37.304327 [11383]: Ending timer event 0x25b66a0 "queue_next_trigger"
2012/10/08 20:26:37.304340 [recoverd:11454]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:26:37.304350 [recoverd:11454]: Resetting ban count to 0 for all nodes
2012/10/08 20:26:37.304357 [recoverd:11454]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:26:37.304495 [11383]: SIGCHLD from 12482
2012/10/08 20:26:37.968615 [11383]: Added timed event "ctdb_statistics_update": 0x25ade00
2012/10/08 20:26:37.968632 [11383]: Ending timer event 0x25adff0 "ctdb_statistics_update"
2012/10/08 20:26:38.125685 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:38.125700 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:38.125721 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b66a0
2012/10/08 20:26:38.125730 [11383]: Ending timer event 0x25aa8e0 "ctdb_wait_until_recovered"
2012/10/08 20:26:38.125740 [11383]: Added timed event "ctdb_time_tick": 0x25aa8e0
2012/10/08 20:26:38.125749 [11383]: Ending timer event 0x25ad7c0 "ctdb_time_tick"
2012/10/08 20:26:38.134823 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:38.134873 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25aef40
2012/10/08 20:26:38.134885 [11383]: Ending timer event 0x25aad80 "ctdb_tcp_node_connect"
2012/10/08 20:26:38.135169 [11383]: Destroying timer event 0x25aef40 "ctdb_tcp_node_connect"
2012/10/08 20:26:38.135194 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25ad7c0
2012/10/08 20:26:38.969255 [11383]: Added timed event "ctdb_statistics_update": 0x25aef40
2012/10/08 20:26:38.969274 [11383]: Ending timer event 0x25ade00 "ctdb_statistics_update"
2012/10/08 20:26:39.126328 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:39.126345 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:39.126357 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b1a30
2012/10/08 20:26:39.126366 [11383]: Ending timer event 0x25b66a0 "ctdb_wait_until_recovered"
2012/10/08 20:26:39.126375 [11383]: Added timed event "ctdb_time_tick": 0x25b66a0
2012/10/08 20:26:39.126384 [11383]: Ending timer event 0x25aa8e0 "ctdb_time_tick"
2012/10/08 20:26:39.135441 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:39.135475 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1b50
2012/10/08 20:26:39.135487 [11383]: Ending timer event 0x25ad7c0 "ctdb_tcp_node_connect"
2012/10/08 20:26:39.135756 [11383]: Destroying timer event 0x25b1b50 "ctdb_tcp_node_connect"
2012/10/08 20:26:39.135778 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1b50
2012/10/08 20:26:39.707689 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:39.707856 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:39.707876 [11383]: Added timed event "queue_next_trigger": 0x25b23a0
2012/10/08 20:26:39.707887 [11383]: Added timed event "ctdb_control_timeout": 0x25b24c0
2012/10/08 20:26:39.707897 [11383]: server/ctdb_server.c:331 ctdb request 92 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.707908 [11383]: Added timed event "queue_next_trigger": 0x25b5630
2012/10/08 20:26:39.707918 [11383]: Ending timer event 0x25b23a0 "queue_next_trigger"
2012/10/08 20:26:39.707928 [11383]: server/ctdb_server.c:331 ctdb request 92 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:39.707942 [11383]: Destroying timer event 0x25b24c0 "ctdb_control_timeout"
2012/10/08 20:26:39.707952 [11383]: Ending timer event 0x25b5630 "queue_next_trigger"
2012/10/08 20:26:39.708014 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:39.708072 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:39.708086 [11383]: Added timed event "queue_next_trigger": 0x25b2790
2012/10/08 20:26:39.708096 [11383]: Added timed event "ctdb_control_timeout": 0x25b28b0
2012/10/08 20:26:39.708106 [11383]: server/ctdb_server.c:331 ctdb request 94 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708117 [11383]: Added timed event "queue_next_trigger": 0x25b4830
2012/10/08 20:26:39.708127 [11383]: Ending timer event 0x25b2790 "queue_next_trigger"
2012/10/08 20:26:39.708136 [11383]: server/ctdb_server.c:331 ctdb request 94 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:39.708149 [11383]: Destroying timer event 0x25b28b0 "ctdb_control_timeout"
2012/10/08 20:26:39.708159 [11383]: Ending timer event 0x25b4830 "queue_next_trigger"
2012/10/08 20:26:39.708207 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708230 [11383]: Added timed event "queue_next_trigger": 0x25b4710
2012/10/08 20:26:39.708240 [11383]: Added timed event "ctdb_control_timeout": 0x25b4830
2012/10/08 20:26:39.708250 [11383]: server/ctdb_server.c:331 ctdb request 95 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708260 [11383]: Added timed event "queue_next_trigger": 0x25b4b50
2012/10/08 20:26:39.708270 [11383]: Ending timer event 0x25b4710 "queue_next_trigger"
2012/10/08 20:26:39.708279 [11383]: server/ctdb_server.c:331 ctdb request 95 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:39.708292 [11383]: Destroying timer event 0x25b4830 "ctdb_control_timeout"
2012/10/08 20:26:39.708301 [11383]: Ending timer event 0x25b4b50 "queue_next_trigger"
2012/10/08 20:26:39.708352 [11383]: server/ctdb_daemon.c:628 client request 2147483451 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708367 [11383]: Added timed event "queue_next_trigger": 0x25b4a30
2012/10/08 20:26:39.708377 [11383]: Added timed event "ctdb_control_timeout": 0x25b4ae0
2012/10/08 20:26:39.708386 [11383]: server/ctdb_server.c:331 ctdb request 96 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708399 [11383]: Added timed event "queue_next_trigger": 0x25b4f30
2012/10/08 20:26:39.708409 [11383]: Ending timer event 0x25b4a30 "queue_next_trigger"
2012/10/08 20:26:39.708418 [11383]: server/ctdb_server.c:331 ctdb request 96 of type 8 length 120 from node 1 to 1
2012/10/08 20:26:39.708431 [11383]: Destroying timer event 0x25b4ae0 "ctdb_control_timeout"
2012/10/08 20:26:39.708441 [11383]: Ending timer event 0x25b4f30 "queue_next_trigger"
2012/10/08 20:26:39.708842 [11383]: server/ctdb_daemon.c:628 client request 2147483452 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708859 [11383]: Added timed event "queue_next_trigger": 0x25b50c0
2012/10/08 20:26:39.708869 [11383]: Added timed event "ctdb_control_timeout": 0x25b51e0
2012/10/08 20:26:39.708880 [11383]: server/ctdb_server.c:331 ctdb request 97 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.708891 [11383]: Added timed event "queue_next_trigger": 0x25b4ee0
2012/10/08 20:26:39.708901 [11383]: Ending timer event 0x25b50c0 "queue_next_trigger"
2012/10/08 20:26:39.708911 [11383]: server/ctdb_server.c:331 ctdb request 97 of type 8 length 56 from node 1 to 1
2012/10/08 20:26:39.708925 [11383]: Destroying timer event 0x25b51e0 "ctdb_control_timeout"
2012/10/08 20:26:39.708935 [11383]: Ending timer event 0x25b4ee0 "queue_next_trigger"
2012/10/08 20:26:39.709208 [11383]: server/ctdb_daemon.c:628 client request 1 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:39.709223 [11383]: Added timed event "queue_next_trigger": 0x25b51e0
2012/10/08 20:26:39.709295 [11383]: Added timed event "ctdb_control_timeout": 0x25b4e50
2012/10/08 20:26:39.709306 [11383]: server/ctdb_server.c:331 ctdb request 98 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.709318 [11383]: Added timed event "queue_next_trigger": 0x25b9bc0
2012/10/08 20:26:39.709328 [11383]: Ending timer event 0x25b51e0 "queue_next_trigger"
2012/10/08 20:26:39.709337 [11383]: server/ctdb_server.c:331 ctdb request 98 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:39.709351 [11383]: Destroying timer event 0x25b4e50 "ctdb_control_timeout"
2012/10/08 20:26:39.709370 [11383]: Ending timer event 0x25b9bc0 "queue_next_trigger"
2012/10/08 20:26:39.709405 [11383]: server/ctdb_daemon.c:628 client request 0 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:39.709419 [11383]: Added timed event "queue_next_trigger": 0x25b4e50
2012/10/08 20:26:39.709429 [11383]: Added timed event "ctdb_control_timeout": 0x25b9b30
2012/10/08 20:26:39.709439 [11383]: server/ctdb_server.c:331 ctdb request 99 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.709450 [11383]: Added timed event "queue_next_trigger": 0x25b9e70
2012/10/08 20:26:39.709459 [11383]: Ending timer event 0x25b4e50 "queue_next_trigger"
2012/10/08 20:26:39.709468 [11383]: server/ctdb_server.c:331 ctdb request 99 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:39.709481 [11383]: Destroying timer event 0x25b9b30 "ctdb_control_timeout"
2012/10/08 20:26:39.709497 [11383]: Ending timer event 0x25b9e70 "queue_next_trigger"
2012/10/08 20:26:39.709676 [11383]: server/ctdb_daemon.c:628 client request 2 of type 7 length 64 from node 0 to 1
2012/10/08 20:26:39.709692 [11383]: Added timed event "queue_next_trigger": 0x25b9d50
2012/10/08 20:26:39.709702 [11383]: Added timed event "ctdb_control_timeout": 0x25b9e70
2012/10/08 20:26:39.709713 [11383]: server/ctdb_server.c:331 ctdb request 100 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:39.709724 [11383]: Added timed event "queue_next_trigger": 0x25ba190
2012/10/08 20:26:39.709734 [11383]: Ending timer event 0x25b9d50 "queue_next_trigger"
2012/10/08 20:26:39.709743 [11383]: server/ctdb_server.c:331 ctdb request 100 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:39.709756 [11383]: Destroying timer event 0x25b9e70 "ctdb_control_timeout"
2012/10/08 20:26:39.709766 [11383]: Ending timer event 0x25ba190 "queue_next_trigger"
2012/10/08 20:26:39.970105 [11383]: Added timed event "ctdb_statistics_update": 0x25b9b30
2012/10/08 20:26:39.970121 [11383]: Ending timer event 0x25aef40 "ctdb_statistics_update"
2012/10/08 20:26:40.127171 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:40.127187 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:40.127198 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b9e70
2012/10/08 20:26:40.127207 [11383]: Ending timer event 0x25b1a30 "ctdb_wait_until_recovered"
2012/10/08 20:26:40.127217 [11383]: Added timed event "ctdb_time_tick": 0x25b50c0
2012/10/08 20:26:40.127225 [11383]: Ending timer event 0x25b66a0 "ctdb_time_tick"
2012/10/08 20:26:40.136283 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:40.136319 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b66a0
2012/10/08 20:26:40.136331 [11383]: Ending timer event 0x25b1b50 "ctdb_tcp_node_connect"
2012/10/08 20:26:40.136585 [11383]: Destroying timer event 0x25b66a0 "ctdb_tcp_node_connect"
2012/10/08 20:26:40.136606 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1a30
2012/10/08 20:26:40.970679 [11383]: Added timed event "ctdb_statistics_update": 0x25b64f0
2012/10/08 20:26:40.970696 [11383]: Ending timer event 0x25b9b30 "ctdb_statistics_update"
2012/10/08 20:26:41.127747 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:41.127761 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:41.127772 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b2160
2012/10/08 20:26:41.127781 [11383]: Ending timer event 0x25b9e70 "ctdb_wait_until_recovered"
2012/10/08 20:26:41.127791 [11383]: Added timed event "ctdb_time_tick": 0x25b9e70
2012/10/08 20:26:41.127799 [11383]: Ending timer event 0x25b50c0 "ctdb_time_tick"
2012/10/08 20:26:41.136857 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:41.136891 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1ce0
2012/10/08 20:26:41.136903 [11383]: Ending timer event 0x25b1a30 "ctdb_tcp_node_connect"
2012/10/08 20:26:41.137162 [11383]: Destroying timer event 0x25b1ce0 "ctdb_tcp_node_connect"
2012/10/08 20:26:41.137184 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b1ce0
2012/10/08 20:26:41.971243 [11383]: Added timed event "ctdb_statistics_update": 0x25ba0e0
2012/10/08 20:26:41.971260 [11383]: Ending timer event 0x25b64f0 "ctdb_statistics_update"
2012/10/08 20:26:42.097317 [11383]: Added timed event "ctdb_check_for_dead_nodes": 0x25b64f0
2012/10/08 20:26:42.097333 [11383]: Ending timer event 0x25b43d0 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:42.128384 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:42.128399 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:42.128410 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b58c0
2012/10/08 20:26:42.128419 [11383]: Ending timer event 0x25b2160 "ctdb_wait_until_recovered"
2012/10/08 20:26:42.128429 [11383]: Added timed event "ctdb_time_tick": 0x25b2160
2012/10/08 20:26:42.128444 [11383]: Ending timer event 0x25b9e70 "ctdb_time_tick"
2012/10/08 20:26:42.137505 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:42.137539 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5970
2012/10/08 20:26:42.137551 [11383]: Ending timer event 0x25b1ce0 "ctdb_tcp_node_connect"
2012/10/08 20:26:42.137890 [11383]: Destroying timer event 0x25b5970 "ctdb_tcp_node_connect"
2012/10/08 20:26:42.138087 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5970
2012/10/08 20:26:42.972151 [11383]: Added timed event "ctdb_statistics_update": 0x25b6a50
2012/10/08 20:26:42.972169 [11383]: Ending timer event 0x25ba0e0 "ctdb_statistics_update"
2012/10/08 20:26:43.129219 [11383]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:26:43.129233 [11383]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:26:43.129245 [11383]: Added timed event "ctdb_wait_until_recovered": 0x25b6c50
2012/10/08 20:26:43.129254 [11383]: Ending timer event 0x25b58c0 "ctdb_wait_until_recovered"
2012/10/08 20:26:43.129263 [11383]: Added timed event "ctdb_time_tick": 0x25b58c0
2012/10/08 20:26:43.129272 [11383]: Ending timer event 0x25b2160 "ctdb_time_tick"
2012/10/08 20:26:43.138330 [11383]: tcp/tcp_connect.c:161 Created TCP SOCKET FD:11
2012/10/08 20:26:43.138365 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5f40
2012/10/08 20:26:43.138377 [11383]: Ending timer event 0x25b5970 "ctdb_tcp_node_connect"
2012/10/08 20:26:43.138648 [11383]: Destroying timer event 0x25b5f40 "ctdb_tcp_node_connect"
2012/10/08 20:26:43.138671 [11383]: Added timed event "ctdb_tcp_node_connect": 0x25b5f40
2012/10/08 20:26:43.972745 [11383]: Added timed event "ctdb_statistics_update": 0x25b60d0
2012/10/08 20:26:43.972782 [11383]: Ending timer event 0x25b6a50 "ctdb_statistics_update"
2012/10/08 20:26:44.022024 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:11 to connected child
2012/10/08 20:26:44.022165 [11383]: server/ctdb_daemon.c:628 client request 2147483448 of type 7 length 64 from node 0 to 4026531841
2012/10/08 20:26:44.022187 [11383]: Added timed event "queue_next_trigger": 0x25b75d0
2012/10/08 20:26:44.022198 [11383]: Added timed event "ctdb_control_timeout": 0x25b76f0
2012/10/08 20:26:44.022210 [11383]: server/ctdb_server.c:331 ctdb request 102 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:44.022259 [11383]: Added timed event "queue_next_trigger": 0x25b78a0
2012/10/08 20:26:44.022273 [11383]: Ending timer event 0x25b75d0 "queue_next_trigger"
2012/10/08 20:26:44.022283 [11383]: server/ctdb_server.c:331 ctdb request 102 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:44.022299 [11383]: Destroying timer event 0x25b76f0 "ctdb_control_timeout"
2012/10/08 20:26:44.022309 [11383]: Ending timer event 0x25b78a0 "queue_next_trigger"
2012/10/08 20:26:44.022386 [11383]: server/ctdb_daemon.c:666 Created SOCKET FD:12 to connected child
2012/10/08 20:26:44.022444 [11383]: server/ctdb_daemon.c:628 client request 2147483449 of type 7 length 64 from node 1 to 4026531841
2012/10/08 20:26:44.022459 [11383]: Added timed event "queue_next_trigger": 0x25b7b70
2012/10/08 20:26:44.022469 [11383]: Added timed event "ctdb_control_timeout": 0x25b7c90
2012/10/08 20:26:44.022479 [11383]: server/ctdb_server.c:331 ctdb request 104 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:44.022490 [11383]: Added timed event "queue_next_trigger": 0x25b7ed0
2012/10/08 20:26:44.022516 [11383]: Ending timer event 0x25b7b70 "queue_next_trigger"
2012/10/08 20:26:44.022526 [11383]: server/ctdb_server.c:331 ctdb request 104 of type 8 length 48 from node 1 to 1
2012/10/08 20:26:44.022540 [11383]: Destroying timer event 0x25b7c90 "ctdb_control_timeout"
2012/10/08 20:26:44.022550 [11383]: Ending timer event 0x25b7ed0 "queue_next_trigger"
2012/10/08 20:26:44.022599 [11383]: server/ctdb_daemon.c:628 client request 2147483450 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:44.022614 [11383]: Added timed event "queue_next_trigger": 0x25b7db0
2012/10/08 20:26:44.022636 [11383]: Added timed event "ctdb_control_timeout": 0x25b7ed0
2012/10/08 20:26:44.022647 [11383]: server/ctdb_server.c:331 ctdb request 105 of type 7 length 64 from node 1 to 1
2012/10/08 20:26:44.022657 [11383]: Shutting down recovery daemon
2012/10/08 20:26:44.022671 [11383]: Destroying timer event 0x25b64f0 "ctdb_check_for_dead_nodes"
2012/10/08 20:26:44.022681 [11383]: Destroying timer event 0x25b6c50 "ctdb_wait_until_recovered"
2012/10/08 20:26:44.022690 [11383]: Monitoring has been stopped
2012/10/08 20:26:44.022718 [11383]: server/ctdb_takeover.c:163 public address '10.100.55.198' now unassigned (old iface '__none__' refs[0])
2012/10/08 20:26:44.022743 [11383]: server/eventscript.c:754 Starting eventscript shutdown
2012/10/08 20:26:44.022832 [11383]: Event script /etc/ctdb/events.d/60.ganesha is not executable. Ignoring this event script
2012/10/08 20:26:44.022868 [11383]: Event script /etc/ctdb/events.d/31.clamd is not executable. Ignoring this event script
2012/10/08 20:26:44.022883 [11383]: Event script /etc/ctdb/events.d/40.fs_use is not executable. Ignoring this event script
2012/10/08 20:26:44.022902 [11383]: Event script /etc/ctdb/events.d/20.multipathd is not executable. Ignoring this event script
2012/10/08 20:26:44.023415 [11383]: server/eventscript.c:414 Created PIPE FD:10 to child eventscript process
2012/10/08 20:26:44.023521 [11383]: Added timed event "ctdb_event_script_timeout": 0x25ade00
2012/10/08 20:26:44.024268 [11383]: Destroying timer event 0x25b7ed0 "ctdb_control_timeout"
2012/10/08 20:26:44.024564 [eventscript-00.ctdb-shutdown:12501]: Executing event script /etc/ctdb/events.d/00.ctdb shutdown
2012/10/08 20:26:44.029289 [11383]: SIGCHLD from 11454
-------------- next part --------------
2012/10/08 20:22:38.517166 [29613]: CTDB starting on node
2012/10/08 20:22:38.520871 [29614]: Starting CTDBD as pid : 29614
2012/10/08 20:22:38.521006 [29614]: Set scheduler to SCHED_FIFO
2012/10/08 20:22:38.676221 [29614]: Keepalive monitoring has been started
2012/10/08 20:22:38.676261 [29614]: Monitoring has been started
2012/10/08 20:22:38.680760 [recoverd:29685]: monitor_cluster starting
2012/10/08 20:22:38.683018 [recoverd:29685]: server/ctdb_recoverd.c:3029 Initial recovery master set - forcing election
2012/10/08 20:22:38.683088 [29614]: Freeze priority 1
2012/10/08 20:22:38.686092 [29614]: Freeze priority 2
2012/10/08 20:22:38.686149 [29614]: Freeze priority 3
2012/10/08 20:22:39.677026 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:40.677542 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:41.677988 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:41.689480 [recoverd:29685]: server/ctdb_recoverd.c:913 Election timed out
2012/10/08 20:22:41.689942 [recoverd:29685]: The interfaces status has changed on local node 0 - force takeover run
2012/10/08 20:22:41.690129 [recoverd:29685]: server/ctdb_recoverd.c:2344 Node:0 was in recovery mode. Restart recovery process
2012/10/08 20:22:41.690152 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:22:41.690163 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:22:41.690182 [recoverd:29685]: Take the recovery lock
2012/10/08 20:22:41.695592 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:22:41.695606 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:22:41.695658 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:22:41.695671 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:22:41.695726 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:22:41.695745 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:22:41.695818 [29614]: Freeze priority 1
2012/10/08 20:22:41.695870 [29614]: Freeze priority 2
2012/10/08 20:22:41.695921 [29614]: Freeze priority 3
2012/10/08 20:22:41.696009 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:22:41.838825 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:22:41.839110 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:22:41.839125 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:22:41.839175 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:22:41.839286 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:22:41.839338 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:22:41.839462 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:22:41.839494 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:22:41.839519 [29614]: Thawing priority 1
2012/10/08 20:22:41.839528 [29614]: Release freeze handler for prio 1
2012/10/08 20:22:41.839557 [29614]: Thawing priority 2
2012/10/08 20:22:41.839567 [29614]: Release freeze handler for prio 2
2012/10/08 20:22:41.839581 [29614]: Thawing priority 3
2012/10/08 20:22:41.839590 [29614]: Release freeze handler for prio 3
2012/10/08 20:22:41.845059 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:22:41.845225 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:22:41.845329 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:22:41.987494 [29614]: Recovery has finished
2012/10/08 20:22:42.114204 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:22:42.114243 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:22:42.114252 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:22:42.114271 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:22:42.678551 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:42.678582 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:43.679312 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:43.679337 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:44.679965 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:44.679987 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:45.680437 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:45.680462 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:46.680970 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:46.680990 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:47.681444 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:47.681463 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:48.681950 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:48.681981 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:49.682445 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:49.682513 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:50.683071 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:50.683099 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:51.683574 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:51.683592 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:52.124379 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:22:52.124852 [recoverd:29685]: The interfaces status has changed on local node 0 - force takeover run
2012/10/08 20:22:52.124993 [recoverd:29685]: Trigger takeoverrun
2012/10/08 20:22:52.126977 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:22:52.127036 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:22:52.128565 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:22:52.128598 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:22:52.128611 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:22:52.128621 [recoverd:29685]: Take the recovery lock
2012/10/08 20:22:52.134321 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:22:52.134336 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:22:52.134403 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:22:52.134415 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:22:52.134468 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:22:52.134481 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:22:52.134552 [29614]: Freeze priority 1
2012/10/08 20:22:52.135449 [29614]: Freeze priority 2
2012/10/08 20:22:52.136396 [29614]: Freeze priority 3
2012/10/08 20:22:52.137307 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:22:52.137382 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:22:52.282746 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:22:52.283031 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:22:52.283057 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:22:52.283109 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:22:52.283198 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:22:52.283249 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:22:52.283372 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:22:52.283404 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:22:52.283429 [29614]: Thawing priority 1
2012/10/08 20:22:52.283439 [29614]: Release freeze handler for prio 1
2012/10/08 20:22:52.283468 [29614]: Thawing priority 2
2012/10/08 20:22:52.283478 [29614]: Release freeze handler for prio 2
2012/10/08 20:22:52.283491 [29614]: Thawing priority 3
2012/10/08 20:22:52.283501 [29614]: Release freeze handler for prio 3
2012/10/08 20:22:52.288835 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:22:52.288945 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:22:52.289043 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:22:52.431196 [29614]: Recovery has finished
2012/10/08 20:22:52.545592 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:22:52.545630 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:22:52.545639 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:22:52.545647 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:22:52.683829 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:52.683857 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:53.684443 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:53.684461 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:54.684900 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:54.684916 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:55.685411 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:55.685431 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:56.685851 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:56.685871 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:57.686352 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:57.686369 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:58.686899 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:58.686936 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:22:59.687497 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:22:59.687517 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:00.688174 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:00.688192 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:01.688628 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:01.688647 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:02.555714 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:23:02.558438 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:23:02.558503 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:23:02.559939 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:23:02.559954 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:23:02.559970 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:23:02.559980 [recoverd:29685]: Take the recovery lock
2012/10/08 20:23:02.565527 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:23:02.565542 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:23:02.565608 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:23:02.565620 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:23:02.565673 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:23:02.565686 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:23:02.565771 [29614]: Freeze priority 1
2012/10/08 20:23:02.566734 [29614]: Freeze priority 2
2012/10/08 20:23:02.567695 [29614]: Freeze priority 3
2012/10/08 20:23:02.568633 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:23:02.568724 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:23:02.689540 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:02.689608 [29614]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/08 20:23:02.714447 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:23:02.714726 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:23:02.714741 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:23:02.714791 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:23:02.714894 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:23:02.714946 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:23:02.715079 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:23:02.715112 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:23:02.715136 [29614]: Thawing priority 1
2012/10/08 20:23:02.715145 [29614]: Release freeze handler for prio 1
2012/10/08 20:23:02.715174 [29614]: Thawing priority 2
2012/10/08 20:23:02.715184 [29614]: Release freeze handler for prio 2
2012/10/08 20:23:02.715198 [29614]: Thawing priority 3
2012/10/08 20:23:02.715208 [29614]: Release freeze handler for prio 3
2012/10/08 20:23:02.720589 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:23:02.720698 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:23:02.720791 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:23:02.828093 [29614]: Recovery has finished
2012/10/08 20:23:02.935249 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:23:02.935288 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:23:02.935297 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:23:02.935305 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:23:03.690629 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:03.690659 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:04.691120 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:04.691138 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:05.691782 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:05.691798 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:06.692253 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:06.692270 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:07.692724 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:07.692755 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:08.693436 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:08.693473 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:09.693982 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:09.694051 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:10.694627 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:10.694648 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:11.695338 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:11.695356 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:12.695807 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:12.695846 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:12.945364 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:23:12.947833 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:23:12.947914 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:23:12.949382 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:23:12.949397 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:23:12.949407 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:23:12.949416 [recoverd:29685]: Take the recovery lock
2012/10/08 20:23:12.954856 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:23:12.954871 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:23:12.954939 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:23:12.954952 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:23:12.955005 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:23:12.955018 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:23:12.955103 [29614]: Freeze priority 1
2012/10/08 20:23:12.956013 [29614]: Freeze priority 2
2012/10/08 20:23:12.956967 [29614]: Freeze priority 3
2012/10/08 20:23:12.957917 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:23:12.958035 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:23:13.102670 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:23:13.102954 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:23:13.102970 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:23:13.103020 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:23:13.103124 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:23:13.103176 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:23:13.103299 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:23:13.103331 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:23:13.103356 [29614]: Thawing priority 1
2012/10/08 20:23:13.103366 [29614]: Release freeze handler for prio 1
2012/10/08 20:23:13.103395 [29614]: Thawing priority 2
2012/10/08 20:23:13.103406 [29614]: Release freeze handler for prio 2
2012/10/08 20:23:13.103420 [29614]: Thawing priority 3
2012/10/08 20:23:13.103430 [29614]: Release freeze handler for prio 3
2012/10/08 20:23:13.108863 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:23:13.108973 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:23:13.109086 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:23:13.251314 [29614]: Recovery has finished
2012/10/08 20:23:13.358125 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:23:13.358163 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:23:13.358172 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:23:13.358180 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:23:13.696507 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:13.696538 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:14.697215 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:14.697235 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:15.697757 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:15.697775 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:16.698414 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:16.698431 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:17.698977 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:17.698994 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:18.699567 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:18.699617 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:19.700169 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:19.700189 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:20.700794 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:20.700819 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:21.701349 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:21.701367 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:22.702141 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:22.702187 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:23.368243 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:23:23.371013 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:23:23.371080 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:23:23.372513 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:23:23.372528 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:23:23.372537 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:23:23.372546 [recoverd:29685]: Take the recovery lock
2012/10/08 20:23:23.378345 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:23:23.378361 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:23:23.378436 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:23:23.378448 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:23:23.378501 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:23:23.378514 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:23:23.378585 [29614]: Freeze priority 1
2012/10/08 20:23:23.379486 [29614]: Freeze priority 2
2012/10/08 20:23:23.380450 [29614]: Freeze priority 3
2012/10/08 20:23:23.381390 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:23:23.381463 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:23:23.522653 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:23:23.522875 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:23:23.522886 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:23:23.522924 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:23:23.523002 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:23:23.523049 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:23:23.523141 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:23:23.523165 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:23:23.523183 [29614]: Thawing priority 1
2012/10/08 20:23:23.523190 [29614]: Release freeze handler for prio 1
2012/10/08 20:23:23.523212 [29614]: Thawing priority 2
2012/10/08 20:23:23.523220 [29614]: Release freeze handler for prio 2
2012/10/08 20:23:23.523232 [29614]: Thawing priority 3
2012/10/08 20:23:23.523239 [29614]: Release freeze handler for prio 3
2012/10/08 20:23:23.528162 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:23:23.528250 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:23:23.528327 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:23:23.635379 [29614]: Recovery has finished
2012/10/08 20:23:23.702921 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:23.702977 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:23.774334 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:23:23.774384 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:23:23.774396 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:23:23.774406 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:23:24.703756 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:24.703786 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:25.704363 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:25.704386 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:26.704854 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:26.704873 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:27.705317 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:27.705333 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:28.705847 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:28.705881 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:29.706374 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:29.706431 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:30.706960 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:30.706978 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:31.707580 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:31.707599 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:32.708034 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:32.708054 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:33.708533 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:33.708551 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:33.784476 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:23:33.787133 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:23:33.787199 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:23:33.788628 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:23:33.788642 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:23:33.788652 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:23:33.788661 [recoverd:29685]: Take the recovery lock
2012/10/08 20:23:33.794164 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:23:33.794179 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:23:33.794245 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:23:33.794258 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:23:33.794311 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:23:33.794324 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:23:33.794408 [29614]: Freeze priority 1
2012/10/08 20:23:33.795335 [29614]: Freeze priority 2
2012/10/08 20:23:33.796292 [29614]: Freeze priority 3
2012/10/08 20:23:33.797246 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:23:33.797319 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:23:33.943105 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:23:33.943392 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:23:33.943406 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:23:33.943457 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:23:33.943560 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:23:33.943611 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:23:33.943733 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:23:33.943765 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:23:33.943789 [29614]: Thawing priority 1
2012/10/08 20:23:33.943798 [29614]: Release freeze handler for prio 1
2012/10/08 20:23:33.943828 [29614]: Thawing priority 2
2012/10/08 20:23:33.943838 [29614]: Release freeze handler for prio 2
2012/10/08 20:23:33.943851 [29614]: Thawing priority 3
2012/10/08 20:23:33.943861 [29614]: Release freeze handler for prio 3
2012/10/08 20:23:33.949583 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:23:33.949691 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:23:33.949791 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:23:34.092902 [29614]: Recovery has finished
2012/10/08 20:23:34.220261 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:23:34.220301 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:23:34.220310 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:23:34.220318 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:23:34.709598 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:34.709626 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:35.710316 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:35.710335 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:36.710819 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:36.710853 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:37.711536 [29614]: CTDB_WAIT_UNTIL_RECOVERED
2012/10/08 20:23:37.711556 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:38.712408 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:39.712864 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:40.713397 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:41.713872 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:42.714372 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:43.715075 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:44.230344 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:23:44.232841 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:23:44.232907 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:23:44.234418 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:23:44.234433 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:23:44.234443 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:23:44.234452 [recoverd:29685]: Take the recovery lock
2012/10/08 20:23:44.240062 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:23:44.240078 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:23:44.240158 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:23:44.240171 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:23:44.240224 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:23:44.240237 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:23:44.240338 [29614]: Freeze priority 1
2012/10/08 20:23:44.241242 [29614]: Freeze priority 2
2012/10/08 20:23:44.242185 [29614]: Freeze priority 3
2012/10/08 20:23:44.243114 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:23:44.243225 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:23:44.388725 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:23:44.389013 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:23:44.389028 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:23:44.389078 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:23:44.389181 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:23:44.389241 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:23:44.389364 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:23:44.389396 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:23:44.389420 [29614]: Thawing priority 1
2012/10/08 20:23:44.389430 [29614]: Release freeze handler for prio 1
2012/10/08 20:23:44.389459 [29614]: Thawing priority 2
2012/10/08 20:23:44.389469 [29614]: Release freeze handler for prio 2
2012/10/08 20:23:44.389482 [29614]: Thawing priority 3
2012/10/08 20:23:44.389492 [29614]: Release freeze handler for prio 3
2012/10/08 20:23:44.394767 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:23:44.394875 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:23:44.394975 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:23:44.536759 [29614]: Recovery has finished
2012/10/08 20:23:44.651951 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:23:44.651992 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:23:44.652002 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:23:44.652009 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:23:44.715195 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:45.715837 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:46.716535 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:47.716987 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:48.717489 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:49.718022 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:50.718642 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:51.719141 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:52.719585 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:53.720070 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:54.662119 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:23:54.664880 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:23:54.664947 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:23:54.666356 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:23:54.666373 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:23:54.666383 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:23:54.666393 [recoverd:29685]: Take the recovery lock
2012/10/08 20:23:54.671996 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:23:54.672011 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:23:54.672080 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:23:54.672093 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:23:54.672146 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:23:54.672160 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:23:54.672244 [29614]: Freeze priority 1
2012/10/08 20:23:54.673140 [29614]: Freeze priority 2
2012/10/08 20:23:54.674118 [29614]: Freeze priority 3
2012/10/08 20:23:54.675057 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:23:54.675132 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:23:54.720598 [29614]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/08 20:23:54.822610 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:23:54.822934 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:23:54.822950 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:23:54.823006 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:23:54.823111 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:23:54.823163 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:23:54.823285 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:23:54.823337 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:23:54.823364 [29614]: Thawing priority 1
2012/10/08 20:23:54.823374 [29614]: Release freeze handler for prio 1
2012/10/08 20:23:54.823406 [29614]: Thawing priority 2
2012/10/08 20:23:54.823416 [29614]: Release freeze handler for prio 2
2012/10/08 20:23:54.823431 [29614]: Thawing priority 3
2012/10/08 20:23:54.823440 [29614]: Release freeze handler for prio 3
2012/10/08 20:23:54.828973 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:23:54.829099 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:23:54.829206 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:23:54.940576 [29614]: Recovery has finished
2012/10/08 20:23:55.049200 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:23:55.049259 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:23:55.049268 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:23:55.049276 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:23:55.721641 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:56.722299 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:57.722970 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:58.723985 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:23:59.724500 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:00.725123 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:01.725761 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:02.726233 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:03.726835 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:04.727501 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:05.050062 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:24:05.052743 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:24:05.052809 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:24:05.054193 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:24:05.054209 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:24:05.054218 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:24:05.054227 [recoverd:29685]: Take the recovery lock
2012/10/08 20:24:05.059828 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:24:05.059843 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:24:05.059924 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:24:05.059937 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:24:05.059990 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:24:05.060003 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:24:05.060098 [29614]: Freeze priority 1
2012/10/08 20:24:05.060996 [29614]: Freeze priority 2
2012/10/08 20:24:05.061933 [29614]: Freeze priority 3
2012/10/08 20:24:05.062884 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:24:05.062990 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:24:05.207528 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:24:05.207812 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:24:05.207827 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:24:05.207877 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:24:05.207979 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:24:05.208031 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:24:05.208153 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:24:05.208185 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:24:05.208209 [29614]: Thawing priority 1
2012/10/08 20:24:05.208243 [29614]: Release freeze handler for prio 1
2012/10/08 20:24:05.208273 [29614]: Thawing priority 2
2012/10/08 20:24:05.208283 [29614]: Release freeze handler for prio 2
2012/10/08 20:24:05.208296 [29614]: Thawing priority 3
2012/10/08 20:24:05.208306 [29614]: Release freeze handler for prio 3
2012/10/08 20:24:05.213611 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:24:05.213719 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:24:05.213818 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:24:05.356936 [29614]: Recovery has finished
2012/10/08 20:24:05.464200 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:24:05.464246 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:24:05.464255 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:24:05.464263 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:24:05.728546 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:06.729356 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:07.729788 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:08.730415 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:09.730900 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:10.731851 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:11.732521 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:12.733135 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:13.733603 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:14.734040 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:15.474310 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:24:15.476959 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:24:15.477023 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:24:15.478432 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:24:15.478448 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:24:15.478458 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:24:15.478466 [recoverd:29685]: Take the recovery lock
2012/10/08 20:24:15.484044 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:24:15.484059 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:24:15.484132 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:24:15.484145 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:24:15.484197 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:24:15.484210 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:24:15.484311 [29614]: Freeze priority 1
2012/10/08 20:24:15.485587 [29614]: Freeze priority 2
2012/10/08 20:24:15.486558 [29614]: Freeze priority 3
2012/10/08 20:24:15.487496 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:24:15.487570 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:24:15.630305 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:24:15.630523 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:24:15.630534 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:24:15.630572 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:24:15.630651 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:24:15.630690 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:24:15.630783 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:24:15.630807 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:24:15.630826 [29614]: Thawing priority 1
2012/10/08 20:24:15.630832 [29614]: Release freeze handler for prio 1
2012/10/08 20:24:15.630855 [29614]: Thawing priority 2
2012/10/08 20:24:15.630863 [29614]: Release freeze handler for prio 2
2012/10/08 20:24:15.630873 [29614]: Thawing priority 3
2012/10/08 20:24:15.630880 [29614]: Release freeze handler for prio 3
2012/10/08 20:24:15.635821 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:24:15.635905 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:24:15.635994 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:24:15.734714 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:15.743065 [29614]: Recovery has finished
2012/10/08 20:24:15.849164 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:24:15.849203 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:24:15.849212 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:24:15.849220 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:24:16.735512 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:17.736030 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:18.736628 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:19.737094 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:20.737579 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:21.738043 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:22.738523 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:23.739219 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:24.739703 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:25.740364 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:25.859284 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:24:25.861893 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:24:25.861964 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:24:25.863483 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:24:25.863498 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:24:25.863508 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:24:25.863516 [recoverd:29685]: Take the recovery lock
2012/10/08 20:24:25.869110 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:24:25.869126 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:24:25.869192 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:24:25.869204 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:24:25.869257 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:24:25.869270 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:24:25.869386 [29614]: Freeze priority 1
2012/10/08 20:24:25.870326 [29614]: Freeze priority 2
2012/10/08 20:24:25.871283 [29614]: Freeze priority 3
2012/10/08 20:24:25.872186 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:24:25.872261 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:24:26.017476 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:24:26.017712 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:24:26.017724 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:24:26.017761 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:24:26.017840 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:24:26.017881 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:24:26.017973 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:24:26.017997 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:24:26.018046 [29614]: Thawing priority 1
2012/10/08 20:24:26.018054 [29614]: Release freeze handler for prio 1
2012/10/08 20:24:26.018067 [29614]: Thawing priority 2
2012/10/08 20:24:26.018074 [29614]: Release freeze handler for prio 2
2012/10/08 20:24:26.018084 [29614]: Thawing priority 3
2012/10/08 20:24:26.018091 [29614]: Release freeze handler for prio 3
2012/10/08 20:24:26.023041 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:24:26.023125 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:24:26.023201 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:24:26.130054 [29614]: Recovery has finished
2012/10/08 20:24:26.265795 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:24:26.265843 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:24:26.265855 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:24:26.265865 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:24:26.741227 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:27.741767 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:28.742275 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:29.742785 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:30.743282 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:31.743959 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:32.744448 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:33.744963 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:34.745426 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:35.745934 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:36.275931 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:24:36.278656 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:24:36.278724 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:24:36.280211 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:24:36.280226 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:24:36.280236 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:24:36.280245 [recoverd:29685]: Take the recovery lock
2012/10/08 20:24:36.285922 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:24:36.285937 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:24:36.286028 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:24:36.286041 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:24:36.286095 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:24:36.286108 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:24:36.286211 [29614]: Freeze priority 1
2012/10/08 20:24:36.287109 [29614]: Freeze priority 2
2012/10/08 20:24:36.288053 [29614]: Freeze priority 3
2012/10/08 20:24:36.289004 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:24:36.289079 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:24:36.433411 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:24:36.433692 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:24:36.433707 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:24:36.433757 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:24:36.433861 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:24:36.433913 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:24:36.434064 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:24:36.434097 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:24:36.434123 [29614]: Thawing priority 1
2012/10/08 20:24:36.434132 [29614]: Release freeze handler for prio 1
2012/10/08 20:24:36.434161 [29614]: Thawing priority 2
2012/10/08 20:24:36.434171 [29614]: Release freeze handler for prio 2
2012/10/08 20:24:36.434185 [29614]: Thawing priority 3
2012/10/08 20:24:36.434194 [29614]: Release freeze handler for prio 3
2012/10/08 20:24:36.439467 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:24:36.439577 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:24:36.439690 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:24:36.582057 [29614]: Recovery has finished
2012/10/08 20:24:36.713097 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:24:36.713144 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:24:36.713153 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:24:36.713161 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:24:36.746339 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:37.746864 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:38.747709 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:39.748242 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:40.748717 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:41.749171 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:42.749635 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:43.750135 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:44.750647 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:45.751400 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:46.723207 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:24:46.725745 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:24:46.725808 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:24:46.727248 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:24:46.727263 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:24:46.727273 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:24:46.727281 [recoverd:29685]: Take the recovery lock
2012/10/08 20:24:46.732892 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:24:46.732908 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:24:46.732984 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:24:46.732996 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:24:46.733049 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:24:46.733062 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:24:46.733173 [29614]: Freeze priority 1
2012/10/08 20:24:46.734077 [29614]: Freeze priority 2
2012/10/08 20:24:46.734987 [29614]: Freeze priority 3
2012/10/08 20:24:46.735906 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:24:46.735979 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:24:46.751629 [29614]: server/ctdb_monitor.c:246 in recovery. Wait one more second
2012/10/08 20:24:46.880773 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:24:46.881054 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:24:46.881069 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:24:46.881119 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:24:46.881222 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:24:46.881273 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:24:46.881394 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:24:46.881426 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:24:46.881462 [29614]: Thawing priority 1
2012/10/08 20:24:46.881472 [29614]: Release freeze handler for prio 1
2012/10/08 20:24:46.881502 [29614]: Thawing priority 2
2012/10/08 20:24:46.881512 [29614]: Release freeze handler for prio 2
2012/10/08 20:24:46.881525 [29614]: Thawing priority 3
2012/10/08 20:24:46.881534 [29614]: Release freeze handler for prio 3
2012/10/08 20:24:46.886870 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:24:46.886977 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:24:46.887072 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:24:47.029536 [29614]: Recovery has finished
2012/10/08 20:24:47.155002 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:24:47.155071 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:24:47.155084 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:24:47.155094 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:24:47.752448 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:48.753038 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:49.753501 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:50.754031 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:51.754501 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:52.754941 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:53.755701 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:54.756477 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:55.756950 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:56.757429 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:57.165153 [recoverd:29685]: The rerecovery timeout has elapsed. We now allow recoveries to trigger again.
2012/10/08 20:24:57.167774 [recoverd:29685]: server/ctdb_recoverd.c:2739 reclock child process returned error 5
2012/10/08 20:24:57.167840 [recoverd:29685]: server/ctdb_recoverd.c:2839 reclock child failed when checking file
2012/10/08 20:24:57.169273 [recoverd:29685]: Failed check_recovery_lock. Force a recovery
2012/10/08 20:24:57.169288 [recoverd:29685]: server/ctdb_recoverd.c:1364 Starting do_recovery
2012/10/08 20:24:57.169297 [recoverd:29685]: Taking out recovery lock from recovery daemon
2012/10/08 20:24:57.169306 [recoverd:29685]: Take the recovery lock
2012/10/08 20:24:57.174917 [recoverd:29685]: Recovery lock taken successfully
2012/10/08 20:24:57.174931 [recoverd:29685]: ctdb_recovery_lock: Got recovery lock on '/mnt/gluster/ctdb/lock'
2012/10/08 20:24:57.174999 [recoverd:29685]: Recovery lock taken successfully by recovery daemon
2012/10/08 20:24:57.175011 [recoverd:29685]: server/ctdb_recoverd.c:1401 Recovery initiated due to problem with node 0
2012/10/08 20:24:57.175063 [recoverd:29685]: server/ctdb_recoverd.c:1426 Recovery - created remote databases
2012/10/08 20:24:57.175076 [recoverd:29685]: server/ctdb_recoverd.c:1433 Recovery - updated db priority for all databases
2012/10/08 20:24:57.175160 [29614]: Freeze priority 1
2012/10/08 20:24:57.176049 [29614]: Freeze priority 2
2012/10/08 20:24:57.177009 [29614]: Freeze priority 3
2012/10/08 20:24:57.177943 [29614]: server/ctdb_recover.c:724 Recovery mode set to ACTIVE
2012/10/08 20:24:57.178041 [29614]: server/ctdb_recover.c:1080 startrecovery eventscript has been invoked
2012/10/08 20:24:57.322767 [recoverd:29685]: server/ctdb_recoverd.c:1470 Recovery - updated flags
2012/10/08 20:24:57.323054 [recoverd:29685]: server/ctdb_recoverd.c:1514 started transactions on all nodes
2012/10/08 20:24:57.323069 [recoverd:29685]: server/ctdb_recoverd.c:1527 Recovery - starting database commits
2012/10/08 20:24:57.323120 [recoverd:29685]: server/ctdb_recoverd.c:1539 Recovery - committed databases
2012/10/08 20:24:57.323224 [recoverd:29685]: server/ctdb_recoverd.c:1589 Recovery - updated vnnmap
2012/10/08 20:24:57.323275 [recoverd:29685]: server/ctdb_recoverd.c:1598 Recovery - updated recmaster
2012/10/08 20:24:57.323399 [recoverd:29685]: server/ctdb_recoverd.c:1615 Recovery - updated flags
2012/10/08 20:24:57.323431 [29614]: server/ctdb_recover.c:724 Recovery mode set to NORMAL
2012/10/08 20:24:57.323455 [29614]: Thawing priority 1
2012/10/08 20:24:57.323465 [29614]: Release freeze handler for prio 1
2012/10/08 20:24:57.323494 [29614]: Thawing priority 2
2012/10/08 20:24:57.323504 [29614]: Release freeze handler for prio 2
2012/10/08 20:24:57.323517 [29614]: Thawing priority 3
2012/10/08 20:24:57.323527 [29614]: Release freeze handler for prio 3
2012/10/08 20:24:57.328926 [recoverd:29685]: server/ctdb_recoverd.c:1624 Recovery - disabled recovery mode
2012/10/08 20:24:57.329037 [recoverd:29685]: Failed to find node to cover ip 10.100.55.198
2012/10/08 20:24:57.329161 [29614]: Forced running of eventscripts with arguments ipreallocated
2012/10/08 20:24:57.471796 [29614]: Recovery has finished
2012/10/08 20:24:57.577986 [recoverd:29685]: server/ctdb_recoverd.c:1650 Recovery - finished the recovered event
2012/10/08 20:24:57.578053 [recoverd:29685]: server/ctdb_recoverd.c:1656 Recovery complete
2012/10/08 20:24:57.578063 [recoverd:29685]: Resetting ban count to 0 for all nodes
2012/10/08 20:24:57.578071 [recoverd:29685]: Just finished a recovery. New recoveries will now be supressed for the rerecovery timeout (10 seconds)
2012/10/08 20:24:57.758349 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:58.758941 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:24:59.759972 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:00.760435 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:01.760903 [29614]: server/ctdb_monitor.c:257 wait for pending recoveries to end. Wait one more second.
2012/10/08 20:25:02.607918 [29614]: Shutting down recovery daemon
2012/10/08 20:25:02.607959 [29614]: Monitoring has been stopped
More information about the samba-technical
mailing list