RAFT and CTDB
Min Wai Chan
dcmwai at gmail.com
Tue Nov 25 11:40:01 MST 2014
Dear All,
What fail...
Both CTDB will start non-stop recovery...
When there is only one node, it is still working
but not on both node...
System info
Linux amtbsrv03 3.14.14-gentoo #2 SMP
Kernel mode
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_OCFS2_FS=y
CONFIG_OCFS2_FS_O2CB=y
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
o2cb 1.8.2
o2info -C --volinfo /dev/sdb1
Label: amtbcluster
UUID: F3BB7490B1044BBEA5BE93282D2E9D6B
Block Size: 4096
Cluster Size: 4096
Node Slots: 2
Features: backup-super strict-journal-super sparse extended-slotmap
Features: inline-data xattr indexed-dirs refcount discontig-bg unwritten
/etc/fstab
none /sys/kernel/config configfs defaults
0 0
none /sys/kernel/dlm ocfs2_dlmfs defaults
0 0
/dev/sdb1 /amtb ocfs2
_netdev,nointr,user_xattr,acl 0 0
/etc/samba/smb.conf only ctdb related.
[global]
workgroup = WORKGROUP
security = ADS
realm = REALS
idmap config * : backend = tdb
idmap config * : range = 60001-70000
idmap config WORKGROUP : backend = ad
idmap config WORKGROUP : schema_mode = rfc2307
idmap config WORKGROUP : range = 500-50000
winbind nss info = rfc2307
winbind enum groups = yes
winbind enum users = yes
winbind use default domain = Yes
winbind refresh tickets = yes
winbind offline logon = yes
winbind nested groups = yes
winbind max clients = 500
winbind cache time = 300
idmap cache time = 300
idmap negative cache time = 300
#Guest OK
guest account = nobody
#CTDB Cluster
netbios name = CTDBCluster
clustering = yes
ctdbd socket = /var/lib/ctdb/ctdb.socket
# idmap backend = tdb2
#Disable Printer
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
LOG Level = 2 winbind:1
max log size = 20000
wins support = Yes
map acl inherit = Yes
hide unreadable = Yes
dont descend = /proc,/dev,/etc,/lib,/lost+found,/initrd
store dos attributes = Yes
vfs object = recycle acl_xattr fileid
#ocfs2 ctdb
fileid:mapping = fsid
#extd_audit
recycle:keeptree = yes
recycle:versions = yes
recycle:repository = .recycle
recycle:touch_mtime = yes
recycle:exclude = *.tmp, *.bak
recycle:exclude_dir = .recycle
CTDB Setting (Same both server)
/etc/ctdb.ctdbd.conf
CTDB_PIDFILE=/var/run/ctdb/ctdbd.pid
CTDB_BASE=/etc/ctdb
CTDB_VARDIR=/var/lib/ctdb
CTDB_DEBUGLEVEL=2
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_RECOVERY_LOCK=/amtb/ctdb.lock
CTDB_SOCKET=/var/lib/ctdb/ctdb.socke
CTDB_MANAGES_SAMBA=yes
CTDB_SERVICE_SMB=samba
CTDB_MONITOR_FREE_MEMORY=10
/etc/ctdb/nodes
192.168.41.30
192.168.41.34
192.168.41.38
/etc/ctdb/public_addresses
192.168.11.29/24 enp2s3
192.168.11.30/24 enp2s3
192.168.11.34/24 enp2s3
ping_pong test result
Running on one node
ping_pong -rw test.dat 2
data increment = 1
298358 locks/sec
Running on both node (about 155 <= 300)
ping_pong -rw test.dat 2
data increment = 1
213 locks/sec
Some explains of the event
2014/11/26 02:15:44.401308 --> Node 1 start
2014/11/26 02:18:20.782510 --> Node 2 Join (Recovery none stop)
2014/11/26 02:19:19.445524 --> Node 2 leave
Node 1 Shutdown after that
The Log
Node_1
2014/11/26 02:15:44.401308 [ 9691]: CTDB starting on node
2014/11/26 02:15:44.403395 [ 9692]: Starting CTDBD (Version 2.5.4) as PID:
9692
2014/11/26 02:15:44.403609 [ 9692]: Created PID file /var/run/ctdb/ctdbd.pid
2014/11/26 02:15:44.403661 [ 9692]: Set real-time scheduler priority
2014/11/26 02:15:44.403836 [ 9692]: Set runstate to INIT (1)
2014/11/26 02:15:45.341683 [ 9692]: Vacuuming is disabled for persistent
database account_policy.tdb
2014/11/26 02:15:45.341737 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/account_policy.tdb.0' with flags 0x400
2014/11/26 02:15:45.357192 [ 9692]: Vacuuming is disabled for persistent
database group_mapping.tdb
2014/11/26 02:15:45.357227 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/group_mapping.tdb.0' with flags 0x400
2014/11/26 02:15:45.374333 [ 9692]: Vacuuming is disabled for persistent
database secrets.tdb
2014/11/26 02:15:45.374369 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/secrets.tdb.0' with flags 0x400
2014/11/26 02:15:45.389092 [ 9692]: Vacuuming is disabled for persistent
database registry.tdb
2014/11/26 02:15:45.389123 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/registry.tdb.0' with flags 0x400
2014/11/26 02:15:45.403709 [ 9692]: Vacuuming is disabled for persistent
database ctdb.tdb
2014/11/26 02:15:45.403745 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/ctdb.tdb.0' with flags 0x400
2014/11/26 02:15:45.418421 [ 9692]: Vacuuming is disabled for persistent
database winbindd_idmap.tdb
2014/11/26 02:15:45.418454 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/winbindd_idmap.tdb.0' with flags 0x400
2014/11/26 02:15:45.433020 [ 9692]: Vacuuming is disabled for persistent
database share_info.tdb
2014/11/26 02:15:45.433051 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/share_info.tdb.0' with flags 0x400
2014/11/26 02:15:45.447742 [ 9692]: Vacuuming is disabled for persistent
database passdb.tdb
2014/11/26 02:15:45.447773 [ 9692]: Attached to database
'/var/lib/ctdb/persistent/passdb.tdb.0' with flags 0x400
2014/11/26 02:15:45.447817 [ 9692]: Freeze priority 1
2014/11/26 02:15:45.449545 [ 9692]: Freeze priority 2
2014/11/26 02:15:45.450673 [ 9692]: Freeze priority 3
2014/11/26 02:15:45.451882 [ 9692]: server/ctdb_takeover.c:3303 Released 0
public IPs
2014/11/26 02:15:45.451914 [ 9692]: Set runstate to SETUP (2)
2014/11/26 02:15:45.763927 [ 9692]: Keepalive monitoring has been started
2014/11/26 02:15:45.764001 [ 9692]: Set runstate to FIRST_RECOVERY (3)
2014/11/26 02:15:45.764268 [recoverd: 9883]: monitor_cluster starting
2014/11/26 02:15:45.766605 [recoverd: 9883]: server/ctdb_recoverd.c:3727
Initial recovery master set - forcing election
2014/11/26 02:15:45.766785 [ 9692]: Freeze priority 1
2014/11/26 02:15:45.766905 [ 9692]: Freeze priority 2
2014/11/26 02:15:45.767014 [ 9692]: Freeze priority 3
2014/11/26 02:15:45.767213 [ 9692]: This node (0) is now the recovery master
2014/11/26 02:15:46.764674 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:47.766066 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:48.766966 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:48.770465 [recoverd: 9883]: Election period ended
2014/11/26 02:15:48.771151 [recoverd: 9883]: Initial interface fetched
2014/11/26 02:15:48.771208 [recoverd: 9883]: The interfaces status has
changed on local node 0 - force takeover run
2014/11/26 02:15:48.771378 [recoverd: 9883]: Node:0 was in recovery mode.
Start recovery process
2014/11/26 02:15:48.771426 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:15:48.771448 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:15:48.771482 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:15:48.775593 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:15:48.775671 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:15:48.775763 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:15:48.775788 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:15:48.775885 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:15:48.776581 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:15:48.776748 [ 9692]: Freeze priority 1
2014/11/26 02:15:48.776839 [ 9692]: Freeze priority 2
2014/11/26 02:15:48.776920 [ 9692]: Freeze priority 3
2014/11/26 02:15:48.777008 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:15:49.086286 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:15:49.093323 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:15:49.094321 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:15:49.094475 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:15:49.095135 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:15:49.103460 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:15:49.105479 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:15:49.105646 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:15:49.106231 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:15:49.114606 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:15:49.116351 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:15:49.116486 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:15:49.117186 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:15:49.125157 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:15:49.126852 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:15:49.126989 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:15:49.127568 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:15:49.135366 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:15:49.137205 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:15:49.137340 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:15:49.138464 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:15:49.146457 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:15:49.148285 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:15:49.148417 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:15:49.149073 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:15:49.157249 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:15:49.158936 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:15:49.159070 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:15:49.159665 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:15:49.167498 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:15:49.169342 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:15:49.169479 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:15:49.170222 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:15:49.178159 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:15:49.178436 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:15:49.906318 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:49.906385 [ 9692]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:15:49.906522 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:15:49.906769 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:15:49.906906 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:15:49.907020 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:15:49.907051 [ 9692]: Thawing priority 1
2014/11/26 02:15:49.907068 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:15:49.907130 [ 9692]: Thawing priority 2
2014/11/26 02:15:49.907151 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:15:49.907187 [ 9692]: Thawing priority 3
2014/11/26 02:15:49.907205 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:15:49.908754 [recoverd: 9883]: server/ctdb_recoverd.c:2048
Recovery - disabled recovery mode
2014/11/26 02:15:49.908951 [recoverd: 9883]: Takeover run starting
2014/11/26 02:15:49.909324 [recoverd: 9883]: Failed to find node to cover
ip 192.168.11.34
2014/11/26 02:15:49.909401 [recoverd: 9883]: Failed to find node to cover
ip 192.168.11.30
2014/11/26 02:15:49.909449 [recoverd: 9883]: Failed to find node to cover
ip 192.168.11.29
2014/11/26 02:15:50.216774 [recoverd: 9883]: Takeover run completed
successfully
2014/11/26 02:15:50.216942 [ 9692]: Recovery has finished
2014/11/26 02:15:50.522535 [ 9692]: Set runstate to STARTUP (4)
2014/11/26 02:15:50.522776 [recoverd: 9883]: server/ctdb_recoverd.c:2068
Recovery - finished the recovered event
2014/11/26 02:15:50.523105 [recoverd: 9883]: server/ctdb_recoverd.c:2079
Recovery complete
2014/11/26 02:15:50.523171 [recoverd: 9883]: Resetting ban count to 0 for
all nodes
2014/11/26 02:15:50.523192 [recoverd: 9883]: Just finished a recovery. New
recoveries will now be supressed for the rerecovery timeout (10 seconds)
2014/11/26 02:15:50.907061 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:50.907146 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:51.907774 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:51.907853 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:52.908437 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:52.908524 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:53.909089 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:53.909169 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:54.909765 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:54.909850 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:55.910495 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:55.910574 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:56.911254 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:56.911340 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:57.911931 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:57.912010 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:58.912612 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:58.912694 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:15:59.913318 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:15:59.913399 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:16:00.533291 [recoverd: 9883]: The rerecovery timeout has
elapsed. We now allow recoveries to trigger again.
2014/11/26 02:16:00.914383 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:16:00.914468 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:16:01.914551 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:16:01.914634 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:16:02.914740 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:16:02.914828 [ 9692]: server/ctdb_monitor.c:304 wait for
pending recoveries to end. Wait one more second.
2014/11/26 02:16:03.915891 [ 9692]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:16:03.916016 [ 9692]: ctdb_recheck_presistent_health: OK[8]
FAIL[0]
2014/11/26 02:16:03.916042 [ 9692]: Running the "startup" event.
2014/11/26 02:16:04.008600 [ 9692]: iface[enp2s3] has changed it's link
status down => up
2014/11/26 02:16:04.208043 [ 9692]: Attached to database
'/var/lib/ctdb/serverid.tdb.0' with flags 0xc41
2014/11/26 02:16:04.218474 [ 9692]: Starting traverse on DB serverid.tdb
(id 61)
2014/11/26 02:16:04.220966 [ 9692]: Ending traverse on DB serverid.tdb (id
61), records 0
2014/11/26 02:16:04.306564 [ 9692]: Attached to database
'/var/lib/ctdb/g_lock.tdb.0' with flags 0xc41
2014/11/26 02:16:04.314687 [ 9692]: 50.samba: * samba -> start: winbind
... [ ok ]
2014/11/26 02:16:04.349307 [ 9692]: Attached to database
'/var/lib/ctdb/dbwrap_watchers.tdb.0' with flags 0xc41
2014/11/26 02:16:04.372973 [ 9692]: Attached to database
'/var/lib/ctdb/smbXsrv_version_global.tdb.0' with flags 0xc41
2014/11/26 02:16:04.375143 [ 9692]: 50.samba: * samba -> start: smbd ... [
ok ]
2014/11/26 02:16:04.378813 [ 9692]: Attached to database
'/var/lib/ctdb/smbXsrv_session_global.tdb.0' with flags 0xc41
2014/11/26 02:16:04.383612 [ 9692]: Attached to database
'/var/lib/ctdb/smbXsrv_tcon_global.tdb.0' with flags 0xc41
2014/11/26 02:16:04.388657 [ 9692]: Attached to database
'/var/lib/ctdb/brlock.tdb.0' with flags 0xc41
2014/11/26 02:16:04.393982 [ 9692]: Attached to database
'/var/lib/ctdb/locking.tdb.0' with flags 0xc41
2014/11/26 02:16:04.398885 [ 9692]: Attached to database
'/var/lib/ctdb/notify_index.tdb.0' with flags 0xc41
2014/11/26 02:16:04.416990 [ 9692]: 50.samba: * samba -> start: nmbd ... [
ok ]
2014/11/26 02:16:04.514505 [ 9692]: Attached to database
'/var/lib/ctdb/smbXsrv_open_global.tdb.0' with flags 0xc41
2014/11/26 02:16:04.538713 [recoverd: 9883]: Interface enp2s3 changed
state: 0 => 1
2014/11/26 02:16:04.538766 [recoverd: 9883]: The interfaces status has
changed on local node 0 - force takeover run
2014/11/26 02:16:04.539047 [recoverd: 9883]: Trigger takeoverrun
2014/11/26 02:16:04.540122 [recoverd: 9883]: Takeover run starting
2014/11/26 02:16:04.540393 [recoverd: 9883]: Failed to find node to cover
ip 192.168.11.34
2014/11/26 02:16:04.540427 [recoverd: 9883]: Failed to find node to cover
ip 192.168.11.30
2014/11/26 02:16:04.540471 [recoverd: 9883]: Failed to find node to cover
ip 192.168.11.29
2014/11/26 02:16:04.564837 [ 9692]: startup event OK - enabling monitoring
2014/11/26 02:16:04.564885 [ 9692]: Set runstate to RUNNING (5)
2014/11/26 02:16:04.848553 [recoverd: 9883]: Takeover run completed
successfully
2014/11/26 02:16:06.948530 [ 9692]: 50.samba: ERROR: samba directory
"/home/samba/profiles" not available
2014/11/26 02:16:09.295232 [ 9692]: 50.samba: ERROR: samba directory
"/home/samba/profiles" not available
2014/11/26 02:16:13.698389 [ 9692]: 50.samba: ERROR: samba directory
"/home/samba/profiles" not available
2014/11/26 02:16:22.043172 [ 9692]: 50.samba: ERROR: samba directory
"/home/samba/profiles" not available
2014/11/26 02:16:37.535719 [ 9692]: monitor event OK - node re-enabled
2014/11/26 02:16:37.535802 [ 9692]: Node became HEALTHY. Ask recovery
master 0 to perform ip reallocation
2014/11/26 02:16:37.536380 [recoverd: 9883]: Node 0 has changed flags - now
0x0 was 0x2
2014/11/26 02:16:37.583370 [recoverd: 9883]: Public IP '192.168.11.34' is
not assigned and we could serve it
2014/11/26 02:16:37.583438 [recoverd: 9883]: Public IP '192.168.11.30' is
not assigned and we could serve it
2014/11/26 02:16:37.583460 [recoverd: 9883]: Public IP '192.168.11.29' is
not assigned and we could serve it
2014/11/26 02:16:37.583709 [recoverd: 9883]: Trigger takeoverrun
2014/11/26 02:16:37.584867 [recoverd: 9883]: Takeover run starting
2014/11/26 02:16:37.585296 [ 9692]: Takeover of IP 192.168.11.34/24 on
interface enp2s3
2014/11/26 02:16:37.585781 [ 9692]: Takeover of IP 192.168.11.30/24 on
interface enp2s3
2014/11/26 02:16:37.586206 [ 9692]: Takeover of IP 192.168.11.29/24 on
interface enp2s3
2014/11/26 02:16:38.364238 [recoverd: 9883]: Takeover run completed
successfully
2014/11/26 02:18:20.782510 [ 9692]: 192.168.41.30:4379: connected to
192.168.41.34:4379 - 1 connected
2014/11/26 02:18:21.099157 [recoverd: 9883]: Election period ended
2014/11/26 02:18:24.604847 [recoverd: 9883]: Election period ended
2014/11/26 02:18:24.713661 [recoverd: 9883]: Node:1 was in recovery mode.
Start recovery process
2014/11/26 02:18:24.713714 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:24.713737 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:24.713754 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:24.713809 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:24.713828 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:24.714052 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:24.714089 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:24.715061 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:24.717314 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:24.917697 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:24.918517 [ 9692]: Freeze priority 1
2014/11/26 02:18:24.920857 [ 9692]: Freeze priority 2
2014/11/26 02:18:24.922639 [ 9692]: Freeze priority 3
2014/11/26 02:18:24.924424 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:25.241952 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:25.268589 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:25.270964 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:25.286483 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:25.289210 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:25.304669 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:25.307240 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:25.323006 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:25.325679 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:25.343029 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:25.345763 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:25.360949 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:25.363564 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:25.378964 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:25.381801 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:25.397452 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:25.400198 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:25.415662 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:25.418277 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:25.433887 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:25.436731 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:25.452372 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:25.454037 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:25.454617 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:25.455248 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:25.470918 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:25.472627 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:25.473193 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:25.473787 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:25.489555 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:25.491330 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:25.491895 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:25.492610 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:25.501727 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:25.503464 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:25.504042 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:25.504683 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:25.513636 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:25.515476 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:25.516023 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:25.517144 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:25.535688 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:25.537625 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:25.538285 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:25.538984 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:25.548052 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:25.549779 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:25.550309 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:25.550982 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:25.559929 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:25.562177 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:25.562723 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:25.563462 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:25.572292 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:25.572600 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:25.916373 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:25.920201 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:25.920692 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:25.920776 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:25.920815 [ 9692]: Thawing priority 1
2014/11/26 02:18:25.920834 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:25.920903 [ 9692]: Thawing priority 2
2014/11/26 02:18:25.920932 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:25.920972 [ 9692]: Thawing priority 3
2014/11/26 02:18:25.920990 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:26.363173 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:26.363257 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:26.363292 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:26.363315 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:26.363334 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:26.363351 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:26.365254 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:26.365294 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:26.365312 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:26.365374 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:26.365391 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:26.365635 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:26.365675 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:26.366634 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:26.368746 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:26.369372 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:26.369949 [ 9692]: Freeze priority 1
2014/11/26 02:18:26.372769 [ 9692]: Freeze priority 2
2014/11/26 02:18:26.374896 [ 9692]: Freeze priority 3
2014/11/26 02:18:26.376968 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:26.690332 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:26.705250 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:26.707479 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:26.716498 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:26.719451 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:26.728197 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:26.730798 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:26.739747 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:26.742372 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:26.751372 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:26.754068 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:26.763182 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:26.765991 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:26.775118 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:26.778100 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:26.787016 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:26.789768 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:26.798750 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:26.801605 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:26.810627 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:26.813645 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:26.822603 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:26.824301 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:26.824825 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:26.825477 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:26.834532 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:26.836250 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:26.836773 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:26.837429 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:26.846194 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:26.847880 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:26.848522 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:26.849222 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:26.858144 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:26.859946 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:26.860462 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:26.861088 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:26.869887 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:26.871561 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:26.872072 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:26.873171 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:26.883066 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:26.884752 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:26.885345 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:26.886089 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:26.895047 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:26.896735 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:26.897248 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:26.897904 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:26.906872 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:26.908539 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:26.909050 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:26.909791 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:26.919011 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:26.919298 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:27.258041 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:27.260853 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:27.261319 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:27.261408 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:27.261448 [ 9692]: Thawing priority 1
2014/11/26 02:18:27.261468 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:27.261521 [ 9692]: Thawing priority 2
2014/11/26 02:18:27.261541 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:27.261580 [ 9692]: Thawing priority 3
2014/11/26 02:18:27.261599 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:27.265705 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:27.265762 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:27.265787 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:27.265807 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:27.265829 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:27.265848 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:27.366106 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:27.366187 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:27.366210 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:27.366260 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:27.366278 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:27.366364 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:27.366391 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:27.367254 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:27.369285 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:27.369851 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:27.370387 [ 9692]: Freeze priority 1
2014/11/26 02:18:27.373039 [ 9692]: Freeze priority 2
2014/11/26 02:18:27.375176 [ 9692]: Freeze priority 3
2014/11/26 02:18:27.377400 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:27.734169 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:27.751010 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:27.753184 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:27.762234 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:27.765276 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:27.774260 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:27.777325 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:27.787879 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:27.790704 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:27.799740 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:27.802466 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:27.811337 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:27.814035 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:27.823071 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:27.826112 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:27.835076 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:27.837942 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:27.847088 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:27.849822 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:27.859581 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:27.862675 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:27.871649 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:27.873455 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:27.874067 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:27.874814 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:27.884002 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:27.885800 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:27.887218 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:27.887939 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:27.896970 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:27.898748 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:27.899385 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:27.900187 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:27.909144 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:27.910883 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:27.911475 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:27.912214 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:27.921038 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:27.922790 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:27.923428 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:27.924628 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:27.935676 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:27.937477 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:27.938164 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:27.938947 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:27.947857 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:27.949605 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:27.950213 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:27.950968 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:27.959948 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:27.961679 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:27.962284 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:27.963126 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:27.972126 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:27.972502 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:28.306203 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:28.308961 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:28.309473 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:28.309559 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:28.309598 [ 9692]: Thawing priority 1
2014/11/26 02:18:28.309618 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:28.309675 [ 9692]: Thawing priority 2
2014/11/26 02:18:28.309696 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:28.309747 [ 9692]: Thawing priority 3
2014/11/26 02:18:28.309775 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:28.312799 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:28.312858 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:28.312887 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:28.312906 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:28.312929 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:28.312948 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:28.366910 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:28.366971 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:28.366993 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:28.367038 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:28.367063 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:28.367143 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:28.367170 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:28.368040 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:28.370049 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:28.370591 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:28.371075 [ 9692]: Freeze priority 1
2014/11/26 02:18:28.373776 [ 9692]: Freeze priority 2
2014/11/26 02:18:28.376003 [ 9692]: Freeze priority 3
2014/11/26 02:18:28.378172 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:28.696196 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:28.711514 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:28.713578 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:28.729023 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:28.732528 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:28.747731 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:28.750332 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:28.765705 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:28.768428 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:28.783921 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:28.786578 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:28.802026 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:28.804603 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:28.820104 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:28.823149 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:28.832072 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:28.835031 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:28.844970 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:28.847672 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:28.856689 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:28.859637 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:28.868448 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:28.870217 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:28.870782 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:28.871434 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:28.880468 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:28.882188 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:28.882735 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:28.883372 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:28.892518 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:28.894216 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:28.894748 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:28.895458 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:28.904575 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:28.906343 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:28.906897 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:28.907521 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:28.916473 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:28.918173 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:28.918699 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:28.919927 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:28.929664 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:28.931379 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:28.932001 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:28.932710 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:28.941700 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:28.943427 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:28.944013 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:28.944683 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:28.953647 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:28.955386 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:28.955910 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:28.956656 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:28.965777 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:28.966111 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:29.306155 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:29.308591 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:29.309204 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:29.309457 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:29.309509 [ 9692]: Thawing priority 1
2014/11/26 02:18:29.309530 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:29.309589 [ 9692]: Thawing priority 2
2014/11/26 02:18:29.309611 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:29.309663 [ 9692]: Thawing priority 3
2014/11/26 02:18:29.309684 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:29.311718 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:29.311783 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:29.311811 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:29.311837 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:29.311881 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:29.311907 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:29.368551 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:29.368609 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:29.368629 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:29.368676 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:29.368700 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:29.368814 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:29.368850 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:29.369767 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:29.372003 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:29.372602 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:29.373173 [ 9692]: Freeze priority 1
2014/11/26 02:18:29.375779 [ 9692]: Freeze priority 2
2014/11/26 02:18:29.377992 [ 9692]: Freeze priority 3
2014/11/26 02:18:29.380134 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:29.849159 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:29.864981 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:29.867198 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:29.882871 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:29.885667 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:29.900990 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:29.903650 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:29.919201 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:29.921855 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:29.937522 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:29.940117 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:29.955691 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:29.958249 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:29.973699 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:29.976650 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:29.991862 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:29.994550 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:30.010280 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:30.013122 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:30.022173 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:30.025274 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:30.034186 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:30.036005 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:30.036662 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:30.037427 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:30.046606 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:30.048367 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:30.049017 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:30.049742 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:30.058721 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:30.060441 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:30.061263 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:30.062098 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:30.071455 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:30.073278 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:30.073896 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:30.074620 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:30.083476 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:30.085182 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:30.085800 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:30.087002 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:30.096760 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:30.098490 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:30.099182 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:30.099964 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:30.108892 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:30.110659 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:30.111285 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:30.112088 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:30.121040 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:30.122703 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:30.123228 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:30.124469 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:30.134369 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:30.134720 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:30.466196 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:30.468931 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:30.469482 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:30.469569 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:30.469627 [ 9692]: Thawing priority 1
2014/11/26 02:18:30.469648 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:30.469716 [ 9692]: Thawing priority 2
2014/11/26 02:18:30.469744 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:30.469820 [ 9692]: Thawing priority 3
2014/11/26 02:18:30.469851 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:30.472639 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:30.472695 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:30.472723 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:30.472742 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:30.472765 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:30.472783 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:30.474669 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:30.474720 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:30.474744 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:30.474795 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:30.474815 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:30.474891 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:30.474931 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:30.475843 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:30.477863 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:30.478396 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:30.478903 [ 9692]: Freeze priority 1
2014/11/26 02:18:30.481458 [ 9692]: Freeze priority 2
2014/11/26 02:18:30.483545 [ 9692]: Freeze priority 3
2014/11/26 02:18:30.485573 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:30.798742 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:30.814169 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:30.816348 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:30.831837 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:30.834542 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:30.849475 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:30.852125 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:30.867631 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:30.870173 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:30.885650 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:30.888211 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:30.903534 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:30.906076 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:30.921773 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:30.924639 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:30.940942 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:30.943858 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:30.952887 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:30.955630 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:30.964434 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:30.967450 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:30.976400 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:30.978223 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:30.978790 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:30.979433 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:30.988583 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:30.990349 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:30.990878 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:30.991521 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:31.001031 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:31.002764 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:31.003351 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:31.004065 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:31.013212 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:31.014860 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:31.015440 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:31.016103 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:31.024946 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:31.026616 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:31.027180 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:31.028285 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:31.038037 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:31.039812 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:31.040429 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:31.041215 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:31.050399 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:31.052061 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:31.052609 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:31.053423 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:31.062470 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:31.064219 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:31.064791 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:31.065556 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:31.074582 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:31.074991 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:31.409962 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:31.412588 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:31.413084 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:31.413221 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:31.413272 [ 9692]: Thawing priority 1
2014/11/26 02:18:31.413292 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:31.413362 [ 9692]: Thawing priority 2
2014/11/26 02:18:31.413386 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:31.413424 [ 9692]: Thawing priority 3
2014/11/26 02:18:31.413443 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:31.415469 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:31.415523 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:31.415550 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:31.415576 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:31.415595 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:31.415658 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:31.476254 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:31.476308 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:31.476329 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:31.476376 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:31.476401 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:31.476524 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:31.476562 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:31.477589 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:31.479685 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:31.480302 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:31.480873 [ 9692]: Freeze priority 1
2014/11/26 02:18:31.483378 [ 9692]: Freeze priority 2
2014/11/26 02:18:31.485441 [ 9692]: Freeze priority 3
2014/11/26 02:18:31.487594 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:31.805893 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:31.821727 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:31.824037 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:31.839531 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:31.842261 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:31.857554 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:31.860132 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:31.875644 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:31.878131 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:31.893517 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:31.896871 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:31.911904 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:31.914420 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:31.930039 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:31.932968 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:31.948331 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:31.951062 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:31.966698 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:31.969172 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:31.984677 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:31.987577 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:32.003122 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:32.004741 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:32.005314 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:32.005975 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:32.022081 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:32.023735 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:32.024348 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:32.024955 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:32.040491 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:32.042367 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:32.042977 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:32.043720 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:32.052897 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:32.054623 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:32.055166 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:32.055838 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:32.064675 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:32.066541 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:32.068760 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:32.069867 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:32.079511 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:32.081317 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:32.081945 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:32.082660 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:32.091720 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:32.093467 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:32.094049 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:32.094727 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:32.103789 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:32.105445 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:32.105980 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:32.106736 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:32.115719 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:32.116120 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:32.466234 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:32.468984 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:32.469688 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:32.469982 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:32.470035 [ 9692]: Thawing priority 1
2014/11/26 02:18:32.470058 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:32.470128 [ 9692]: Thawing priority 2
2014/11/26 02:18:32.470157 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:32.470195 [ 9692]: Thawing priority 3
2014/11/26 02:18:32.470214 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:32.472086 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:32.472293 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:32.472484 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:32.472667 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:32.472849 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:32.473025 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:32.476560 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:32.476683 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:32.476771 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:32.476883 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:32.476972 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:32.477156 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:32.477260 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:32.478301 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:32.482810 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:32.483587 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:32.484023 [ 9692]: Freeze priority 1
2014/11/26 02:18:32.486305 [ 9692]: Freeze priority 2
2014/11/26 02:18:32.488236 [ 9692]: Freeze priority 3
2014/11/26 02:18:32.490072 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:32.805099 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:32.820740 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:32.823000 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:32.839139 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:32.842016 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:32.857367 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:32.859958 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:32.876328 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:32.878902 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:32.894144 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:32.896695 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:32.911872 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:32.914361 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:32.929871 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:32.932915 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:32.941732 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:32.944847 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:32.953849 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:32.956538 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:32.965442 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:32.968443 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:32.977343 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:32.978998 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:32.979527 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:32.980168 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:32.989345 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:32.991004 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:32.991538 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:32.992187 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:33.001439 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:33.003425 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:33.003992 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:33.004714 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:33.013673 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:33.015330 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:33.015896 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:33.016534 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:33.025312 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:33.027086 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:33.027610 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:33.028721 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:33.038568 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:33.040366 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:33.041038 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:33.041749 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:33.050654 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:33.052418 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:33.052950 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:33.053620 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:33.062508 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:33.064240 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:33.064757 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:33.065508 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:33.074524 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:33.074902 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:33.399890 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:33.401327 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:33.401899 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:33.402047 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:33.402105 [ 9692]: Thawing priority 1
2014/11/26 02:18:33.402126 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:33.402191 [ 9692]: Thawing priority 2
2014/11/26 02:18:33.402221 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:33.402261 [ 9692]: Thawing priority 3
2014/11/26 02:18:33.402285 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:33.404154 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:33.404211 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:33.404239 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:33.404265 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:33.404284 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:33.404301 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:33.477164 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:33.477219 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:33.477240 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:33.477286 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:33.477311 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:33.477420 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:33.477507 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:33.478828 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:33.483943 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:33.484504 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:33.484919 [ 9692]: Freeze priority 1
2014/11/26 02:18:33.487317 [ 9692]: Freeze priority 2
2014/11/26 02:18:33.489296 [ 9692]: Freeze priority 3
2014/11/26 02:18:33.491292 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:33.805985 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:33.821538 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:33.823577 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:33.839023 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:33.841861 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:33.857280 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:33.861798 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:33.878213 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:33.881048 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:33.889875 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:33.892642 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:33.901674 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:33.904462 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:33.914159 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:33.917112 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:33.925967 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:33.928728 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:33.937741 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:33.940315 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:33.949164 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:33.952199 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:33.961130 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:33.962806 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:33.963343 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:33.964001 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:33.973148 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:33.974830 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:33.975381 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:33.976035 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:33.984878 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:33.986598 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:33.987120 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:33.987822 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:33.996925 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:33.998884 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:33.999428 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:34.000047 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:34.008726 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:34.010393 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:34.010938 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:34.012054 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:34.021821 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:34.023480 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:34.024100 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:34.024775 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:34.034066 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:34.035807 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:34.036636 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:34.037289 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:34.046198 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:34.047835 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:34.048373 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:34.049116 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:34.057981 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:34.058255 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:34.397712 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:34.398899 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:34.399448 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:34.399587 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:34.399637 [ 9692]: Thawing priority 1
2014/11/26 02:18:34.399658 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:34.399719 [ 9692]: Thawing priority 2
2014/11/26 02:18:34.399741 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:34.399779 [ 9692]: Thawing priority 3
2014/11/26 02:18:34.399798 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:34.401726 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:34.401782 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:34.401812 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:34.401839 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:34.401858 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:34.401899 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:34.477756 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:34.477815 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:34.477836 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:34.477880 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:34.477906 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:34.478017 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:34.478054 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:34.479083 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:34.483932 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:34.484531 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:34.484971 [ 9692]: Freeze priority 1
2014/11/26 02:18:34.487381 [ 9692]: Freeze priority 2
2014/11/26 02:18:34.489432 [ 9692]: Freeze priority 3
2014/11/26 02:18:34.491289 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:34.822445 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:34.838135 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:34.840385 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:34.855890 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:34.858568 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:34.874115 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:34.876744 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:34.885773 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:34.888330 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:34.903775 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:34.906301 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:34.921692 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:34.924269 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:34.939805 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:34.942584 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:34.958025 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:34.960604 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:34.976080 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:34.978625 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:34.994142 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:34.997080 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:35.012896 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:35.014573 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:35.015121 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:35.015795 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:35.031501 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:35.033202 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:35.034568 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:35.035168 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:35.050665 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:35.052323 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:35.052863 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:35.053532 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:35.069177 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:35.071066 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:35.071746 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:35.072544 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:35.081470 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:35.083343 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:35.083989 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:35.085224 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:35.095115 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:35.097011 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:35.097926 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:35.098739 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:35.107888 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:35.109660 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:35.110336 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:35.111120 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:35.120178 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:35.121939 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:35.122576 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:35.123456 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:35.132652 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:35.133062 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:35.468130 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:35.470900 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:35.471539 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:35.471817 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:35.471867 [ 9692]: Thawing priority 1
2014/11/26 02:18:35.471906 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:35.471976 [ 9692]: Thawing priority 2
2014/11/26 02:18:35.472000 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:35.472039 [ 9692]: Thawing priority 3
2014/11/26 02:18:35.472057 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:35.475464 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:35.475519 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:35.475547 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:35.475573 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:35.475592 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:35.475640 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:35.478103 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:35.478150 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:35.478170 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:35.478215 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:35.478239 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:35.478348 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:35.478385 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:35.479340 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:35.483322 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:35.483895 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:35.484334 [ 9692]: Freeze priority 1
2014/11/26 02:18:35.486731 [ 9692]: Freeze priority 2
2014/11/26 02:18:35.488755 [ 9692]: Freeze priority 3
2014/11/26 02:18:35.490700 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:35.804820 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:35.813010 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:35.815218 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:35.824186 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:35.827049 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:35.835764 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:35.838471 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:35.853922 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:35.856537 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:35.872292 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:35.874860 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:35.890345 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:35.892941 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:35.908603 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:35.911497 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:35.927037 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:35.929713 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:35.945270 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:35.947800 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:35.962877 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:35.965682 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:35.974542 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:35.976316 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:35.976855 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:35.977551 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:35.986546 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:35.988180 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:35.988687 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:35.989315 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:36.004561 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:36.006322 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:36.006898 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:36.007578 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:36.023431 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:36.025143 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:36.025766 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:36.026376 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:36.041961 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:36.043611 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:36.044198 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:36.045264 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:36.061575 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:36.063224 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:36.063826 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:36.064480 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:36.080055 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:36.081918 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:36.082440 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:36.083121 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:36.091837 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:36.093469 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:36.093974 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:36.094725 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:36.103554 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:36.103861 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:36.436339 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:36.440747 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:36.441533 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:36.441830 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:36.441881 [ 9692]: Thawing priority 1
2014/11/26 02:18:36.441902 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:36.441971 [ 9692]: Thawing priority 2
2014/11/26 02:18:36.441995 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:36.442034 [ 9692]: Thawing priority 3
2014/11/26 02:18:36.442053 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:36.443885 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:36.443944 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:36.443972 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:36.443998 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:36.444017 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:36.444034 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:36.478830 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:36.478884 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:36.478904 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:36.478950 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:36.478974 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:36.479094 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:36.479129 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:36.480139 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:36.484385 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:36.485008 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:36.485507 [ 9692]: Freeze priority 1
2014/11/26 02:18:36.487818 [ 9692]: Freeze priority 2
2014/11/26 02:18:36.489730 [ 9692]: Freeze priority 3
2014/11/26 02:18:36.491540 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:36.805766 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:36.814138 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:36.816480 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:36.825501 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:36.828403 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:36.926126 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:36.928936 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:36.944598 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:36.947380 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:36.962673 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:36.965324 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:36.980575 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:36.983203 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:36.998670 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:37.002059 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:37.011236 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:37.013917 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:37.029326 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:37.031864 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:37.047265 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:37.050074 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:37.065408 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:37.067288 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:37.067970 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:37.068756 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:37.077972 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:37.079763 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:37.080414 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:37.081172 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:37.090144 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:37.091979 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:37.092605 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:37.093429 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:37.102567 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:37.104325 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:37.104978 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:37.105771 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:37.114607 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:37.116402 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:37.117040 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:37.118264 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:37.128052 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:37.129796 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:37.130411 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:37.131066 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:37.146502 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:37.148212 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:37.148789 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:37.149413 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:37.165061 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:37.166738 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:37.167318 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:37.168042 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:37.183449 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:37.183743 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:37.838254 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:37.839372 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:37.839881 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:37.839953 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:37.839984 [ 9692]: Thawing priority 1
2014/11/26 02:18:37.840001 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:37.840063 [ 9692]: Thawing priority 2
2014/11/26 02:18:37.840085 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:37.840129 [ 9692]: Thawing priority 3
2014/11/26 02:18:37.840149 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:37.844649 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:37.844702 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:37.844728 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:37.844747 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:37.844770 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:37.844788 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:37.846715 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:37.846775 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:37.846796 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:37.846840 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:37.846865 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:37.846946 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:37.846986 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:37.847808 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:37.849846 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:37.850394 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:37.850860 [ 9692]: Freeze priority 1
2014/11/26 02:18:37.853365 [ 9692]: Freeze priority 2
2014/11/26 02:18:37.855393 [ 9692]: Freeze priority 3
2014/11/26 02:18:37.857599 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:38.173542 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:38.188795 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:38.190895 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:38.207002 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:38.209833 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:38.225146 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:38.227937 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:38.236742 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:38.239490 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:38.248463 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:38.251392 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:38.260047 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:38.262800 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:38.271674 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:38.274567 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:38.283228 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:38.285991 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:38.294976 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:38.297574 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:38.306466 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:38.309419 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:38.318228 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:38.319928 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:38.320480 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:38.321197 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:38.330143 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:38.331856 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:38.332403 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:38.333055 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:38.341923 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:38.343570 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:38.344107 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:38.344864 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:38.353799 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:38.355486 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:38.356026 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:38.356688 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:38.365386 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:38.367078 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:38.367620 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:38.368757 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:38.378639 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:38.380306 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:38.380950 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:38.381695 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:38.390583 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:38.392264 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:38.392786 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:38.393481 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:38.402487 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:38.404117 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:38.404683 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:38.405475 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:38.414422 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:38.414707 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:39.508200 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:39.509742 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:39.510380 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:39.510592 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:39.510633 [ 9692]: Thawing priority 1
2014/11/26 02:18:39.510652 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:39.510724 [ 9692]: Thawing priority 2
2014/11/26 02:18:39.510747 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:39.510797 [ 9692]: Thawing priority 3
2014/11/26 02:18:39.510819 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:39.517901 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:39.518094 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:39.518258 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:39.518410 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:39.518558 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:39.518704 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:39.520707 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:39.520823 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:39.520910 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:39.521024 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:39.521113 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:39.521259 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:39.521360 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:39.522309 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:39.525510 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:39.526198 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:39.526733 [ 9692]: Freeze priority 1
2014/11/26 02:18:39.529251 [ 9692]: Freeze priority 2
2014/11/26 02:18:39.531307 [ 9692]: Freeze priority 3
2014/11/26 02:18:39.533448 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:39.849611 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:39.865235 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:39.867516 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:39.876499 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:39.879398 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:39.888163 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:39.890794 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:39.899895 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:39.902550 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:39.911493 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:39.914088 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:39.923106 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:39.925764 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:39.934869 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:39.937851 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:39.946887 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:39.949618 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:39.958582 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:39.961201 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:39.970049 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:39.973029 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:39.981863 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:39.983813 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:39.984621 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:39.985430 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:39.994658 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:39.996438 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:39.996996 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:39.997722 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:40.006541 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:40.008310 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:40.008879 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:40.009655 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:40.018978 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:40.020751 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:40.021316 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:40.022048 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:40.030856 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:40.032593 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:40.033344 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:40.034644 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:40.044557 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:40.046471 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:40.107952 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:40.108919 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:40.118035 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:40.119974 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:40.120570 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:40.121331 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:40.130341 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:40.132048 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:40.132621 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:40.133449 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:40.142409 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:40.142734 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:40.467993 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:40.468983 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:40.469490 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:40.469579 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:40.469619 [ 9692]: Thawing priority 1
2014/11/26 02:18:40.469639 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:40.469705 [ 9692]: Thawing priority 2
2014/11/26 02:18:40.469734 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:40.469829 [ 9692]: Thawing priority 3
2014/11/26 02:18:40.469857 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:40.476770 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:40.476836 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:40.476864 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:40.476884 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:40.476902 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:40.476944 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:40.521222 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:40.521282 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:40.521303 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:40.521348 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:40.521373 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:40.521583 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:40.521630 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:40.522722 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:40.527332 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:40.528048 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:40.528596 [ 9692]: Freeze priority 1
2014/11/26 02:18:40.531215 [ 9692]: Freeze priority 2
2014/11/26 02:18:40.533442 [ 9692]: Freeze priority 3
2014/11/26 02:18:40.535770 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:40.850522 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:40.865840 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:40.867937 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:40.883553 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:40.886398 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:40.901962 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:40.904537 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:40.920416 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:40.923024 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:40.938456 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:40.941072 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:40.956485 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:40.959285 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:40.974930 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:40.977921 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:40.986575 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:40.989340 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:40.998253 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:41.000861 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:41.009722 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:41.012745 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:41.021539 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:41.023199 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:41.023741 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:41.024435 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:41.033512 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:41.035160 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:41.035724 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:41.036397 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:41.045210 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:41.046825 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:41.047361 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:41.048124 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:41.057245 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:41.058875 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:41.059432 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:41.060138 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:41.068924 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:41.070576 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:41.071110 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:41.072272 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:41.082058 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:41.083709 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:41.084349 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:41.085072 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:41.093955 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:41.095587 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:41.096186 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:41.096891 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:41.105717 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:41.107352 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:41.107898 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:41.108681 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:41.117425 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:41.117698 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:41.456235 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:41.458996 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:41.459503 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:41.459598 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:41.459630 [ 9692]: Thawing priority 1
2014/11/26 02:18:41.459656 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:41.459742 [ 9692]: Thawing priority 2
2014/11/26 02:18:41.459771 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:41.459870 [ 9692]: Thawing priority 3
2014/11/26 02:18:41.459901 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:41.465485 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:41.465548 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:41.465576 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:41.465596 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:41.465661 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:41.465687 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:41.521733 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:41.521801 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:41.521824 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:41.521870 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:41.521896 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:41.521977 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:41.522051 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:41.522909 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:41.526055 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:41.526613 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:41.527083 [ 9692]: Freeze priority 1
2014/11/26 02:18:41.529727 [ 9692]: Freeze priority 2
2014/11/26 02:18:41.532084 [ 9692]: Freeze priority 3
2014/11/26 02:18:41.534260 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:41.850941 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:41.866459 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:41.868554 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:41.884252 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:41.887070 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:41.902309 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:41.904916 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:41.920304 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:41.922999 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:41.931760 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:41.934516 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:41.943199 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:41.945840 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:41.954804 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:41.957745 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:41.966679 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:41.969441 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:41.978307 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:41.980865 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:41.989683 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:41.992629 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:42.001444 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:42.003118 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:42.003631 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:42.004417 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:42.013531 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:42.015233 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:42.015789 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:42.016462 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:42.025084 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:42.026751 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:42.027271 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:42.027999 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:42.036938 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:42.038598 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:42.039122 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:42.039772 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:42.048572 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:42.050231 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:42.050781 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:42.051927 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:42.061555 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:42.063227 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:42.063865 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:42.064575 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:42.073821 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:42.075500 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:42.076068 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:42.076745 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:42.085861 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:42.087534 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:42.088080 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:42.088832 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:42.097667 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:42.097948 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:42.437657 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:42.438864 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:42.439386 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:42.439510 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:42.439573 [ 9692]: Thawing priority 1
2014/11/26 02:18:42.439595 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:42.439656 [ 9692]: Thawing priority 2
2014/11/26 02:18:42.439678 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:42.439731 [ 9692]: Thawing priority 3
2014/11/26 02:18:42.439753 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:42.443052 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:42.443140 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:42.443198 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:42.443338 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:42.443421 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:42.443505 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:42.522620 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:42.522714 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:42.522765 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:42.522917 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:42.523028 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:42.523180 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:42.523277 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:42.524278 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:42.527442 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:42.528044 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:42.528535 [ 9692]: Freeze priority 1
2014/11/26 02:18:42.531198 [ 9692]: Freeze priority 2
2014/11/26 02:18:42.533355 [ 9692]: Freeze priority 3
2014/11/26 02:18:42.535453 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:42.852228 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:42.867918 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:42.870035 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:42.885705 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:42.888454 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:42.904428 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:42.907164 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:42.922780 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:42.925338 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:42.940740 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:42.943313 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:42.958808 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:42.961615 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:42.970837 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:42.974305 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:42.983255 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:42.986184 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:42.995284 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:42.998027 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:43.007116 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:43.010075 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:43.019006 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:43.020793 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:43.021411 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:43.022160 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:43.031220 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:43.032990 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:43.033622 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:43.034342 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:43.043131 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:43.044850 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:43.045478 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:43.046295 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:43.055254 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:43.056970 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:43.057569 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:43.058280 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:43.067220 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:43.068899 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:43.069463 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:43.070591 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:43.080336 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:43.082089 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:43.082785 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:43.083565 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:43.092604 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:43.094321 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:43.094901 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:43.095692 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:43.104605 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:43.106360 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:43.106944 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:43.107824 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:43.116775 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:43.117153 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:43.456253 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:43.457609 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:43.458210 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:43.458422 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:43.458467 [ 9692]: Thawing priority 1
2014/11/26 02:18:43.458486 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:43.458558 [ 9692]: Thawing priority 2
2014/11/26 02:18:43.458581 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:43.458632 [ 9692]: Thawing priority 3
2014/11/26 02:18:43.458654 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:43.464417 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:43.464600 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:43.464761 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:43.464912 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:43.465058 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:43.465203 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:43.523415 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:43.523591 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:43.523737 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:43.523910 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:43.524056 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:43.524262 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:43.524418 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:43.525583 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:43.528807 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:43.529502 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:43.530172 [ 9692]: Freeze priority 1
2014/11/26 02:18:43.532905 [ 9692]: Freeze priority 2
2014/11/26 02:18:43.535117 [ 9692]: Freeze priority 3
2014/11/26 02:18:43.537277 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:43.858239 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:43.874005 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:43.876289 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:43.885265 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:43.888219 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:43.903696 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:43.906429 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:43.922009 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:43.924638 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:43.940394 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:43.943128 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:43.958729 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:43.961442 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:43.977002 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:43.979942 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:43.995419 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:43.998304 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:44.014538 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:44.017254 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:44.032724 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:44.035705 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:44.051114 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:44.052871 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:44.053533 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:44.054272 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:44.070290 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:44.071990 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:44.072520 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:44.073140 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:44.082027 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:44.083806 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:44.084439 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:44.085271 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:44.094237 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:44.096022 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:44.096582 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:44.097229 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:44.106012 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:44.107702 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:44.108518 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:44.109717 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:44.119611 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:44.121405 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:44.122135 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:44.122977 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:44.132106 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:44.133845 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:44.134531 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:44.135289 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:44.144501 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:44.146312 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:44.147062 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:44.147959 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:44.156935 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:44.157319 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:44.497791 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:44.498897 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:44.499399 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:44.499488 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:44.499528 [ 9692]: Thawing priority 1
2014/11/26 02:18:44.499548 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:44.499637 [ 9692]: Thawing priority 2
2014/11/26 02:18:44.499660 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:44.499716 [ 9692]: Thawing priority 3
2014/11/26 02:18:44.499739 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:44.504389 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:44.504445 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:44.504482 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:44.504503 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:44.504521 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:44.504543 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:44.523494 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:44.523557 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:44.523578 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:44.523623 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:44.523648 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:44.523758 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:44.523815 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:44.718458 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:44.721713 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:44.722346 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:44.722898 [ 9692]: Freeze priority 1
2014/11/26 02:18:44.725698 [ 9692]: Freeze priority 2
2014/11/26 02:18:44.727912 [ 9692]: Freeze priority 3
2014/11/26 02:18:44.730082 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:45.045156 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:45.060824 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:45.063116 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:45.078383 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:45.081049 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:45.096288 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:45.098776 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:45.113881 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:45.116434 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:45.131707 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:45.134222 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:45.149954 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:45.152719 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:45.161409 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:45.164392 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:45.173322 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:45.176340 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:45.185283 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:45.187934 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:45.196846 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:45.200007 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:45.212786 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 3
2014/11/26 02:18:45.214528 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:45.215190 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:45.215922 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:45.224889 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:45.227015 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:45.228754 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:45.229444 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:45.238419 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:45.240246 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:45.240825 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:45.241529 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:45.257133 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:45.258851 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:45.259676 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:45.260270 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:45.275597 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:45.277292 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:45.277859 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:45.278977 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:45.295407 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:45.297224 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:45.297871 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:45.298532 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:45.313669 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:45.315350 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:45.315899 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:45.316546 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:45.331781 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:45.333547 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:45.334109 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:45.334838 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:45.353344 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:45.353708 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:45.999792 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:46.001280 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:46.002000 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:46.002225 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:46.002267 [ 9692]: Thawing priority 1
2014/11/26 02:18:46.002287 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:46.002362 [ 9692]: Thawing priority 2
2014/11/26 02:18:46.002385 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:46.002429 [ 9692]: Thawing priority 3
2014/11/26 02:18:46.002450 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:46.007070 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:46.007136 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:46.007165 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:46.007186 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:46.007209 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:46.007227 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:46.009597 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:46.009649 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:46.009670 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:46.009714 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:46.009738 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:46.009854 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:46.009892 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:46.010824 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:46.014531 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:46.015097 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:46.015528 [ 9692]: Freeze priority 1
2014/11/26 02:18:46.017959 [ 9692]: Freeze priority 2
2014/11/26 02:18:46.020024 [ 9692]: Freeze priority 3
2014/11/26 02:18:46.022160 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:46.337980 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:46.353482 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:46.355741 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:46.371657 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:46.374302 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:46.389998 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:46.392687 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:46.408358 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:46.410984 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:46.426661 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:46.429333 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:46.444938 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:46.447768 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:46.463260 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:46.466144 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:46.481637 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:46.484382 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:46.499848 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:46.502642 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:46.511518 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:46.514548 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:46.523648 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:46.525501 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:46.526235 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:46.527028 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:46.536492 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:46.538303 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:46.538948 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:46.539692 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:46.548645 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:46.550512 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:46.551198 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:46.552029 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:46.561324 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:46.563181 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:46.563822 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:46.564571 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:46.573446 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:46.575325 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:46.576017 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:46.577266 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:46.587086 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:46.588885 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:46.589612 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:46.590419 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:46.599422 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:46.601342 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:46.601997 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:46.602810 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:46.611961 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:46.613810 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:46.614507 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:46.615384 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:46.624374 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:46.624795 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:46.976415 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:46.977886 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:46.978574 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:46.978874 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:46.978927 [ 9692]: Thawing priority 1
2014/11/26 02:18:46.978950 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:46.979040 [ 9692]: Thawing priority 2
2014/11/26 02:18:46.979066 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:46.979105 [ 9692]: Thawing priority 3
2014/11/26 02:18:46.979124 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:46.981176 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:46.981222 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:46.981256 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:46.982611 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:46.982660 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:46.982683 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:47.011649 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:47.011710 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:47.011731 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:47.011777 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:47.011802 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:47.011946 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:47.011985 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:47.012937 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:47.017640 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:47.018424 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:47.018880 [ 9692]: Freeze priority 1
2014/11/26 02:18:47.021346 [ 9692]: Freeze priority 2
2014/11/26 02:18:47.023725 [ 9692]: Freeze priority 3
2014/11/26 02:18:47.025816 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:47.438623 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:47.454057 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:47.456399 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:47.471865 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:47.474697 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:47.490107 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:47.492890 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:47.508458 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:47.511044 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:47.526394 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:47.528998 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:47.544355 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:47.547078 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:47.555861 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:47.558924 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:47.568671 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:47.571540 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:47.580679 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:47.583401 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:47.592516 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:47.595590 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:47.604625 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:47.606426 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:47.607072 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:47.607814 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:47.617113 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:47.618870 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:47.619507 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:47.620222 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:47.629085 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:47.630853 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:47.631453 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:47.632241 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:47.641415 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:47.643148 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:47.643764 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:47.644475 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:47.653285 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:47.655179 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:47.655791 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:47.656999 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:47.666745 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:47.668586 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:47.669283 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:47.669956 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:47.685456 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:47.687457 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:47.688025 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:47.688698 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:47.697787 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:47.699581 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:47.700129 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:47.700894 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:47.709678 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:47.709985 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:48.046376 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:48.047582 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:48.048085 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:48.048212 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:48.048268 [ 9692]: Thawing priority 1
2014/11/26 02:18:48.048290 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:48.048355 [ 9692]: Thawing priority 2
2014/11/26 02:18:48.048384 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:48.048423 [ 9692]: Thawing priority 3
2014/11/26 02:18:48.048442 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:48.050549 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:48.050596 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:48.050620 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:48.050955 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:48.050993 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:48.051012 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:48.053553 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:48.053595 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:48.053612 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:48.053656 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:48.053674 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:48.053781 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:48.053812 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:48.054838 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:48.059380 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:48.059923 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:48.060355 [ 9692]: Freeze priority 1
2014/11/26 02:18:48.062626 [ 9692]: Freeze priority 2
2014/11/26 02:18:48.064413 [ 9692]: Freeze priority 3
2014/11/26 02:18:48.066228 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:48.386465 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:48.402486 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:48.404587 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:48.419985 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:48.422721 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:48.438042 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:48.440667 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:48.455957 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:48.458572 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:48.474065 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:48.476668 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:48.491963 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:48.494527 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:48.509858 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:48.512697 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:48.528349 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:48.531348 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:48.540517 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:48.543223 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:48.552265 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:48.556081 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:48.565175 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:48.567086 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:48.567773 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:48.568542 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:48.577273 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:48.579074 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:48.579726 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:48.580476 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:48.589439 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:48.591501 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:48.592141 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:48.592995 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:48.601989 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:48.603878 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:48.604530 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:48.605270 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:48.614266 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:48.616179 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:48.616825 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:48.618474 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:48.628900 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:48.630810 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:48.631778 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:48.632604 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:48.641750 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:48.643627 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:48.644252 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:48.645027 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:48.654272 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:48.656167 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:48.656802 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:48.657554 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:48.672936 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:48.673238 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:50.628242 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:50.629869 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:50.632061 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:50.632197 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:50.632235 [ 9692]: Thawing priority 1
2014/11/26 02:18:50.632254 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:50.632328 [ 9692]: Thawing priority 2
2014/11/26 02:18:50.632351 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:50.632396 [ 9692]: Thawing priority 3
2014/11/26 02:18:50.632416 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:50.636475 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:50.636535 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:50.636562 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:50.636582 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:50.636605 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:50.636624 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:50.638384 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:50.638434 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:50.638454 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:50.638496 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:50.638520 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:50.638597 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:50.638631 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:50.639506 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:50.641572 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:50.642187 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:50.642686 [ 9692]: Freeze priority 1
2014/11/26 02:18:50.645489 [ 9692]: Freeze priority 2
2014/11/26 02:18:50.647720 [ 9692]: Freeze priority 3
2014/11/26 02:18:50.649832 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:50.963260 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:50.978652 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:50.980869 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:50.989772 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:50.992834 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:51.002058 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:51.005471 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:51.014513 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:51.017565 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:51.028400 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:51.031367 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:51.041517 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:51.056525 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:51.065123 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 0
2014/11/26 02:18:51.068127 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:51.078637 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:51.083148 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:51.093329 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:51.100420 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:51.109547 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:51.112703 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:51.121876 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:51.123696 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:51.124245 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:51.124918 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:51.134133 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:51.135995 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:51.136559 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:51.137211 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:51.146391 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:51.148185 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:51.148716 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:51.149436 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:51.159553 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:51.161333 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:51.162058 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:51.162699 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:51.171812 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:51.173554 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:51.174119 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:51.175255 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:51.187298 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:51.189197 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:51.190778 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:51.191712 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:51.201169 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:51.202971 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:51.203516 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:51.204194 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:51.221589 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:51.223508 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:51.224226 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:51.225055 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:51.236483 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:51.237064 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:51.599756 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:51.600959 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:51.601500 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:51.601639 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:51.601697 [ 9692]: Thawing priority 1
2014/11/26 02:18:51.601720 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:51.601791 [ 9692]: Thawing priority 2
2014/11/26 02:18:51.601820 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:51.601860 [ 9692]: Thawing priority 3
2014/11/26 02:18:51.601878 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:51.604384 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:51.604449 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:51.604478 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:51.604497 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:51.604513 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:51.604530 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:51.639605 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:51.639680 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:51.639704 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:51.639767 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:51.639789 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:51.639870 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:51.639897 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:51.640954 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:51.642547 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:51.643339 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:51.643918 [ 9692]: Freeze priority 1
2014/11/26 02:18:51.646651 [ 9692]: Freeze priority 2
2014/11/26 02:18:51.648976 [ 9692]: Freeze priority 3
2014/11/26 02:18:51.651211 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:51.966355 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:51.981849 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:51.983945 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:51.999356 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:52.002171 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:52.017541 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:52.020151 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:52.035748 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:52.038375 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:52.053968 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:52.056639 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:52.071998 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:52.074808 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:52.090744 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:52.093834 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:52.109202 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:52.112166 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:52.127716 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:52.130786 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:52.146164 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:52.149044 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:52.164501 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:52.166375 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:52.166957 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:52.167579 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:52.183337 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:52.185133 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:52.185753 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:52.186343 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:52.201768 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:52.203518 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:52.204119 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:52.204787 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:52.220853 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:52.222735 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:52.223312 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:52.223907 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:52.241515 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:52.243508 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:52.244258 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:52.245527 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:52.256316 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:52.258416 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:52.259088 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:52.259918 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:52.268943 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:52.270818 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:52.271483 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:52.272268 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:52.281299 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:52.283196 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:52.283856 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:52.284724 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:52.293832 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:52.294282 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:52.619782 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:52.621208 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:52.621865 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:52.622128 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:52.622183 [ 9692]: Thawing priority 1
2014/11/26 02:18:52.622205 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:52.622278 [ 9692]: Thawing priority 2
2014/11/26 02:18:52.622302 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:52.622341 [ 9692]: Thawing priority 3
2014/11/26 02:18:52.622359 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:52.624963 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:52.625192 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:52.625401 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:52.625675 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:52.625877 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:52.626061 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:52.641065 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:52.641283 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:52.641461 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:52.641594 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:52.641688 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:52.641880 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:52.641986 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:52.643192 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:52.646744 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:52.647348 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:52.647889 [ 9692]: Freeze priority 1
2014/11/26 02:18:52.650677 [ 9692]: Freeze priority 2
2014/11/26 02:18:52.653039 [ 9692]: Freeze priority 3
2014/11/26 02:18:52.655389 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:52.978318 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:52.993540 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:52.995695 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:53.011231 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:53.013958 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:53.029643 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:53.032199 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:53.047628 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:53.050232 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:53.065605 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:53.069371 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:53.084647 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:53.087562 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:53.102936 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:53.105935 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:53.121415 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:53.124124 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:53.139635 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:53.142464 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:53.151340 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:53.154466 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:53.163999 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:53.165927 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:53.166580 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:53.167371 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:53.176665 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:53.178484 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:53.179184 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:53.180068 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:53.188855 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:53.190646 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:53.191300 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:53.192161 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:53.201334 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:53.203168 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:53.203823 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:53.204570 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:53.213622 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:53.215363 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:53.215905 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:53.216974 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:53.233196 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:53.234964 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:53.235586 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:53.236299 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:53.251545 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:53.253319 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:53.253890 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:53.254529 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:53.270003 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:53.271905 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:53.272576 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:53.273456 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:53.282509 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:53.282919 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:53.619820 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:53.621196 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:53.621847 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:53.622119 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:53.622172 [ 9692]: Thawing priority 1
2014/11/26 02:18:53.622193 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:53.622265 [ 9692]: Thawing priority 2
2014/11/26 02:18:53.622310 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:53.622352 [ 9692]: Thawing priority 3
2014/11/26 02:18:53.622370 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:53.624797 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:53.624868 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:53.624899 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:53.624937 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:53.624957 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:53.624975 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:53.645687 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:53.645750 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:53.645772 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:53.645818 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:53.645837 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:53.645944 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:53.645972 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:53.647091 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:53.651952 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:53.652554 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:53.653077 [ 9692]: Freeze priority 1
2014/11/26 02:18:53.655442 [ 9692]: Freeze priority 2
2014/11/26 02:18:53.657429 [ 9692]: Freeze priority 3
2014/11/26 02:18:53.659622 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:53.996378 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:54.011991 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:54.014229 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:54.030935 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:54.033658 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:54.049656 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:54.052289 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:54.067809 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:54.070367 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:54.086092 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:54.088625 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:54.103978 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:54.107198 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:54.122953 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:54.125909 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:54.141436 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:54.144114 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:54.159627 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:54.162215 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:54.177546 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:54.180397 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:54.195757 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:54.197529 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:54.198073 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:54.198677 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:54.214465 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:54.216214 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:54.216774 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:54.217361 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:54.233112 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:54.234851 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:54.235434 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:54.236175 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:54.251735 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:54.253588 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:54.254115 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:54.254738 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:54.263600 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:54.265314 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:54.265847 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:54.266961 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:54.276657 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:54.278423 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:54.279011 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:54.279719 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:54.288501 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:54.290205 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:54.291104 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:54.291759 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:54.300675 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:54.302369 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:54.302864 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:54.303623 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:54.312638 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:54.312925 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:54.646321 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:54.647817 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:54.648484 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:54.648770 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:54.648829 [ 9692]: Thawing priority 1
2014/11/26 02:18:54.648852 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:54.648916 [ 9692]: Thawing priority 2
2014/11/26 02:18:54.648946 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:54.648985 [ 9692]: Thawing priority 3
2014/11/26 02:18:54.649003 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:54.651113 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:54.651296 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:54.651465 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:54.651654 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:54.651813 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:54.651979 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:54.654560 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:54.654679 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:54.654769 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:54.654932 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:54.655025 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:54.655214 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:54.655318 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:54.656392 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:54.660397 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:54.661188 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:54.661588 [ 9692]: Freeze priority 1
2014/11/26 02:18:54.668194 [ 9692]: Freeze priority 2
2014/11/26 02:18:54.671785 [ 9692]: Freeze priority 3
2014/11/26 02:18:54.705641 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:55.019582 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:55.035243 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:55.037382 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:55.053108 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:55.055896 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:55.071171 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:55.073767 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:55.089216 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:55.091873 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:55.107538 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:55.110466 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:55.125883 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:18:55.129182 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:55.145080 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:55.148276 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:55.163647 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:55.166723 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:55.182295 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:55.184949 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:55.200565 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:55.203484 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:55.219083 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:55.220864 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:55.221463 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:55.222081 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:55.237852 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:55.239639 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:55.240919 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:55.241506 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:55.257085 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:55.258841 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:55.259492 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:55.260170 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:55.275532 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:55.277270 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:55.277822 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:55.278408 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:55.294065 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:55.295789 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:55.296363 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:55.297429 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:55.313722 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:55.315430 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:55.316053 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:55.316720 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:55.332172 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:55.333890 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:55.334461 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:55.335088 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:55.350480 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:55.352270 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:55.352794 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:55.353562 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:55.362404 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:55.362792 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:56.008248 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:56.009620 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:56.010313 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:56.010535 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:56.010578 [ 9692]: Thawing priority 1
2014/11/26 02:18:56.010598 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:56.010673 [ 9692]: Thawing priority 2
2014/11/26 02:18:56.010696 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:56.010741 [ 9692]: Thawing priority 3
2014/11/26 02:18:56.010762 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:56.016632 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:56.016693 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:56.016720 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:56.016740 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:56.016763 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:56.016781 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:56.019182 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:56.019241 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:56.019261 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:56.019304 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:56.019328 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:56.019440 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:56.019477 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:56.020584 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:56.024614 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:56.025186 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:56.025656 [ 9692]: Freeze priority 1
2014/11/26 02:18:56.027890 [ 9692]: Freeze priority 2
2014/11/26 02:18:56.029720 [ 9692]: Freeze priority 3
2014/11/26 02:18:56.031520 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:56.402427 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:56.418864 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:56.421258 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:56.437042 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:56.440382 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:56.456742 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:56.459585 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:56.475137 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:56.478042 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:56.493185 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:56.497500 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:56.512872 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:18:56.516252 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:56.531750 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:56.534945 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:56.550529 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:56.553405 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:56.562505 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:56.565208 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:56.573955 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:56.576961 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:56.585881 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:56.587619 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:56.588184 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:56.588838 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:56.597819 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:56.599568 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:56.600080 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:56.600699 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:56.609204 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:56.610965 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:56.611501 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:56.612190 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:56.621221 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:56.622954 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:56.623502 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:56.624120 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:56.632845 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:56.634527 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:56.635102 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:56.636237 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:56.646553 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:56.648394 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:56.649034 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:56.649737 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:56.658620 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:56.660291 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:56.660824 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:56.661499 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:56.670351 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:56.672194 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:56.672759 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:56.673514 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:56.682518 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:56.682903 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:57.338112 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:57.339153 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:57.339630 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:57.339716 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:57.339755 [ 9692]: Thawing priority 1
2014/11/26 02:18:57.339775 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:57.339828 [ 9692]: Thawing priority 2
2014/11/26 02:18:57.339847 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:57.339895 [ 9692]: Thawing priority 3
2014/11/26 02:18:57.339915 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:57.345760 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:57.345824 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:57.345853 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:57.345873 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:57.345895 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:57.345914 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:57.347665 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:57.347713 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:57.347733 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:57.347778 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:57.347802 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:57.347879 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:57.347912 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:57.348736 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:57.350885 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:57.351410 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:57.351896 [ 9692]: Freeze priority 1
2014/11/26 02:18:57.354504 [ 9692]: Freeze priority 2
2014/11/26 02:18:57.356580 [ 9692]: Freeze priority 3
2014/11/26 02:18:57.358677 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:57.672455 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:57.687447 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:57.689559 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:57.704825 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:57.707607 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:57.722843 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:57.725397 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:57.740690 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:57.743333 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:57.759789 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:57.763011 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:57.778258 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:18:57.781525 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:57.797067 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:57.800145 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:57.816721 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:57.819433 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:57.834718 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:57.837356 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:57.852588 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:57.855483 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:57.870884 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:57.872588 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:57.873195 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:57.873820 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:57.889458 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:57.891306 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:57.891870 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:57.892493 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:57.901255 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:57.902947 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:57.903452 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:57.904157 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:57.912938 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:57.914766 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:57.915305 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:57.915978 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:57.924775 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:57.926681 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:57.927294 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:57.928399 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:57.938087 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:57.939946 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:57.940535 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:57.941245 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:57.950127 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:57.951801 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:57.952327 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:57.952979 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:57.961762 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:57.963531 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:57.964131 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:57.964873 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:57.973688 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:57.974010 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:58.298035 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:58.299170 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:58.299675 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:58.299809 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:58.299860 [ 9692]: Thawing priority 1
2014/11/26 02:18:58.299881 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:58.299952 [ 9692]: Thawing priority 2
2014/11/26 02:18:58.299976 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:58.300009 [ 9692]: Thawing priority 3
2014/11/26 02:18:58.300026 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:58.303443 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:58.303500 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:58.303527 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:58.303548 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:58.303571 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:58.303591 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:58.349312 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:58.349394 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:58.349418 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:58.349467 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:58.349492 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:58.349627 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:58.349666 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:58.350677 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:58.354562 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:58.355154 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:58.355642 [ 9692]: Freeze priority 1
2014/11/26 02:18:58.358060 [ 9692]: Freeze priority 2
2014/11/26 02:18:58.360022 [ 9692]: Freeze priority 3
2014/11/26 02:18:58.361930 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:58.754896 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:58.770371 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:58.773051 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:58.788724 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:58.791686 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:58.807193 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:58.810159 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:58.825591 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:58.828342 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:58.843470 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:58.847709 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:58.863080 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:18:58.866374 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:58.881914 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:58.885017 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:58.900398 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:58.903581 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:58.912564 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:58.915355 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:58.924260 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:58.927289 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:58.936205 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:58.937885 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:58.938446 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:58.939130 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:58.948290 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:58.950025 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:58.950586 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:58.951245 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:58.960841 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:58.962516 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:58.963113 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:58.963847 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:58.972889 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:58.974605 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:58.975149 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:58.975858 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:58.984609 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:58.986312 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:58.986849 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:58.987967 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:58.997720 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:58.999410 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:59.000017 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:59.000730 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:18:59.009644 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:18:59.011341 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:18:59.011882 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:18:59.012571 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:18:59.021587 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:18:59.023250 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:18:59.023783 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:18:59.024566 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:18:59.033624 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:18:59.033935 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:18:59.376284 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:18:59.377743 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:18:59.378459 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:18:59.378708 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:59.378760 [ 9692]: Thawing priority 1
2014/11/26 02:18:59.378781 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:18:59.378844 [ 9692]: Thawing priority 2
2014/11/26 02:18:59.378866 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:18:59.378926 [ 9692]: Thawing priority 3
2014/11/26 02:18:59.378947 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:18:59.382071 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:59.382128 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:18:59.382156 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:18:59.382182 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:18:59.382201 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:18:59.382224 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:18:59.384686 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:18:59.384734 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:18:59.384754 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:18:59.384796 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:18:59.384821 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:59.384931 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:18:59.384966 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:18:59.386100 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:18:59.393159 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:18:59.393776 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:18:59.394210 [ 9692]: Freeze priority 1
2014/11/26 02:18:59.396595 [ 9692]: Freeze priority 2
2014/11/26 02:18:59.398516 [ 9692]: Freeze priority 3
2014/11/26 02:18:59.400382 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:59.713439 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:18:59.721493 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:18:59.723741 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:18:59.732628 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:18:59.735575 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:18:59.750774 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:18:59.753638 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:18:59.769117 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:18:59.771963 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:18:59.787300 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:18:59.790214 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:18:59.806871 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:18:59.810256 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:18:59.825881 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:18:59.829241 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:18:59.844533 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:18:59.847369 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:18:59.862985 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:18:59.865664 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:18:59.880948 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:18:59.883802 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:18:59.899455 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:18:59.901188 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:18:59.901852 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:18:59.902489 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:18:59.917879 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:18:59.919582 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:18:59.920167 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:18:59.920817 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:18:59.936245 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:18:59.937963 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:18:59.938538 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:18:59.939237 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:18:59.954858 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:18:59.956585 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:18:59.957176 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:18:59.957857 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:18:59.973232 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:18:59.974956 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:18:59.975542 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:18:59.976664 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:18:59.992872 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:18:59.994573 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:18:59.995213 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:18:59.995937 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:00.011354 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:00.013051 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:00.013701 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:00.014353 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:00.029720 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:00.031442 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:00.032121 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:00.032916 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:00.105973 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:00.106536 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:00.758164 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:00.759438 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:00.760100 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:00.760282 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:00.760322 [ 9692]: Thawing priority 1
2014/11/26 02:19:00.760341 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:00.760416 [ 9692]: Thawing priority 2
2014/11/26 02:19:00.760439 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:00.760485 [ 9692]: Thawing priority 3
2014/11/26 02:19:00.760505 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:00.766465 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:00.766625 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:00.766761 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:00.766887 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:00.767008 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:00.767129 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:00.769112 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:00.769224 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:00.769307 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:00.769418 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:00.769508 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:00.769653 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:00.769750 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:00.770748 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:00.772864 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:00.773487 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:00.774077 [ 9692]: Freeze priority 1
2014/11/26 02:19:00.776713 [ 9692]: Freeze priority 2
2014/11/26 02:19:00.778840 [ 9692]: Freeze priority 3
2014/11/26 02:19:00.780921 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:01.096593 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:01.111798 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:01.114120 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:01.129156 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:01.131875 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:01.147006 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:01.149596 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:01.165072 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:01.167704 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:01.183042 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:01.185896 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:01.201158 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:01.204394 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:01.220508 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:01.223698 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:01.239059 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:01.241750 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:01.257224 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:01.259884 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:01.275060 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:01.277996 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:01.293399 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:01.295120 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:01.295749 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:01.296360 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:01.311948 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:01.313660 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:01.314240 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:01.314834 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:01.330377 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:01.332079 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:01.332677 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:01.333342 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:01.348842 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:01.350570 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:01.351130 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:01.351719 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:01.367286 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:01.368998 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:01.369561 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:01.370631 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:01.386743 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:01.388445 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:01.389041 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:01.389696 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:01.404899 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:01.406633 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:01.407236 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:01.407856 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:01.423134 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:01.424815 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:01.425385 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:01.426153 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:01.441522 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:01.441814 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:02.098182 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:02.099405 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:02.100011 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:02.100086 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:02.100118 [ 9692]: Thawing priority 1
2014/11/26 02:19:02.100135 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:02.100205 [ 9692]: Thawing priority 2
2014/11/26 02:19:02.100229 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:02.100275 [ 9692]: Thawing priority 3
2014/11/26 02:19:02.100295 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:02.105961 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:02.106022 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:02.106052 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:02.106072 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:02.106095 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:02.106114 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:02.108638 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:02.108683 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:02.108702 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:02.108745 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:02.108763 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:02.108839 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:02.108864 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:02.109694 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:02.111828 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:02.112376 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:02.112994 [ 9692]: Freeze priority 1
2014/11/26 02:19:02.115498 [ 9692]: Freeze priority 2
2014/11/26 02:19:02.117628 [ 9692]: Freeze priority 3
2014/11/26 02:19:02.119679 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:02.433493 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:02.448866 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:02.451033 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:02.466545 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:02.469294 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:02.484685 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:02.487334 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:02.503023 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:02.505808 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:02.521483 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:02.524343 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:02.539900 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 2
2014/11/26 02:19:02.543272 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:02.552452 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:02.555719 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:02.564605 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:02.567990 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:02.577112 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:02.579829 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:02.588925 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:02.591981 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:02.600925 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:02.602641 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:02.603185 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:02.603828 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:02.612846 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:02.614549 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:02.615170 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:02.615863 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:02.624967 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:02.626643 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:02.627327 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:02.628025 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:02.637264 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:02.638913 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:02.639499 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:02.640107 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:02.649008 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:02.650688 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:02.651443 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:02.652559 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:02.663008 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:02.664689 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:02.665357 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:02.666068 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:02.674897 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:02.676582 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:02.677125 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:02.677775 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:02.686652 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:02.688354 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:02.688930 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:02.689683 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:02.698740 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:02.699017 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:03.046140 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:03.047574 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:03.048097 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:03.048177 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:03.048207 [ 9692]: Thawing priority 1
2014/11/26 02:19:03.048233 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:03.048297 [ 9692]: Thawing priority 2
2014/11/26 02:19:03.048326 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:03.048366 [ 9692]: Thawing priority 3
2014/11/26 02:19:03.048385 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:03.053998 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:03.054052 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:03.054078 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:03.054098 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:03.054120 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:03.054139 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:03.108081 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:03.108140 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:03.108162 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:03.108206 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:03.108232 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:03.108311 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:03.108344 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:03.109195 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:03.111269 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:03.111785 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:03.112250 [ 9692]: Freeze priority 1
2014/11/26 02:19:03.114895 [ 9692]: Freeze priority 2
2014/11/26 02:19:03.117050 [ 9692]: Freeze priority 3
2014/11/26 02:19:03.119165 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:03.438336 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:03.453695 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:03.455824 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:03.471369 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:03.474049 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:03.489405 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:03.492111 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:03.501089 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:03.503917 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:03.512744 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:03.515700 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:03.524423 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:03.529161 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:03.538168 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:03.541383 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:03.550255 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:03.552954 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:03.561947 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:03.564547 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:03.573435 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:03.576441 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:03.585249 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:03.587076 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:03.587641 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:03.588287 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:03.597259 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:03.598914 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:03.599482 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:03.600094 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:03.608842 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:03.610509 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:03.611050 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:03.611751 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:03.620733 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:03.622382 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:03.622939 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:03.623557 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:03.632381 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:03.634085 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:03.634625 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:03.635786 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:03.645444 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:03.647151 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:03.647767 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:03.648438 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:03.657752 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:03.659433 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:03.659927 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:03.660577 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:03.669382 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:03.671055 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:03.671563 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:03.672295 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:03.680906 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:03.681184 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:04.066203 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:04.067436 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:04.067991 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:04.068083 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:04.068125 [ 9692]: Thawing priority 1
2014/11/26 02:19:04.068145 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:04.068252 [ 9692]: Thawing priority 2
2014/11/26 02:19:04.068321 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:04.068370 [ 9692]: Thawing priority 3
2014/11/26 02:19:04.068396 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:04.075581 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:04.075692 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:04.075727 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:04.075755 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:04.075773 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:04.075790 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:04.108856 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:04.108931 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:04.108954 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:04.109000 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:04.109025 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:04.109104 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:04.109130 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:04.110014 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:04.112038 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:04.112547 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:04.113066 [ 9692]: Freeze priority 1
2014/11/26 02:19:04.115541 [ 9692]: Freeze priority 2
2014/11/26 02:19:04.117713 [ 9692]: Freeze priority 3
2014/11/26 02:19:04.119778 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:04.434047 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:04.449235 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:04.451858 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:04.467392 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:04.470331 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:04.485855 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:04.488651 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:04.504067 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:04.506903 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:04.522401 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:04.525201 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:04.540790 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:04.543973 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:04.559904 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:04.563387 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:04.572448 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:04.575669 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:04.591202 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:04.593848 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:04.609103 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:04.612214 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:04.621385 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:04.623259 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:04.623912 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:04.624681 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:04.633844 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:04.637759 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:04.638447 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:04.639210 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:04.648584 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:04.650355 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:04.651004 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:04.651715 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:04.661000 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:04.662810 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:04.663425 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:04.664172 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:04.673170 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:04.674979 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:04.675604 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:04.676858 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:04.686641 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:04.688499 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:04.689203 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:04.690005 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:04.698988 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:04.700787 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:04.701434 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:04.702225 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:04.711315 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:04.713089 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:04.713743 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:04.714608 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:04.723590 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:04.724031 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:05.056192 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:05.057393 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:05.057979 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:05.058163 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:05.058204 [ 9692]: Thawing priority 1
2014/11/26 02:19:05.058223 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:05.058294 [ 9692]: Thawing priority 2
2014/11/26 02:19:05.058317 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:05.058362 [ 9692]: Thawing priority 3
2014/11/26 02:19:05.058401 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:05.064036 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:05.064186 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:05.064324 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:05.064451 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:05.064569 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:05.064693 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:05.109726 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:05.109873 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:05.109994 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:05.110138 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:05.110261 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:05.110441 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:05.110579 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:05.111499 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:05.113651 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:05.114231 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:05.114795 [ 9692]: Freeze priority 1
2014/11/26 02:19:05.117317 [ 9692]: Freeze priority 2
2014/11/26 02:19:05.119507 [ 9692]: Freeze priority 3
2014/11/26 02:19:05.121505 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:05.447656 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:05.462867 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:05.465123 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:05.480641 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:05.483312 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:05.498591 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:05.501179 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:05.516660 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:05.519335 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:05.534668 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:05.537605 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:05.563881 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:05.575266 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:05.596695 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:05.599955 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:05.688857 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:05.691805 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:05.700966 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:05.703663 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:05.720709 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:05.723727 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:05.739135 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:05.741201 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:05.741775 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:05.742427 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:05.751646 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:05.753451 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:05.754058 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:05.754681 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:05.763658 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:05.765381 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:05.765925 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:05.766675 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:05.775833 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:05.777681 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:05.778239 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:05.778860 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:05.787741 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:05.789397 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:05.789989 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:05.791091 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:05.800911 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:05.802621 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:05.803218 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:05.803897 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:05.812723 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:05.814416 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:05.814941 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:05.815715 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:05.825742 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:05.827510 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:05.828077 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:05.828782 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:05.844005 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:05.844303 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:06.508114 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:06.509398 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:06.510137 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:06.510383 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:06.510427 [ 9692]: Thawing priority 1
2014/11/26 02:19:06.510447 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:06.510535 [ 9692]: Thawing priority 2
2014/11/26 02:19:06.510560 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:06.510611 [ 9692]: Thawing priority 3
2014/11/26 02:19:06.510633 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:06.516475 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:06.516651 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:06.516790 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:06.516916 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:06.517036 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:06.517158 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:06.519029 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:06.519144 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:06.519230 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:06.519343 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:06.519433 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:06.519579 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:06.519677 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:06.520629 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:06.522814 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:06.523419 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:06.523974 [ 9692]: Freeze priority 1
2014/11/26 02:19:06.526563 [ 9692]: Freeze priority 2
2014/11/26 02:19:06.528778 [ 9692]: Freeze priority 3
2014/11/26 02:19:06.530795 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:06.859558 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:06.874824 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:06.876943 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:06.892401 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:06.895163 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:06.910587 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:06.913190 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:06.928666 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:06.931287 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:06.947016 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:06.950022 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:06.958861 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:06.962199 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:06.971378 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:06.974764 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:06.983492 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:06.986237 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:06.996398 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:06.999090 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:07.007928 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:07.011793 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:07.020777 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:07.022517 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:07.023071 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:07.023750 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:07.032751 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:07.034429 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:07.034944 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:07.035571 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:07.044807 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:07.046501 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:07.047073 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:07.047775 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:07.057063 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:07.058731 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:07.059262 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:07.059904 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:07.075406 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:07.077205 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:07.077781 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:07.078847 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:07.095136 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:07.096857 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:07.097524 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:07.098173 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:07.113737 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:07.115394 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:07.116017 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:07.116646 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:07.131992 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:07.133670 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:07.134283 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:07.134993 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:07.150285 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:07.150600 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:07.798126 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:07.799397 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:07.800015 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:07.800200 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:07.800242 [ 9692]: Thawing priority 1
2014/11/26 02:19:07.800262 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:07.800333 [ 9692]: Thawing priority 2
2014/11/26 02:19:07.800356 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:07.800400 [ 9692]: Thawing priority 3
2014/11/26 02:19:07.800421 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:07.806375 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:07.806541 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:07.806677 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:07.806803 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:07.806921 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:07.807041 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:07.808879 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:07.808992 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:07.809079 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:07.809190 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:07.809280 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:07.809427 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:07.809524 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:07.810425 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:07.812556 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:07.813172 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:07.813724 [ 9692]: Freeze priority 1
2014/11/26 02:19:07.816928 [ 9692]: Freeze priority 2
2014/11/26 02:19:07.819184 [ 9692]: Freeze priority 3
2014/11/26 02:19:07.821334 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:08.140255 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:08.155904 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:08.158184 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:08.173901 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:08.176717 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:08.192208 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:08.194837 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:08.210399 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:08.213207 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:08.228817 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:08.231849 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:08.247215 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:08.250506 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:08.266173 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:08.269395 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:08.285134 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:08.288067 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:08.303738 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:08.306452 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:08.321789 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:08.324734 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:08.340360 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:08.342111 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:08.342635 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:08.343259 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:08.352307 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:08.354090 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:08.354762 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:08.355483 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:08.364583 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:08.366345 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:08.366965 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:08.367767 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:08.378194 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:08.380045 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:08.380738 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:08.381473 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:08.389985 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:08.391806 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:08.392410 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:08.393621 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:08.403439 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:08.405150 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:08.405755 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:08.406477 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:08.415475 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:08.417196 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:08.417752 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:08.418444 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:08.427250 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:08.428956 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:08.429532 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:08.430302 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:08.439247 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:08.439561 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:08.778001 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:08.779204 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:08.779725 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:08.779889 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:08.779949 [ 9692]: Thawing priority 1
2014/11/26 02:19:08.779972 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:08.780029 [ 9692]: Thawing priority 2
2014/11/26 02:19:08.780054 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:08.780093 [ 9692]: Thawing priority 3
2014/11/26 02:19:08.780111 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:08.786318 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:08.786367 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:08.786391 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:08.786411 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:08.786427 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:08.786444 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:08.810023 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:08.810068 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:08.810086 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:08.810128 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:08.810145 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:08.810245 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:08.810296 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:08.811129 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:08.813188 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:08.813678 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:08.814179 [ 9692]: Freeze priority 1
2014/11/26 02:19:08.816917 [ 9692]: Freeze priority 2
2014/11/26 02:19:08.819199 [ 9692]: Freeze priority 3
2014/11/26 02:19:08.821409 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:09.135555 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:09.150574 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:09.152657 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:09.167837 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:09.170568 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:09.185721 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:09.188233 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:09.203641 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:09.206268 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:09.222366 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:09.225224 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:09.240785 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:09.244548 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:09.260251 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:09.263474 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:09.280340 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:09.283789 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:09.299275 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:09.301861 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:09.317287 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:09.320255 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:09.335839 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:09.337600 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:09.338194 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:09.338804 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:09.354088 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:09.355825 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:09.356406 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:09.356997 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:09.372452 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:09.374172 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:09.374784 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:09.375459 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:09.391328 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:09.393031 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:09.393640 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:09.394221 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:09.409693 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:09.411575 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:09.412162 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:09.413283 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:09.423153 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:09.424991 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:09.617719 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:09.618435 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:09.633941 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:09.635909 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:09.636512 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:09.637140 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:09.652591 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:09.654281 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:09.654834 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:09.655549 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:09.670951 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:09.671242 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:11.968415 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:11.969570 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:11.970245 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:11.970319 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:11.970364 [ 9692]: Thawing priority 1
2014/11/26 02:19:11.970382 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:11.970454 [ 9692]: Thawing priority 2
2014/11/26 02:19:11.970476 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:11.970521 [ 9692]: Thawing priority 3
2014/11/26 02:19:11.970540 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:11.976252 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:11.976312 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:11.976338 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:11.976359 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:11.976382 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:11.976400 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:11.978215 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:11.978264 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:11.978319 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:11.978367 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:11.978398 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:11.978478 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:11.978517 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:11.979390 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:11.981453 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:11.981998 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:11.982523 [ 9692]: Freeze priority 1
2014/11/26 02:19:11.985148 [ 9692]: Freeze priority 2
2014/11/26 02:19:11.987397 [ 9692]: Freeze priority 3
2014/11/26 02:19:11.989613 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:12.433889 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:12.448997 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:12.451209 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:12.495461 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:12.498442 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:12.513790 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:12.516532 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:12.532255 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:12.534902 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:12.550617 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:12.553614 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:12.569308 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:12.572838 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:12.582161 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:12.585520 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:12.594434 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:12.597176 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:12.606236 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:12.608838 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:12.617569 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:12.620435 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:12.629593 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:12.631281 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:12.631843 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:12.632515 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:12.641689 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:12.643359 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:12.643883 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:12.644582 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:12.653404 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:12.655065 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:12.655653 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:12.656395 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:12.665419 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:12.667090 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:12.667613 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:12.668295 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:12.677083 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:12.678730 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:12.679264 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:12.680398 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:12.690052 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:12.691775 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:12.692405 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:12.693126 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:12.702537 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:12.704246 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:12.704820 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:12.705489 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:12.714546 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:12.716260 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:12.716843 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:12.717657 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:12.726649 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:12.726928 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:13.057924 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:13.059002 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:13.059858 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:13.060096 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:13.060148 [ 9692]: Thawing priority 1
2014/11/26 02:19:13.060195 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:13.060273 [ 9692]: Thawing priority 2
2014/11/26 02:19:13.060347 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:13.060406 [ 9692]: Thawing priority 3
2014/11/26 02:19:13.060466 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:13.066089 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:13.066148 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:13.066176 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:13.066195 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:13.066218 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:13.066237 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:13.068034 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:13.068085 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:13.068105 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:13.068149 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:13.068173 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:13.068279 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:13.068323 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:13.268090 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:13.271306 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:13.271932 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:13.272609 [ 9692]: Freeze priority 1
2014/11/26 02:19:13.276136 [ 9692]: Freeze priority 2
2014/11/26 02:19:13.278444 [ 9692]: Freeze priority 3
2014/11/26 02:19:13.280548 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:13.599784 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:13.614761 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:13.616936 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:13.632368 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:13.635165 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:13.650702 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:13.653347 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:13.668953 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:13.671863 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:13.681073 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:13.684106 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:13.693144 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:13.696650 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:13.706034 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:13.709348 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:13.718449 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:13.721265 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:13.730454 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:13.733197 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:13.742254 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:13.745546 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:13.760970 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:13.762904 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:13.763499 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:13.764240 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:13.773173 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:13.774967 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:13.775560 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:13.776382 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:13.785311 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:13.787222 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:13.787845 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:13.788630 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:13.797775 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:13.799560 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:13.800187 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:13.800958 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:13.809939 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:13.811684 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:13.812259 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:13.813445 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:13.823314 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:13.825079 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:13.825787 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:13.826570 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:13.835517 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:13.838406 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:13.839014 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:13.839762 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:13.848578 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:13.850335 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:13.850926 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:13.851793 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:13.860629 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:13.861011 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:14.196404 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:14.197626 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:14.198114 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:14.198203 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:14.198249 [ 9692]: Thawing priority 1
2014/11/26 02:19:14.198269 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:14.198379 [ 9692]: Thawing priority 2
2014/11/26 02:19:14.198412 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:14.198502 [ 9692]: Thawing priority 3
2014/11/26 02:19:14.198531 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:14.204301 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:14.204362 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:14.204391 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:14.204411 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:14.204434 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:14.204453 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:14.206240 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:14.206296 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:14.206315 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:14.206361 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:14.206386 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:14.206493 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:14.206557 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:14.207381 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:14.210404 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:14.210942 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:14.211414 [ 9692]: Freeze priority 1
2014/11/26 02:19:14.214641 [ 9692]: Freeze priority 2
2014/11/26 02:19:14.216813 [ 9692]: Freeze priority 3
2014/11/26 02:19:14.219208 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:14.544285 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:14.559272 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:14.561433 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:14.576801 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:14.579901 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:14.595307 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:14.597906 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:14.613248 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:14.615879 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:14.631441 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:14.634432 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:14.649972 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:14.653485 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:14.662565 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:14.665864 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:14.674610 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:14.677516 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:14.686574 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:14.689378 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:14.698262 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:14.701284 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:14.710204 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:14.712017 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:14.907778 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:14.908736 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:14.917840 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:14.919880 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:14.920579 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:14.921328 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:14.930229 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:14.931987 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:14.932702 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:14.933500 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:14.942705 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:14.944493 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:14.945072 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:14.945848 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:14.954630 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:14.956387 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:14.957017 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:14.958211 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:14.967959 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:14.969704 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:14.970414 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:14.971163 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:14.983468 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:14.985206 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:14.985827 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:14.986583 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:14.995480 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:14.997347 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:14.997952 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:14.998859 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:15.014508 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:15.014863 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:15.678178 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:15.679359 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:15.679907 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:15.680029 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:15.680093 [ 9692]: Thawing priority 1
2014/11/26 02:19:15.680114 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:15.680177 [ 9692]: Thawing priority 2
2014/11/26 02:19:15.680199 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:15.680258 [ 9692]: Thawing priority 3
2014/11/26 02:19:15.680280 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:15.686053 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:15.686147 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:15.686205 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:15.686317 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:15.686389 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:15.686472 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:15.688320 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:15.688402 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:15.688461 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:15.688546 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:15.688609 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:15.688726 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:15.688795 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:15.689703 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:15.692769 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:15.693341 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:15.693840 [ 9692]: Freeze priority 1
2014/11/26 02:19:15.696700 [ 9692]: Freeze priority 2
2014/11/26 02:19:15.698960 [ 9692]: Freeze priority 3
2014/11/26 02:19:15.701125 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:16.151134 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:16.166222 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:16.168417 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:16.183746 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:16.186562 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:16.202144 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:16.204754 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:16.220875 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:16.224109 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:16.239399 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:16.242548 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:16.251437 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:16.254881 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:16.264102 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:16.267499 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:16.276543 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:16.279322 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:16.294944 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:16.297628 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:16.306488 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:16.309465 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:16.318302 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:16.320087 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:16.320656 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:16.321316 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:16.330411 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:16.332133 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:16.332677 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:16.333285 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:16.342085 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:16.343732 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:16.344285 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:16.344984 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:16.354061 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:16.355775 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:16.356328 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:16.356949 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:16.366316 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:16.368039 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:16.368572 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:16.369763 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:16.379585 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:16.381264 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:16.381894 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:16.382689 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:16.391616 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:16.393324 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:16.393858 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:16.394586 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:16.403405 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:16.405168 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:16.405733 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:16.406587 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:16.415538 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:16.416036 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:16.756260 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:16.757710 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:16.758334 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:16.758467 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:16.758504 [ 9692]: Thawing priority 1
2014/11/26 02:19:16.758521 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:16.758595 [ 9692]: Thawing priority 2
2014/11/26 02:19:16.758617 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:16.758685 [ 9692]: Thawing priority 3
2014/11/26 02:19:16.758708 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:16.763898 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:16.763961 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:16.763988 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:16.764006 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:16.764023 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:16.764040 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:16.765963 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:16.766017 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:16.766069 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:16.766125 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:16.766145 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:16.766263 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:16.766365 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:16.767246 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:16.770426 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:16.770981 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:16.771532 [ 9692]: Freeze priority 1
2014/11/26 02:19:16.774254 [ 9692]: Freeze priority 2
2014/11/26 02:19:16.776749 [ 9692]: Freeze priority 3
2014/11/26 02:19:16.779011 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:17.098606 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:17.113915 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:17.116133 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:17.131471 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:17.134282 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:17.149627 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:17.152171 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:17.167439 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:17.170036 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:17.185444 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:17.188396 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:17.316206 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:17.319780 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:17.335529 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:17.338981 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:17.347817 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:17.350644 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:17.363222 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:17.366337 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:17.375195 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:17.378295 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:17.387452 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:17.389277 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:17.389915 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:17.390643 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:17.399808 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:17.401565 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:17.402176 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:17.402873 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:17.411905 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:17.413707 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:17.414326 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:17.415101 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:17.424257 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:17.426076 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:17.426736 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:17.427433 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:17.436379 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:17.438108 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:17.438809 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:17.439873 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:17.455863 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:17.457618 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:17.458293 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:17.458952 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:17.474093 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:17.475883 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:17.476432 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:17.477051 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:17.492530 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:17.494237 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:17.494814 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:17.495537 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:17.510921 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:17.511226 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:18.168115 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:18.169575 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:18.170214 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:18.170291 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:18.170322 [ 9692]: Thawing priority 1
2014/11/26 02:19:18.170339 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:18.170409 [ 9692]: Thawing priority 2
2014/11/26 02:19:18.170433 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:18.170484 [ 9692]: Thawing priority 3
2014/11/26 02:19:18.170505 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:18.176047 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:18.176105 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:18.176132 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:18.176152 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:18.176174 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:18.176193 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:18.178138 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:18.178191 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:18.178230 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:18.178276 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:18.178301 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:18.178408 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:18.178471 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:18.179444 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:18.182596 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:18.183201 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:18.183727 [ 9692]: Freeze priority 1
2014/11/26 02:19:18.186304 [ 9692]: Freeze priority 2
2014/11/26 02:19:18.188464 [ 9692]: Freeze priority 3
2014/11/26 02:19:18.190591 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:18.505638 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:18.520955 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:18.523092 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:18.538612 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:18.541449 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:18.556746 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:18.559452 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:18.574961 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:18.577821 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:18.586998 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:18.590490 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:18.599770 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 1
2014/11/26 02:19:18.603317 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:18.612469 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:18.615786 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:18.624704 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:18.627656 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:18.636871 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:18.639784 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:18.648929 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:18.652305 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:18.661365 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:18.663173 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:18.663780 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:18.664530 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:18.673594 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:18.676518 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:18.677181 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:18.677913 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:18.686795 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:18.689461 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:18.690154 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:18.690953 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:18.699979 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:18.701770 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:18.702392 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:18.703112 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:18.712067 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:18.713831 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:18.714420 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:18.715599 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:18.725282 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:18.727052 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:18.727698 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:18.728460 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:18.737343 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:18.739121 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:18.739720 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:18.740561 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:18.749361 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:18.751134 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:18.751748 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:18.752594 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:18.761452 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:18.761836 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:19.118068 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:19.119847 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:19.120541 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:19.120858 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:19.120921 [ 9692]: Thawing priority 1
2014/11/26 02:19:19.120955 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:19.121124 [ 9692]: Thawing priority 2
2014/11/26 02:19:19.121175 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:19.121229 [ 9692]: Thawing priority 3
2014/11/26 02:19:19.121251 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:19.124231 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:19.124397 [recoverd: 9883]: ctdb_control error: 'managed
to lock reclock file from inside daemon'
2014/11/26 02:19:19.124536 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=16
2014/11/26 02:19:19.124665 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:19.124786 [recoverd: 9883]: server/ctdb_recoverd.c:393
Unable to set recovery mode. Recovery failed.
2014/11/26 02:19:19.124908 [recoverd: 9883]: server/ctdb_recoverd.c:2044
Unable to set recovery mode to normal on cluster
2014/11/26 02:19:19.179134 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:19.179197 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:19.179219 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:19.179265 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:19.179290 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:19.179401 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:19.179465 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 0
2014/11/26 02:19:19.180413 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:19.183433 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:19.184033 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to ACTIVE
2014/11/26 02:19:19.184541 [ 9692]: Freeze priority 1
2014/11/26 02:19:19.187079 [ 9692]: Freeze priority 2
2014/11/26 02:19:19.189260 [ 9692]: Freeze priority 3
2014/11/26 02:19:19.191532 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:19.445524 [ 9692]: 192.168.41.30:4379: node
192.168.41.34:4379 is dead: 0 connected
2014/11/26 02:19:19.445720 [ 9692]: Tearing down connection to dead node :1
2014/11/26 02:19:19.445936 [recoverd: 9883]: ctdb_control error: 'node is
disconnected'
2014/11/26 02:19:19.445974 [recoverd: 9883]: ctdb_control error: 'node is
disconnected'
2014/11/26 02:19:19.446055 [recoverd: 9883]: Async operation failed with
ret=-1 res=-1 opcode=70
2014/11/26 02:19:19.446081 [recoverd: 9883]: server/ctdb_recoverd.c:279
Node 1 failed the startrecovery event. Setting it as recovery fail culprit
2014/11/26 02:19:19.503153 [recoverd: 9883]: Async wait failed -
fail_count=1
2014/11/26 02:19:19.503260 [recoverd: 9883]: server/ctdb_recoverd.c:303
Unable to run the 'startrecovery' event. Recovery failed.
2014/11/26 02:19:19.503290 [recoverd: 9883]: server/ctdb_recoverd.c:1888
Unable to run the 'startrecovery' event on cluster
2014/11/26 02:19:20.178738 [recoverd: 9883]: server/ctdb_recoverd.c:1797
Starting do_recovery
2014/11/26 02:19:20.178822 [recoverd: 9883]: Taking out recovery lock from
recovery daemon
2014/11/26 02:19:20.178843 [recoverd: 9883]: Take the recovery lock
2014/11/26 02:19:20.178902 [recoverd: 9883]: Recovery lock taken
successfully
2014/11/26 02:19:20.178921 [recoverd: 9883]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:20.179097 [recoverd: 9883]: Recovery lock taken
successfully by recovery daemon
2014/11/26 02:19:20.179132 [recoverd: 9883]: server/ctdb_recoverd.c:1838
Recovery initiated due to problem with node 1
2014/11/26 02:19:20.179243 [recoverd: 9883]: server/ctdb_recoverd.c:1863
Recovery - created remote databases
2014/11/26 02:19:20.180895 [recoverd: 9883]: server/ctdb_recoverd.c:1870
Recovery - updated db priority for all databases
2014/11/26 02:19:20.181130 [ 9692]: Freeze priority 1
2014/11/26 02:19:20.181264 [ 9692]: Freeze priority 2
2014/11/26 02:19:20.181418 [ 9692]: Freeze priority 3
2014/11/26 02:19:20.181571 [ 9692]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:20.497902 [recoverd: 9883]: server/ctdb_recoverd.c:1911
Recovery - updated flags
2014/11/26 02:19:20.505219 [recoverd: 9883]: server/ctdb_recoverd.c:1955
started transactions on all nodes
2014/11/26 02:19:20.506960 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x66f71b8c
2014/11/26 02:19:20.515013 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x66f71b8c of size 0
2014/11/26 02:19:20.517072 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xaf029e9d
2014/11/26 02:19:20.525242 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xaf029e9d of size 0
2014/11/26 02:19:20.527248 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7a19d84d
2014/11/26 02:19:20.535220 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7a19d84d of size 0
2014/11/26 02:19:20.537208 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4e66c2b2
2014/11/26 02:19:20.545143 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4e66c2b2 of size 0
2014/11/26 02:19:20.547335 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x68c12c2c
2014/11/26 02:19:20.555296 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x68c12c2c of size 0
2014/11/26 02:19:20.557956 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6b06a26d
2014/11/26 02:19:20.566032 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6b06a26d of size 1
2014/11/26 02:19:20.568560 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x521b7544
2014/11/26 02:19:20.576558 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x521b7544 of size 1
2014/11/26 02:19:20.578661 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6afb8c09
2014/11/26 02:19:20.586821 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6afb8c09 of size 0
2014/11/26 02:19:20.588788 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x4d2a432b
2014/11/26 02:19:20.596813 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x4d2a432b of size 0
2014/11/26 02:19:20.599008 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x9ec2a880
2014/11/26 02:19:20.607117 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x9ec2a880 of size 4
2014/11/26 02:19:20.608921 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x3ef19640
2014/11/26 02:19:20.609193 [recoverd: 9883]: Pull persistent db:0x3ef19640
from node 0 with highest seqnum:1
2014/11/26 02:19:20.609919 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x3ef19640
2014/11/26 02:19:20.618213 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x3ef19640 of size 4
2014/11/26 02:19:20.619999 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3078fba
2014/11/26 02:19:20.620268 [recoverd: 9883]: Pull persistent db:0xc3078fba
from node 0 with highest seqnum:1
2014/11/26 02:19:20.620969 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3078fba
2014/11/26 02:19:20.629032 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3078fba of size 2
2014/11/26 02:19:20.630918 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xc3db117c
2014/11/26 02:19:20.631178 [recoverd: 9883]: Pull persistent db:0xc3db117c
from node 0 with highest seqnum:8
2014/11/26 02:19:20.632007 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xc3db117c
2014/11/26 02:19:20.640020 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xc3db117c of size 12
2014/11/26 02:19:20.641864 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6645c6c4
2014/11/26 02:19:20.642129 [recoverd: 9883]: Pull persistent db:0x6645c6c4
from node 0 with highest seqnum:0
2014/11/26 02:19:20.642834 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6645c6c4
2014/11/26 02:19:20.650901 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6645c6c4 of size 1
2014/11/26 02:19:20.652732 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x6cf2837d
2014/11/26 02:19:20.652990 [recoverd: 9883]: Pull persistent db:0x6cf2837d
from node 0 with highest seqnum:4
2014/11/26 02:19:20.654168 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x6cf2837d
2014/11/26 02:19:20.662483 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x6cf2837d of size 73
2014/11/26 02:19:20.664306 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x7132c184
2014/11/26 02:19:20.664572 [recoverd: 9883]: Pull persistent db:0x7132c184
from node 0 with highest seqnum:22
2014/11/26 02:19:20.665333 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x7132c184
2014/11/26 02:19:20.673610 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x7132c184 of size 8
2014/11/26 02:19:20.675361 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0xa1413774
2014/11/26 02:19:20.675533 [recoverd: 9883]: Pull persistent db:0xa1413774
from node 0 with highest seqnum:4
2014/11/26 02:19:20.676258 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0xa1413774
2014/11/26 02:19:20.683935 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0xa1413774 of size 5
2014/11/26 02:19:20.685727 [recoverd: 9883]: Scan for highest seqnum pdb
for db:0x2ca251cf
2014/11/26 02:19:20.685954 [recoverd: 9883]: Pull persistent db:0x2ca251cf
from node 0 with highest seqnum:1
2014/11/26 02:19:20.686790 [recoverd: 9883]: server/ctdb_recoverd.c:1485
Recovery - pulled remote database 0x2ca251cf
2014/11/26 02:19:20.694574 [recoverd: 9883]: server/ctdb_recoverd.c:1446
Recovery - pushed remote database 0x2ca251cf of size 18
2014/11/26 02:19:20.694924 [recoverd: 9883]: server/ctdb_recoverd.c:1968
Recovery - starting database commits
2014/11/26 02:19:21.016121 [recoverd: 9883]: server/ctdb_recoverd.c:1980
Recovery - committed databases
2014/11/26 02:19:21.016395 [recoverd: 9883]: server/ctdb_recoverd.c:2030
Recovery - updated vnnmap
2014/11/26 02:19:21.016570 [recoverd: 9883]: server/ctdb_recoverd.c:2039
Recovery - updated recmaster
2014/11/26 02:19:21.016715 [ 9692]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:21.016751 [ 9692]: Thawing priority 1
2014/11/26 02:19:21.016769 [ 9692]: Release freeze handler for prio 1
2014/11/26 02:19:21.016839 [ 9692]: Thawing priority 2
2014/11/26 02:19:21.016862 [ 9692]: Release freeze handler for prio 2
2014/11/26 02:19:21.016896 [ 9692]: Thawing priority 3
2014/11/26 02:19:21.016913 [ 9692]: Release freeze handler for prio 3
2014/11/26 02:19:21.019391 [recoverd: 9883]: server/ctdb_recoverd.c:2048
Recovery - disabled recovery mode
2014/11/26 02:19:21.019640 [recoverd: 9883]: Takeover run starting
2014/11/26 02:19:21.335862 [recoverd: 9883]: Takeover run completed
successfully
2014/11/26 02:19:21.335981 [ 9692]: Recovery has finished
2014/11/26 02:19:21.680044 [recoverd: 9883]: server/ctdb_recoverd.c:2068
Recovery - finished the recovered event
2014/11/26 02:19:21.681205 [recoverd: 9883]: server/ctdb_recoverd.c:2079
Recovery complete
2014/11/26 02:19:21.681249 [recoverd: 9883]: Resetting ban count to 0 for
all nodes
2014/11/26 02:19:21.681270 [recoverd: 9883]: Just finished a recovery. New
recoveries will now be supressed for the rerecovery timeout (10 seconds)
2014/11/26 02:19:21.681690 [ 9692]: Starting traverse on DB brlock.tdb (id
14118)
2014/11/26 02:19:21.684481 [ 9692]: Ending traverse on DB brlock.tdb (id
14118), records 0
2014/11/26 02:19:31.685701 [recoverd: 9883]: The rerecovery timeout has
elapsed. We now allow recoveries to trigger again.
2014/11/26 02:20:27.429629 [ 9692]: Received SHUTDOWN command.
2014/11/26 02:20:27.429711 [ 9692]: Shutdown sequence commencing.
2014/11/26 02:20:27.429734 [ 9692]: Set runstate to SHUTDOWN (6)
2014/11/26 02:20:27.429772 [ 9692]: Shutting down recovery daemon
2014/11/26 02:20:27.429805 [ 9692]: Monitoring has been stopped
2014/11/26 02:20:27.505464 [ 9692]: 10.interface: Killing TCP connection
192.168.11.210:56059 192.168.11.34:445
2014/11/26 02:20:27.545515 [ 9692]: 10.interface: Killed 1 TCP connections
to released IP 192.168.11.34
2014/11/26 02:20:28.497708 [ 9692]: server/ctdb_takeover.c:3303 Released 3
public IPs
2014/11/26 02:20:28.525749 [ 9692]: common/ctdb_fork.c:123 waitpid()
returned error. errno:10
2014/11/26 02:20:28.788737 [ 9692]: 50.samba: * samba -> stop: winbind ...
[ ok ]
2014/11/26 02:20:28.804466 [ 9692]: 50.samba: * samba -> stop: smbd ... [
ok ]
2014/11/26 02:20:28.844707 [ 9692]: 50.samba: * samba -> stop: nmbd ... [
ok ]
2014/11/26 02:20:28.952147 [ 9692]: Shutdown sequence complete, exiting.
2014/11/26 02:20:28.952248 [ 9692]: CTDB daemon shutting down
2014/11/26 02:20:29.952403 [ 9692]: Removed PID file /var/run/ctdb/ctdbd.pid
The Log Node2
2014/11/26 02:18:16.080450 [24277]: CTDB starting on node
2014/11/26 02:18:16.082499 [24278]: Starting CTDBD (Version 2.5.4) as PID:
24278
2014/11/26 02:18:16.082761 [24278]: Created PID file /var/run/ctdb/ctdbd.pid
2014/11/26 02:18:16.082813 [24278]: Set real-time scheduler priority
2014/11/26 02:18:16.082992 [24278]: Set runstate to INIT (1)
2014/11/26 02:18:17.081342 [24278]: Vacuuming is disabled for persistent
database passdb.tdb
2014/11/26 02:18:17.081398 [24278]: Attached to database
'/var/lib/ctdb/persistent/passdb.tdb.1' with flags 0x400
2014/11/26 02:18:17.097296 [24278]: Vacuuming is disabled for persistent
database ctdb.tdb
2014/11/26 02:18:17.097347 [24278]: Attached to database
'/var/lib/ctdb/persistent/ctdb.tdb.1' with flags 0x400
2014/11/26 02:18:17.114552 [24278]: Vacuuming is disabled for persistent
database account_policy.tdb
2014/11/26 02:18:17.114593 [24278]: Attached to database
'/var/lib/ctdb/persistent/account_policy.tdb.1' with flags 0x400
2014/11/26 02:18:17.129244 [24278]: Vacuuming is disabled for persistent
database winbindd_idmap.tdb
2014/11/26 02:18:17.129276 [24278]: Attached to database
'/var/lib/ctdb/persistent/winbindd_idmap.tdb.1' with flags 0x400
2014/11/26 02:18:17.143987 [24278]: Vacuuming is disabled for persistent
database group_mapping.tdb
2014/11/26 02:18:17.144019 [24278]: Attached to database
'/var/lib/ctdb/persistent/group_mapping.tdb.1' with flags 0x400
2014/11/26 02:18:17.158685 [24278]: Vacuuming is disabled for persistent
database share_info.tdb
2014/11/26 02:18:17.158718 [24278]: Attached to database
'/var/lib/ctdb/persistent/share_info.tdb.1' with flags 0x400
2014/11/26 02:18:17.173418 [24278]: Vacuuming is disabled for persistent
database registry.tdb
2014/11/26 02:18:17.173449 [24278]: Attached to database
'/var/lib/ctdb/persistent/registry.tdb.1' with flags 0x400
2014/11/26 02:18:17.188087 [24278]: Vacuuming is disabled for persistent
database secrets.tdb
2014/11/26 02:18:17.188121 [24278]: Attached to database
'/var/lib/ctdb/persistent/secrets.tdb.1' with flags 0x400
2014/11/26 02:18:17.188185 [24278]: Freeze priority 1
2014/11/26 02:18:17.191842 [24278]: Attached to database
'/var/lib/ctdb/serverid.tdb.1' with flags 0xc41
2014/11/26 02:18:17.697510 [24278]: Attached to database
'/var/lib/ctdb/g_lock.tdb.1' with flags 0xc41
2014/11/26 02:18:17.703275 [24278]: Attached to database
'/var/lib/ctdb/dbwrap_watchers.tdb.1' with flags 0xc41
2014/11/26 02:18:17.708371 [24278]: Attached to database
'/var/lib/ctdb/smbXsrv_version_global.tdb.1' with flags 0xc41
2014/11/26 02:18:17.713777 [24278]: Attached to database
'/var/lib/ctdb/smbXsrv_session_global.tdb.1' with flags 0xc41
2014/11/26 02:18:17.718698 [24278]: Attached to database
'/var/lib/ctdb/smbXsrv_tcon_global.tdb.1' with flags 0xc41
2014/11/26 02:18:17.723815 [24278]: Attached to database
'/var/lib/ctdb/brlock.tdb.1' with flags 0xc41
2014/11/26 02:18:17.735527 [24278]: Attached to database
'/var/lib/ctdb/locking.tdb.1' with flags 0xc41
2014/11/26 02:18:17.739857 [24278]: Attached to database
'/var/lib/ctdb/notify_index.tdb.1' with flags 0xc41
2014/11/26 02:18:17.759371 [24278]: Attached to database
'/var/lib/ctdb/smbXsrv_open_global.tdb.1' with flags 0xc41
2014/11/26 02:18:17.761425 [24278]: Freeze priority 2
2014/11/26 02:18:17.763287 [24278]: Freeze priority 3
2014/11/26 02:18:17.764976 [24278]: server/ctdb_takeover.c:3303 Released 0
public IPs
2014/11/26 02:18:17.765010 [24278]: Set runstate to SETUP (2)
2014/11/26 02:18:18.092273 [24278]: Keepalive monitoring has been started
2014/11/26 02:18:18.092366 [24278]: Set runstate to FIRST_RECOVERY (3)
2014/11/26 02:18:18.092997 [recoverd:24470]: monitor_cluster starting
2014/11/26 02:18:18.094418 [recoverd:24470]: server/ctdb_recoverd.c:3727
Initial recovery master set - forcing election
2014/11/26 02:18:18.094538 [24278]: Freeze priority 1
2014/11/26 02:18:18.094616 [24278]: Freeze priority 2
2014/11/26 02:18:18.094686 [24278]: Freeze priority 3
2014/11/26 02:18:18.094807 [24278]: This node (1) is now the recovery master
2014/11/26 02:18:18.596718 [24278]: This node (1) is no longer the recovery
master
2014/11/26 02:18:19.092719 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:20.092952 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:21.093235 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:21.599920 [recoverd:24470]: Election period ended
2014/11/26 02:18:21.600454 [recoverd:24470]: server/ctdb_recoverd.c:3749
Current recmaster node 0 does not have CAP_RECMASTER, but we (node 1) have
- force an election
2014/11/26 02:18:21.600563 [24278]: Freeze priority 1
2014/11/26 02:18:21.600640 [24278]: Freeze priority 2
2014/11/26 02:18:21.600711 [24278]: Freeze priority 3
2014/11/26 02:18:21.600826 [24278]: This node (1) is now the recovery master
2014/11/26 02:18:22.094063 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:22.102808 [24278]: This node (1) is no longer the recovery
master
2014/11/26 02:18:23.092989 [24278]: 192.168.41.34:4379: connected to
192.168.41.30:4379 - 1 connected
2014/11/26 02:18:23.095155 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:24.095401 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:24.916681 [24278]: Freeze priority 1
2014/11/26 02:18:24.918910 [24278]: Freeze priority 2
2014/11/26 02:18:24.920670 [24278]: Freeze priority 3
2014/11/26 02:18:24.922597 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:25.095874 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:25.104919 [recoverd:24470]: Election period ended
2014/11/26 02:18:25.106924 [recoverd:24470]: Initial interface fetched
2014/11/26 02:18:25.106968 [recoverd:24470]: The interfaces status has
changed on local node 1 - force takeover run
2014/11/26 02:18:25.238548 [24278]: server/ctdb_monitor.c:495 Node 0 became
healthy - force recovery for startup
2014/11/26 02:18:25.239040 [recoverd:24470]: Node 0 has changed flags - now
0x0 was 0x2
2014/11/26 02:18:25.921823 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:25.921891 [24278]: Thawing priority 1
2014/11/26 02:18:25.921910 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:25.921983 [24278]: Thawing priority 2
2014/11/26 02:18:25.922004 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:25.922050 [24278]: Thawing priority 3
2014/11/26 02:18:25.922068 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:26.096212 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:26.096282 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:26.359860 [set_recmode:24546]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:26.360186 [set_recmode:24546]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:26.368002 [24278]: Freeze priority 1
2014/11/26 02:18:26.370745 [24278]: Freeze priority 2
2014/11/26 02:18:26.372873 [24278]: Freeze priority 3
2014/11/26 02:18:26.375129 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:27.255331 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:27.255397 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:27.260291 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:27.260332 [24278]: Thawing priority 1
2014/11/26 02:18:27.260349 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:27.260411 [24278]: Thawing priority 2
2014/11/26 02:18:27.260430 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:27.260475 [24278]: Thawing priority 3
2014/11/26 02:18:27.260493 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:27.262742 [set_recmode:24610]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:27.262870 [set_recmode:24610]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:27.368487 [24278]: Freeze priority 1
2014/11/26 02:18:27.371048 [24278]: Freeze priority 2
2014/11/26 02:18:27.373172 [24278]: Freeze priority 3
2014/11/26 02:18:27.375722 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:28.296987 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:28.297073 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:28.308402 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:28.308444 [24278]: Thawing priority 1
2014/11/26 02:18:28.308462 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:28.308532 [24278]: Thawing priority 2
2014/11/26 02:18:28.308552 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:28.308600 [24278]: Thawing priority 3
2014/11/26 02:18:28.308619 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:28.309657 [set_recmode:24675]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:28.309899 [set_recmode:24675]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:28.369119 [24278]: Freeze priority 1
2014/11/26 02:18:28.371787 [24278]: Freeze priority 2
2014/11/26 02:18:28.374017 [24278]: Freeze priority 3
2014/11/26 02:18:28.376278 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:29.297576 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:29.297645 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:29.307132 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:29.307174 [24278]: Thawing priority 1
2014/11/26 02:18:29.307192 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:29.307257 [24278]: Thawing priority 2
2014/11/26 02:18:29.307277 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:29.307324 [24278]: Thawing priority 3
2014/11/26 02:18:29.307343 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:29.308486 [set_recmode:24739]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:29.308748 [set_recmode:24739]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:29.371244 [24278]: Freeze priority 1
2014/11/26 02:18:29.373816 [24278]: Freeze priority 2
2014/11/26 02:18:29.376028 [24278]: Freeze priority 3
2014/11/26 02:18:29.378279 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:30.457013 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:30.457080 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:30.468494 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:30.468536 [24278]: Thawing priority 1
2014/11/26 02:18:30.468555 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:30.468616 [24278]: Thawing priority 2
2014/11/26 02:18:30.468636 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:30.468680 [24278]: Thawing priority 3
2014/11/26 02:18:30.468698 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:30.469701 [set_recmode:24804]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:30.469837 [set_recmode:24804]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:30.476928 [24278]: Freeze priority 1
2014/11/26 02:18:30.479422 [24278]: Freeze priority 2
2014/11/26 02:18:30.481521 [24278]: Freeze priority 3
2014/11/26 02:18:30.483787 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:31.410887 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:31.410953 [24278]: Thawing priority 1
2014/11/26 02:18:31.410973 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:31.411039 [24278]: Thawing priority 2
2014/11/26 02:18:31.411060 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:31.411104 [24278]: Thawing priority 3
2014/11/26 02:18:31.411122 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:31.412327 [set_recmode:24870]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:31.412552 [set_recmode:24870]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:31.457245 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:31.457292 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:31.478927 [24278]: Freeze priority 1
2014/11/26 02:18:31.481386 [24278]: Freeze priority 2
2014/11/26 02:18:31.483487 [24278]: Freeze priority 3
2014/11/26 02:18:31.485752 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:32.457636 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:32.457705 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:32.467661 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:32.467704 [24278]: Thawing priority 1
2014/11/26 02:18:32.467721 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:32.467787 [24278]: Thawing priority 2
2014/11/26 02:18:32.467808 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:32.467860 [24278]: Thawing priority 3
2014/11/26 02:18:32.467880 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:32.468904 [set_recmode:24933]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:32.469140 [set_recmode:24933]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:32.481720 [24278]: Freeze priority 1
2014/11/26 02:18:32.484035 [24278]: Freeze priority 2
2014/11/26 02:18:32.485923 [24278]: Freeze priority 3
2014/11/26 02:18:32.487779 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:33.399734 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:33.399797 [24278]: Thawing priority 1
2014/11/26 02:18:33.399817 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:33.399881 [24278]: Thawing priority 2
2014/11/26 02:18:33.399901 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:33.399943 [24278]: Thawing priority 3
2014/11/26 02:18:33.399961 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:33.400973 [set_recmode:24999]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:33.401207 [set_recmode:24999]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:33.457749 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:33.457796 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:33.482598 [24278]: Freeze priority 1
2014/11/26 02:18:33.485020 [24278]: Freeze priority 2
2014/11/26 02:18:33.486973 [24278]: Freeze priority 3
2014/11/26 02:18:33.488978 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:34.397284 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:34.397351 [24278]: Thawing priority 1
2014/11/26 02:18:34.397372 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:34.397441 [24278]: Thawing priority 2
2014/11/26 02:18:34.397462 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:34.397509 [24278]: Thawing priority 3
2014/11/26 02:18:34.397528 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:34.398557 [set_recmode:25062]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:34.398792 [set_recmode:25062]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:34.458362 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:34.458421 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:34.482658 [24278]: Freeze priority 1
2014/11/26 02:18:34.485069 [24278]: Freeze priority 2
2014/11/26 02:18:34.487090 [24278]: Freeze priority 3
2014/11/26 02:18:34.488985 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:35.465354 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:35.465419 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:35.469512 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:35.469551 [24278]: Thawing priority 1
2014/11/26 02:18:35.469569 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:35.469634 [24278]: Thawing priority 2
2014/11/26 02:18:35.469674 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:35.469728 [24278]: Thawing priority 3
2014/11/26 02:18:35.469748 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:35.472200 [set_recmode:25128]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:35.472481 [set_recmode:25128]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:35.482005 [24278]: Freeze priority 1
2014/11/26 02:18:35.484422 [24278]: Freeze priority 2
2014/11/26 02:18:35.486461 [24278]: Freeze priority 3
2014/11/26 02:18:35.488391 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:36.439523 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:36.439591 [24278]: Thawing priority 1
2014/11/26 02:18:36.439612 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:36.439674 [24278]: Thawing priority 2
2014/11/26 02:18:36.439694 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:36.439742 [24278]: Thawing priority 3
2014/11/26 02:18:36.439761 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:36.440900 [set_recmode:25192]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:36.441047 [set_recmode:25192]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:36.465492 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:36.465537 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:36.483188 [24278]: Freeze priority 1
2014/11/26 02:18:36.485516 [24278]: Freeze priority 2
2014/11/26 02:18:36.487423 [24278]: Freeze priority 3
2014/11/26 02:18:36.489235 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:37.465972 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:37.466044 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:37.838846 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:37.838915 [24278]: Thawing priority 1
2014/11/26 02:18:37.838936 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:37.839006 [24278]: Thawing priority 2
2014/11/26 02:18:37.839027 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:37.839073 [24278]: Thawing priority 3
2014/11/26 02:18:37.839092 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:37.841480 [set_recmode:25258]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:37.841717 [set_recmode:25258]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:37.848835 [24278]: Freeze priority 1
2014/11/26 02:18:37.851313 [24278]: Freeze priority 2
2014/11/26 02:18:37.853452 [24278]: Freeze priority 3
2014/11/26 02:18:37.855766 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:39.505352 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:39.505418 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:39.510777 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:39.510819 [24278]: Thawing priority 1
2014/11/26 02:18:39.510836 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:39.510901 [24278]: Thawing priority 2
2014/11/26 02:18:39.510922 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:39.510969 [24278]: Thawing priority 3
2014/11/26 02:18:39.510988 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:39.513398 [set_recmode:25325]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:39.513630 [set_recmode:25325]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:39.524700 [24278]: Freeze priority 1
2014/11/26 02:18:39.527209 [24278]: Freeze priority 2
2014/11/26 02:18:39.529334 [24278]: Freeze priority 3
2014/11/26 02:18:39.531565 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:40.469736 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:40.469823 [24278]: Thawing priority 1
2014/11/26 02:18:40.469844 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:40.469913 [24278]: Thawing priority 2
2014/11/26 02:18:40.469935 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:40.469979 [24278]: Thawing priority 3
2014/11/26 02:18:40.469998 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:40.472418 [set_recmode:25389]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:40.472561 [set_recmode:25389]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:40.505476 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:40.505537 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:40.526582 [24278]: Freeze priority 1
2014/11/26 02:18:40.529260 [24278]: Freeze priority 2
2014/11/26 02:18:40.531499 [24278]: Freeze priority 3
2014/11/26 02:18:40.533982 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:41.459767 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:41.459834 [24278]: Thawing priority 1
2014/11/26 02:18:41.459854 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:41.459910 [24278]: Thawing priority 2
2014/11/26 02:18:41.459930 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:41.459979 [24278]: Thawing priority 3
2014/11/26 02:18:41.459998 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:41.461037 [set_recmode:25455]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:41.461272 [set_recmode:25455]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:41.506408 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:41.506465 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:41.525106 [24278]: Freeze priority 1
2014/11/26 02:18:41.527697 [24278]: Freeze priority 2
2014/11/26 02:18:41.530079 [24278]: Freeze priority 3
2014/11/26 02:18:41.532445 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:42.438472 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:42.438575 [24278]: Thawing priority 1
2014/11/26 02:18:42.438599 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:42.438657 [24278]: Thawing priority 2
2014/11/26 02:18:42.438677 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:42.438709 [24278]: Thawing priority 3
2014/11/26 02:18:42.438731 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:42.439821 [set_recmode:25518]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:42.440084 [set_recmode:25518]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:42.507060 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:42.507106 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:42.526582 [24278]: Freeze priority 1
2014/11/26 02:18:42.529253 [24278]: Freeze priority 2
2014/11/26 02:18:42.531375 [24278]: Freeze priority 3
2014/11/26 02:18:42.533570 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:43.458570 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:43.458642 [24278]: Thawing priority 1
2014/11/26 02:18:43.458661 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:43.458723 [24278]: Thawing priority 2
2014/11/26 02:18:43.458742 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:43.458790 [24278]: Thawing priority 3
2014/11/26 02:18:43.458810 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:43.459852 [set_recmode:25585]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:43.460079 [set_recmode:25585]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:43.507338 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:43.507392 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:43.528201 [24278]: Freeze priority 1
2014/11/26 02:18:43.530928 [24278]: Freeze priority 2
2014/11/26 02:18:43.533108 [24278]: Freeze priority 3
2014/11/26 02:18:43.535393 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:44.498434 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:44.498502 [24278]: Thawing priority 1
2014/11/26 02:18:44.498523 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:44.498591 [24278]: Thawing priority 2
2014/11/26 02:18:44.498614 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:44.498662 [24278]: Thawing priority 3
2014/11/26 02:18:44.498682 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:44.501089 [set_recmode:25648]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:44.501320 [set_recmode:25648]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:44.508406 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:44.508451 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:44.721005 [24278]: Freeze priority 1
2014/11/26 02:18:44.723705 [24278]: Freeze priority 2
2014/11/26 02:18:44.725917 [24278]: Freeze priority 3
2014/11/26 02:18:44.728214 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:45.508810 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:45.508883 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:46.002604 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:46.002673 [24278]: Thawing priority 1
2014/11/26 02:18:46.002694 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:46.002766 [24278]: Thawing priority 2
2014/11/26 02:18:46.002787 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:46.002836 [24278]: Thawing priority 3
2014/11/26 02:18:46.002856 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:46.003949 [set_recmode:25714]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:46.004185 [set_recmode:25714]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:46.013202 [24278]: Freeze priority 1
2014/11/26 02:18:46.015658 [24278]: Freeze priority 2
2014/11/26 02:18:46.017788 [24278]: Freeze priority 3
2014/11/26 02:18:46.019872 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:46.509935 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:46.510019 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:46.976586 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:46.976653 [24278]: Thawing priority 1
2014/11/26 02:18:46.976674 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:46.976737 [24278]: Thawing priority 2
2014/11/26 02:18:46.976758 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:46.976802 [24278]: Thawing priority 3
2014/11/26 02:18:46.976821 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:46.977865 [set_recmode:25777]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:46.978099 [set_recmode:25777]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:47.016557 [24278]: Freeze priority 1
2014/11/26 02:18:47.019027 [24278]: Freeze priority 2
2014/11/26 02:18:47.021386 [24278]: Freeze priority 3
2014/11/26 02:18:47.023518 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:47.510480 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:47.510563 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:48.045913 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:48.045981 [24278]: Thawing priority 1
2014/11/26 02:18:48.046002 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:48.046062 [24278]: Thawing priority 2
2014/11/26 02:18:48.046101 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:48.046148 [24278]: Thawing priority 3
2014/11/26 02:18:48.046167 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:48.047217 [set_recmode:25843]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:48.047458 [set_recmode:25843]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:48.058028 [24278]: Freeze priority 1
2014/11/26 02:18:48.060306 [24278]: Freeze priority 2
2014/11/26 02:18:48.062099 [24278]: Freeze priority 3
2014/11/26 02:18:48.063912 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:48.511107 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:48.511191 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:49.511314 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:49.511394 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:50.625441 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:50.625510 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:50.632290 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:50.632334 [24278]: Thawing priority 1
2014/11/26 02:18:50.632352 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:50.632416 [24278]: Thawing priority 2
2014/11/26 02:18:50.632437 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:50.632482 [24278]: Thawing priority 3
2014/11/26 02:18:50.632501 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:50.633501 [set_recmode:25909]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:50.633644 [set_recmode:25909]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:50.640736 [24278]: Freeze priority 1
2014/11/26 02:18:50.643574 [24278]: Freeze priority 2
2014/11/26 02:18:50.645741 [24278]: Freeze priority 3
2014/11/26 02:18:50.647963 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:51.599330 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:51.599399 [24278]: Thawing priority 1
2014/11/26 02:18:51.599419 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:51.599489 [24278]: Thawing priority 2
2014/11/26 02:18:51.599511 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:51.599559 [24278]: Thawing priority 3
2014/11/26 02:18:51.599580 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:51.600628 [set_recmode:25975]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:51.600890 [set_recmode:25975]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:51.625701 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:51.625748 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:51.642046 [24278]: Freeze priority 1
2014/11/26 02:18:51.644788 [24278]: Freeze priority 2
2014/11/26 02:18:51.647093 [24278]: Freeze priority 3
2014/11/26 02:18:51.650659 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:52.619849 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:52.619917 [24278]: Thawing priority 1
2014/11/26 02:18:52.619938 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:52.620000 [24278]: Thawing priority 2
2014/11/26 02:18:52.620020 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:52.620070 [24278]: Thawing priority 3
2014/11/26 02:18:52.620091 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:52.621182 [set_recmode:26038]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:52.621426 [set_recmode:26038]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:52.626043 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:52.626090 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:52.645986 [24278]: Freeze priority 1
2014/11/26 02:18:52.648780 [24278]: Freeze priority 2
2014/11/26 02:18:52.651079 [24278]: Freeze priority 3
2014/11/26 02:18:52.653541 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:53.619796 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:53.619864 [24278]: Thawing priority 1
2014/11/26 02:18:53.619884 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:53.619948 [24278]: Thawing priority 2
2014/11/26 02:18:53.619968 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:53.620016 [24278]: Thawing priority 3
2014/11/26 02:18:53.620035 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:53.621062 [set_recmode:26104]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:53.621201 [set_recmode:26104]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:53.626683 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:53.626728 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:53.650768 [24278]: Freeze priority 1
2014/11/26 02:18:53.653140 [24278]: Freeze priority 2
2014/11/26 02:18:53.655397 [24278]: Freeze priority 3
2014/11/26 02:18:53.657757 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:54.637071 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:54.637139 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:54.646464 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:54.646507 [24278]: Thawing priority 1
2014/11/26 02:18:54.646525 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:54.646581 [24278]: Thawing priority 2
2014/11/26 02:18:54.646600 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:54.646649 [24278]: Thawing priority 3
2014/11/26 02:18:54.646668 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:54.647729 [set_recmode:26167]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:54.647858 [set_recmode:26167]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:54.659283 [24278]: Freeze priority 1
2014/11/26 02:18:54.667671 [24278]: Freeze priority 2
2014/11/26 02:18:54.679243 [24278]: Freeze priority 3
2014/11/26 02:18:54.703918 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:55.637517 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:55.637593 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:56.010969 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:56.011058 [24278]: Thawing priority 1
2014/11/26 02:18:56.011077 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:56.011124 [24278]: Thawing priority 2
2014/11/26 02:18:56.011141 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:56.011185 [24278]: Thawing priority 3
2014/11/26 02:18:56.011204 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:56.013696 [set_recmode:26235]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:56.013820 [set_recmode:26235]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:56.023351 [24278]: Freeze priority 1
2014/11/26 02:18:56.025575 [24278]: Freeze priority 2
2014/11/26 02:18:56.027412 [24278]: Freeze priority 3
2014/11/26 02:18:56.029213 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:56.640845 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:56.640929 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:57.339805 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:57.339874 [24278]: Thawing priority 1
2014/11/26 02:18:57.339895 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:57.339960 [24278]: Thawing priority 2
2014/11/26 02:18:57.339980 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:57.340050 [24278]: Thawing priority 3
2014/11/26 02:18:57.340072 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:57.342503 [set_recmode:26299]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:57.342732 [set_recmode:26299]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:57.349962 [24278]: Freeze priority 1
2014/11/26 02:18:57.352506 [24278]: Freeze priority 2
2014/11/26 02:18:57.354573 [24278]: Freeze priority 3
2014/11/26 02:18:57.356837 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:57.641760 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:57.641826 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:58.297516 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:58.297584 [24278]: Thawing priority 1
2014/11/26 02:18:58.297605 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:58.297676 [24278]: Thawing priority 2
2014/11/26 02:18:58.297698 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:58.297746 [24278]: Thawing priority 3
2014/11/26 02:18:58.297765 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:58.300230 [set_recmode:26364]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:58.300462 [set_recmode:26364]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:58.353347 [24278]: Freeze priority 1
2014/11/26 02:18:58.355772 [24278]: Freeze priority 2
2014/11/26 02:18:58.357737 [24278]: Freeze priority 3
2014/11/26 02:18:58.359642 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:58.642080 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:58.642148 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:18:59.376428 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:18:59.376491 [24278]: Thawing priority 1
2014/11/26 02:18:59.376510 [24278]: Release freeze handler for prio 1
2014/11/26 02:18:59.376573 [24278]: Thawing priority 2
2014/11/26 02:18:59.376595 [24278]: Release freeze handler for prio 2
2014/11/26 02:18:59.376637 [24278]: Thawing priority 3
2014/11/26 02:18:59.376655 [24278]: Release freeze handler for prio 3
2014/11/26 02:18:59.379058 [set_recmode:26430]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:18:59.379196 [set_recmode:26430]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:18:59.391901 [24278]: Freeze priority 1
2014/11/26 02:18:59.394288 [24278]: Freeze priority 2
2014/11/26 02:18:59.396198 [24278]: Freeze priority 3
2014/11/26 02:18:59.398093 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:18:59.642226 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:18:59.642295 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:00.755454 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:00.755520 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:00.760542 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:00.760584 [24278]: Thawing priority 1
2014/11/26 02:19:00.760603 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:00.760673 [24278]: Thawing priority 2
2014/11/26 02:19:00.760694 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:00.760742 [24278]: Thawing priority 3
2014/11/26 02:19:00.760762 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:00.763233 [set_recmode:26494]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:00.763544 [set_recmode:26494]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:00.772169 [24278]: Freeze priority 1
2014/11/26 02:19:00.774695 [24278]: Freeze priority 2
2014/11/26 02:19:00.776792 [24278]: Freeze priority 3
2014/11/26 02:19:00.779084 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:01.756298 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:01.756379 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:02.100516 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:02.100582 [24278]: Thawing priority 1
2014/11/26 02:19:02.100601 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:02.100646 [24278]: Thawing priority 2
2014/11/26 02:19:02.100663 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:02.100713 [24278]: Thawing priority 3
2014/11/26 02:19:02.100732 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:02.103084 [set_recmode:26561]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:02.103216 [set_recmode:26561]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:02.111061 [24278]: Freeze priority 1
2014/11/26 02:19:02.113524 [24278]: Freeze priority 2
2014/11/26 02:19:02.115614 [24278]: Freeze priority 3
2014/11/26 02:19:02.117911 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:03.025417 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:03.025487 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:03.048310 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:03.048355 [24278]: Thawing priority 1
2014/11/26 02:19:03.048374 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:03.048436 [24278]: Thawing priority 2
2014/11/26 02:19:03.048455 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:03.048497 [24278]: Thawing priority 3
2014/11/26 02:19:03.048515 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:03.050884 [set_recmode:26626]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:03.051127 [set_recmode:26626]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:03.110322 [24278]: Freeze priority 1
2014/11/26 02:19:03.112936 [24278]: Freeze priority 2
2014/11/26 02:19:03.115090 [24278]: Freeze priority 3
2014/11/26 02:19:03.117315 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:04.026464 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:04.026535 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:04.068640 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:04.068685 [24278]: Thawing priority 1
2014/11/26 02:19:04.068704 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:04.068752 [24278]: Thawing priority 2
2014/11/26 02:19:04.068769 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:04.068821 [24278]: Thawing priority 3
2014/11/26 02:19:04.068841 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:04.071906 [set_recmode:26692]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:04.072153 [set_recmode:26692]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:04.111109 [24278]: Freeze priority 1
2014/11/26 02:19:04.113547 [24278]: Freeze priority 2
2014/11/26 02:19:04.115708 [24278]: Freeze priority 3
2014/11/26 02:19:04.117913 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:05.045419 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:05.045488 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:05.058330 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:05.058375 [24278]: Thawing priority 1
2014/11/26 02:19:05.058394 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:05.058455 [24278]: Thawing priority 2
2014/11/26 02:19:05.058475 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:05.058524 [24278]: Thawing priority 3
2014/11/26 02:19:05.058543 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:05.060912 [set_recmode:26756]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:05.061168 [set_recmode:26756]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:05.112803 [24278]: Freeze priority 1
2014/11/26 02:19:05.115335 [24278]: Freeze priority 2
2014/11/26 02:19:05.117451 [24278]: Freeze priority 3
2014/11/26 02:19:05.119623 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:06.046459 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:06.046546 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:06.510657 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:06.510723 [24278]: Thawing priority 1
2014/11/26 02:19:06.510743 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:06.510811 [24278]: Thawing priority 2
2014/11/26 02:19:06.510831 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:06.510880 [24278]: Thawing priority 3
2014/11/26 02:19:06.510899 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:06.513282 [set_recmode:26821]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:06.513525 [set_recmode:26821]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:06.521988 [24278]: Freeze priority 1
2014/11/26 02:19:06.524628 [24278]: Freeze priority 2
2014/11/26 02:19:06.526747 [24278]: Freeze priority 3
2014/11/26 02:19:06.528960 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:07.052559 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:07.052642 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:07.800460 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:07.800526 [24278]: Thawing priority 1
2014/11/26 02:19:07.800546 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:07.800617 [24278]: Thawing priority 2
2014/11/26 02:19:07.800638 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:07.800684 [24278]: Thawing priority 3
2014/11/26 02:19:07.800704 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:07.803158 [set_recmode:26887]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:07.803425 [set_recmode:26887]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:07.811767 [24278]: Freeze priority 1
2014/11/26 02:19:07.814986 [24278]: Freeze priority 2
2014/11/26 02:19:07.817230 [24278]: Freeze priority 3
2014/11/26 02:19:07.819499 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:08.052824 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:08.052896 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:08.777590 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:08.777658 [24278]: Thawing priority 1
2014/11/26 02:19:08.777677 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:08.777738 [24278]: Thawing priority 2
2014/11/26 02:19:08.777757 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:08.777796 [24278]: Thawing priority 3
2014/11/26 02:19:08.777814 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:08.783330 [set_recmode:26950]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:08.783515 [set_recmode:26950]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:08.812268 [24278]: Freeze priority 1
2014/11/26 02:19:08.815006 [24278]: Freeze priority 2
2014/11/26 02:19:08.817281 [24278]: Freeze priority 3
2014/11/26 02:19:08.819595 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:09.053071 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:09.053137 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:11.965420 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:11.965485 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:11.970382 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:11.970468 [24278]: Thawing priority 1
2014/11/26 02:19:11.970488 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:11.970554 [24278]: Thawing priority 2
2014/11/26 02:19:11.970575 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:11.970624 [24278]: Thawing priority 3
2014/11/26 02:19:11.970642 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:11.973061 [set_recmode:27019]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:11.973329 [set_recmode:27019]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:11.980534 [24278]: Freeze priority 1
2014/11/26 02:19:11.983222 [24278]: Freeze priority 2
2014/11/26 02:19:11.985510 [24278]: Freeze priority 3
2014/11/26 02:19:11.987840 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:13.055315 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:13.055384 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:13.060579 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:13.060624 [24278]: Thawing priority 1
2014/11/26 02:19:13.060643 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:13.060702 [24278]: Thawing priority 2
2014/11/26 02:19:13.060723 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:13.060764 [24278]: Thawing priority 3
2014/11/26 02:19:13.060783 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:13.063158 [set_recmode:27083]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:13.063293 [set_recmode:27083]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:13.271255 [24278]: Freeze priority 1
2014/11/26 02:19:13.274248 [24278]: Freeze priority 2
2014/11/26 02:19:13.276440 [24278]: Freeze priority 3
2014/11/26 02:19:13.278998 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:14.185386 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:14.185450 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:14.198451 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:14.198495 [24278]: Thawing priority 1
2014/11/26 02:19:14.198512 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:14.198581 [24278]: Thawing priority 2
2014/11/26 02:19:14.198603 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:14.198649 [24278]: Thawing priority 3
2014/11/26 02:19:14.198669 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:14.201060 [set_recmode:27148]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:14.201293 [set_recmode:27148]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:14.209469 [24278]: Freeze priority 1
2014/11/26 02:19:14.212705 [24278]: Freeze priority 2
2014/11/26 02:19:14.214864 [24278]: Freeze priority 3
2014/11/26 02:19:14.217367 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:15.186178 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:15.186259 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:15.680163 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:15.680230 [24278]: Thawing priority 1
2014/11/26 02:19:15.680250 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:15.680314 [24278]: Thawing priority 2
2014/11/26 02:19:15.680335 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:15.680378 [24278]: Thawing priority 3
2014/11/26 02:19:15.680396 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:15.682848 [set_recmode:27214]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:15.683111 [set_recmode:27214]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:15.691930 [24278]: Freeze priority 1
2014/11/26 02:19:15.694730 [24278]: Freeze priority 2
2014/11/26 02:19:15.696977 [24278]: Freeze priority 3
2014/11/26 02:19:15.699277 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:16.187011 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:16.187091 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:16.758749 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:16.758820 [24278]: Thawing priority 1
2014/11/26 02:19:16.758841 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:16.758906 [24278]: Thawing priority 2
2014/11/26 02:19:16.758926 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:16.758955 [24278]: Thawing priority 3
2014/11/26 02:19:16.758971 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:16.760901 [set_recmode:27276]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:16.761036 [set_recmode:27276]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:16.769661 [24278]: Freeze priority 1
2014/11/26 02:19:16.772266 [24278]: Freeze priority 2
2014/11/26 02:19:16.774795 [24278]: Freeze priority 3
2014/11/26 02:19:16.777237 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:17.304905 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:17.304985 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:18.170398 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:18.170465 [24278]: Thawing priority 1
2014/11/26 02:19:18.170486 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:18.170544 [24278]: Thawing priority 2
2014/11/26 02:19:18.170563 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:18.170613 [24278]: Thawing priority 3
2014/11/26 02:19:18.170632 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:18.173120 [set_recmode:27344]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:18.173257 [set_recmode:27344]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:18.181788 [24278]: Freeze priority 1
2014/11/26 02:19:18.184319 [24278]: Freeze priority 2
2014/11/26 02:19:18.186498 [24278]: Freeze priority 3
2014/11/26 02:19:18.188842 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:18.305747 [24278]: CTDB_WAIT_UNTIL_RECOVERED
2014/11/26 02:19:18.305812 [24278]: server/ctdb_monitor.c:293 in recovery.
Wait one more second
2014/11/26 02:19:19.115811 [24278]: Received SHUTDOWN command.
2014/11/26 02:19:19.115874 [24278]: Shutdown sequence commencing.
2014/11/26 02:19:19.115894 [24278]: Set runstate to SHUTDOWN (6)
2014/11/26 02:19:19.115910 [24278]: Shutting down recovery daemon
2014/11/26 02:19:19.115935 [24278]: Monitoring has been stopped
2014/11/26 02:19:19.115997 [24278]: server/ctdb_takeover.c:3303 Released 0
public IPs
2014/11/26 02:19:19.118506 [24278]: server/ctdb_recover.c:612 Recovery mode
set to NORMAL
2014/11/26 02:19:19.118548 [24278]: Thawing priority 1
2014/11/26 02:19:19.118565 [24278]: Release freeze handler for prio 1
2014/11/26 02:19:19.118621 [24278]: Thawing priority 2
2014/11/26 02:19:19.118641 [24278]: Release freeze handler for prio 2
2014/11/26 02:19:19.118680 [24278]: Thawing priority 3
2014/11/26 02:19:19.118698 [24278]: Release freeze handler for prio 3
2014/11/26 02:19:19.120947 [set_recmode:27428]: ctdb_recovery_lock: Got
recovery lock on '/amtb/ctdb.lock'
2014/11/26 02:19:19.121246 [set_recmode:27428]: ERROR: recovery lock file
/amtb/ctdb.lock not locked when recovering!
2014/11/26 02:19:19.182627 [24278]: Freeze priority 1
2014/11/26 02:19:19.185153 [24278]: Freeze priority 2
2014/11/26 02:19:19.187311 [24278]: Freeze priority 3
2014/11/26 02:19:19.189720 [24278]: server/ctdb_recover.c:988 startrecovery
eventscript has been invoked
2014/11/26 02:19:19.335622 [24278]: 50.samba: * WARNING: samba is already
stopped
2014/11/26 02:19:19.442717 [24278]: Shutdown sequence complete, exiting.
2014/11/26 02:19:19.442811 [24278]: CTDB daemon shutting down
2014/11/26 02:19:19.443689 [24278]: Removed PID file /var/run/ctdb/ctdbd.pid
On Tue, Nov 25, 2014 at 1:49 AM, Richard Sharpe <realrichardsharpe at gmail.com
> wrote:
> On Mon, Nov 24, 2014 at 7:43 AM, Rowland Penny <repenny241155 at gmail.com>
> wrote:
> > On 24/11/14 15:38, Volker Lendecke wrote:
> >>
> >> On Mon, Nov 24, 2014 at 03:35:06PM +0000, Rowland Penny wrote:
> >>>
> >>> On 24/11/14 15:24, Richard Sharpe wrote:
> >>>>
> >>>> On Sun, Nov 23, 2014 at 9:31 PM, Min Wai Chan <dcmwai at gmail.com>
> wrote:
> >>>>>
> >>>>> Dear Steve,
> >>>>>
> >>>>> Ok ok, Will send it over then there are on one login to the server
> >>>>> maybe
> >>>>> during mid night.
> >>>>>
> >>>>> I saw something about the ping_ping -RW test
> >>>>> something like below?
> >>>>>
> >>>>> ping_pong -rw test.dat N
> >>>>>
> >>>>> Do you need this test result as well?
> >>>>>
> >>>>> I just wonder if we mount the ocfs2 wrong?
> >>>>> If there is a way to mount ocfs2 via user space?
> >>>>
> >>>> It is not an issue of mounting ocfs2 incorrectly, I believe.
> >>>>
> >>>> Rather, you have to have their user-space posix locking daemon
> running.
> >>>>
> >>> I think I see a possible answer, does this mean that you would get
> >>> two locking daemons, the CTDB one AND the OCFS2 one ??
> >>
> >> Assuming ocfs2 really requires a daemon to do fcntl locking,
> >> then yes, there will be two daemons. ctdb will rely on
> >> ocfs2's daemon for the reclockfile. Both serve different
> >> purposes.
> >>
> >> Volker
> >>
> > OK, but would/could they work together, or could they interfere with each
> > other and cause problems with the smbd daemon i.e stop it starting.
>
> Firstly, Ralph is correct. ctdb does not use a locking daemon.
>
> What happens is that the ctdb recovery daemon tries to take out an
> fcntl lock on an ocfs2 file in the cluster. This is sent to their
> user-space lock daemon that uses Corosync Closed Process Groups to
> implement their distributed lock manager. If their DLM is not running
> or not properly configured you will have problems and ctdb will be
> unable to come up correctly.
>
> They are two separate daemons performing separate functions.
>
> I do not have an OCFS2 setup but it would be good to get to the bottom of
> this.
>
> --
> Regards,
> Richard Sharpe
> (何以解憂？唯有杜康。--曹操)
>
More information about the samba-technical
mailing list