[Samba] smbd does not start under ctdb

[steve]

On Thu, 2014-07-10 at 14:33 +0200, David Disseldorp wrote:
> On Thu, 10 Jul 2014 11:41:23 +0200, steve wrote:
> 
> > Hi. OK. We've now tried that. We're on openSUSE 13.1. With the stock 2.3
> > ctdb:
> > Nothing is started.
> > 
> > With 2.5.3 from the ha repo indeed, smb starts, but both nodes are
> > reported as UNHEALTHY. Always.
> 
> Are there errors or similar in the smbd or ctdbd logs?
TIA for any advice:

Restarted ctdb:
primary:secondary with cluster ocfs2 mounted:

2014/07/11 08:02:19.331410 [18680]: Running the "startup" event.
2014-07-11T08:02:20.026136+02:00 smb1 systemd[1]: Stopped Samba SMB
Daemon.
2014-07-11T08:02:20.418811+02:00 smb1 systemd[1]: Stopped Samba NMB
Daemon.
2014/07/11 08:02:20.785233 [18680]: 50.samba: messaging_init failed
2014/07/11 08:02:20.785577 [18680]: 50.samba: failed to attach to ctdb
serverid.tdb
2014-07-11T08:02:21.202349+02:00 smb1 systemd[1]: Starting Samba NMB
Daemon...
2014-07-11T08:02:21.402868+02:00 smb1 systemd[1]: nmb.service: main
process exited, code=exited, status=1/FAILURE
2014-07-11T08:02:21.408960+02:00 smb1 systemd[1]: Failed to start Samba
NMB Daemon.
2014/07/11 08:02:21.414374 [18680]: 50.samba: Job for nmb.service
failed. See 'systemctl status nmb.service' and 'journalctl -xn' for
details.
2014/07/11 08:02:21.419886 [18680]: 50.samba: Failed to start nmbd
2014/07/11 08:02:21.421543 [18680]: startup event failed
2014-07-11T08:02:21.444883+02:00 smb1 systemd[1]: Unit nmb.service
entered failed state.
2014/07/11 08:02:26.422281 [18680]: Running the "startup" event.
2014-07-11T08:02:27.123540+02:00 smb1 systemd[1]: Stopped Samba SMB
Daemon.
2014-07-11T08:02:27.540738+02:00 smb1 systemd[1]: Stopped Samba NMB
Daemon.
2014/07/11 08:02:27.717057 [18680]: 50.samba: messaging_init failed
2014/07/11 08:02:27.717227 [18680]: 50.samba: failed to attach to ctdb
serverid.tdb
2014-0

primary:primary, ocfs2 mounted on both:
node 1:
2014/07/11 08:18:28.823617 [27437]: CTDB_WAIT_UNTIL_RECOVERED
2014/07/11 08:18:28.823878 [27437]: server/ctdb_monitor.c:293 in
recovery. Wait one more second
2014/07/11 08:18:28.846918 [27437]: server/ctdb_recover.c:612 Recovery
mode set to NORMAL
2014/07/11 08:18:28.847008 [27437]: Thawing priority 1
2014/07/11 08:18:28.847033 [27437]: Release freeze handler for prio 1
2014/07/11 08:18:28.847301 [27437]: Thawing priority 2
2014/07/11 08:18:28.847337 [27437]: Release freeze handler for prio 2
2014/07/11 08:18:28.847480 [27437]: Thawing priority 3
2014/07/11 08:18:28.847551 [27437]: Release freeze handler for prio 3
2014/07/11 08:18:28.865342 [set_recmode:27959]: ctdb_recovery_lock: Got
recovery lock on '/cluster/ctdb/lockfile'
2014/07/11 08:18:28.871862 [set_recmode:27959]: ERROR: recovery lock
file /cluster/ctdb/lockfile not locked when recovering!
2014/07/11 08:18:29.014265 [27437]: Freeze priority 1
2014/07/11 08:18:29.029978 [27437]: Freeze priority 2
2014/07/11 08:18:29.044783 [27437]: Freeze priority 3
2014/07/11 08:18:29.063558 [27437]: server/ctdb_recover.c:988
startrecovery eventscript has been invoked
2014/07/11 08:18:29.825326 [27437]: CTDB_WAIT_UNTIL_RECOVERED
2014/07/11 08:18:29.825635 [27437]: server/ctdb_monitor.c:293 in
recovery. Wait one more second
2014/07/11 08:18:30.826668 [27437]: CTDB_WAIT_UNTIL_RECOVERED
2014/07/11 08:18:30.827093 [27437]: server/ctdb_monitor.c:293 in
recovery. Wait one more second
2014/07/11 08:18:31.660176 [27437]: server/ctdb_recover.c:612 Recovery
mode set to NORMAL
2014/07/11 08:18:31.660484 [27437]: Thawing priority 1
2014/07/11 08:18:31.660519 [27437]: Release freeze handler for prio 1
2014/07/11 08:18:31.660771 [27437]: Thawing priority 2
2014/07/11 08:18:31.660809 [27437]: Release freeze handler for prio 2
2014/07/11 08:18:31.660959 [27437]: Thawing priority 3
2014/07/11 08:18:31.661021 [27437]: Release freeze handler for prio 3
2014/07/11 08:18:31.676584 [set_recmode:28005]: ctdb_recovery_lock: Got
recovery lock on '/cluster/ctdb/lockfile'
2014/07/11 08:18:31.678640 [set_recmode:28005]: ERROR: recovery lock
file /cluster/ctdb/lockfile not locked when recovering!
2014/07/11 08:18:31.830130 [27437]: CTDB_WAIT_UNTIL_RECOVERED
2014/07/11 08:18:31.830833 [27437]: server/ctdb_monitor.c:293 in
recovery. Wait one more second
2014/07/11 08:18:31.855559 [27437]: Freeze priority 1
2014/07/11 08:18:31.872937 [27437]: Freeze priority 2
2014/07/11 08:18:31.888767 [27437]: Freeze priority 3
2014/07/11 08:18:31.907771 [27437]: server/ctdb_recover.c:988
startrecovery eventscript has been invoked

node 2:
2014/07/11 08:20:59.249970 [recoverd:18545]: Pull persistent
db:0x3ef19640 from node 1 with highest seqnum:1
2014/07/11 08:20:59.254829 [recoverd:18545]: server/ctdb_recoverd.c:1453
Recovery - pulled remote database 0x3ef19640
2014/07/11 08:20:59.300397 [recoverd:18545]: server/ctdb_recoverd.c:1414
Recovery - pushed remote database 0x3ef19640 of size 4
2014/07/11 08:20:59.353466 [recoverd:18545]: Scan for highest seqnum pdb
for db:0x7132c184
2014/07/11 08:20:59.365020 [18393]: Running the "startup" event.
2014/07/11 08:20:59.365438 [18393]: Refusing to run event scripts call
'startup' while in recovery
2014/07/11 08:20:59.365498 [18393]: Unable to launch startup event
script

sysconfig:
CTDB_RECOVERY_LOCK="/cluster/ctdb/lockfile"
CTDB_PUBLIC_INTERFACE=enp0s3
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_LVS_PUBLIC_IP=
CTDB_MANAGES_SAMBA=yes
CTDB_SAMBA_SKIP_SHARE_CHECK=yes
CTDB_NFS_SKIP_SHARE_CHECK=yes
CTDB_MANAGES_WINBIND=yes
CTDB_MANAGES_VSFTPD=no
CTDB_MANAGES_ISCSI=no
CTDB_MANAGES_NFS=no
CTDB_MANAGES_HTTPD=no
CTDB_INIT_STYLE=
CTDB_SERVICE_SMB=smb
CTDB_SERVICE_WINBIND=winbind
CTDB_NODES=/etc/ctdb/nodes
CTDB_NOTIFY_SCRIPT=/etc/ctdb/notify.sh
CTDB_DBDIR=/var/lib/ctdb
CTDB_DBDIR_PERSISTENT=/var/lib/ctdb/persistent
CTDB_EVENT_SCRIPT_DIR=/etc/ctdb/events.d
CTDB_SOCKET=/var/lib/ctdb/ctdb.socket
CTDB_TRANSPORT="tcp"
CTDB_MONITOR_FREE_MEMORY=100
CTDB_START_AS_DISABLED="yes"
CTDB_CAPABILITY_RECMASTER=yes
CTDB_CAPABILITY_LMASTER=yes
NATGW_PUBLIC_IP=
NATGW_PUBLIC_IFACE=
NATGW_DEFAULT_GATEWAY=
NATGW_PRIVATE_IFACE=
NATGW_PRIVATE_NETWORK=
NATGW_NODES=/etc/ctdb/natgw_nodes
CTDB_LOGFILE=/var/log/messages
CTDB_DEBUGLEVEL=2
CTDB_OPTIONS=

smb.conf not yet joined to the domain:
[global]
workgroup = HH3
netbios name = SMBCLUSTER
realm = HH3.SITE
security = ADS
kerberos method = secrets and keytab 
winbind enum users = Yes
winbind enum groups = Yes
winbind use default domain = Yes
winbind nss info = rfc2307
idmap config * : backend = tdb
idmap config * : range = 19900-19999
idmap config HH3 : backend  = ad
idmap config HH3 : range = 20000-4000000
idmap config HH3 : schema_mode = rfc2307
clustering = Yes
ctdbd socket = /var/lib/ctdb/ctdb.socket

[users]
path = /cluster/users
read only = No