Debugging two simple ctdb problems

[Steve French]

In looking at a simple ctdb two node configuration (two Samba servers
exporting the same clustered share), I ran into two hopefully easy to
debug problems - but would like some suggestions on how to attack
debugging this type of problem.

After setting up the /etc/ctdb/nodes files on each server (with the ip
addresses of the 2nd NIC for each of the two servers) and an empty
public addresses file (and updating smb.conf to enable clustering), I
started ctdb on each - and saw ctdb status display both nodes as
healthy and ctdb getdbmap showed locking.tdb and brlock.tdb but I
didn't see deny-write enforced (I mounted r: to \\server1\share and s:
to \\server2\share from Windows then I opened deny shared write and
wrote to the same file ie r:\file and s:\file at the same time - which
shouldn't have worked, and doesn't if both opens are to the same samba
server).  Any suggestions on what to look for in debugging this simple
issue?

Later the VM for one of the two Samba servers got corrupted, so I
installed a 3rd system, updated the /etc/ctdb/nodes files to remove
the unhealth one, and include the ip addresses for only the two
healthy servers, then restarted ctdb on the two healthy nodes - but
ctdb node status always continues to show one of the two as unhealthy.
They can ping each other. Any ideas what to look for in debugging
this?

Rebooting both systems, the systems come up, ctdb and samba starts as
expected, but ctdb status shows unhealthy for both now.

Ideas on how to approach debugging unhealthy ctdb nodes that can ping
each other, and non-enforcement of deny-modes in a healthy ctdb
cluster?

-- 
Thanks,

Steve