[Samba] ctdb + gluster9 = not working

[Anton Shevtsov]

29.12.2024 04:25, Martin Schwenke via samba пишет:
> Hi Anton,
>
> On Fri, 27 Dec 2024 20:04:07 +0500, Anton Shevtsov via samba
> <samba at lists.samba.org> wrote:
>
>> Hi, i tried make a samba cluster (ctdb) with gluster9 (do not use vfs
>> module)
>>
>> But i get an errors.
>>
>> [root at samba1 ctdb]# rpm -q samba samba-ctdb
>> samba-4.19.9-alt3.x86_64
>> samba-ctdb-4.19.9-alt3.x86_64
>>
>> [root at samba1 ctdb]# grep -v '^#' /etc/ctdb/ctdb.conf
>> [cluster]
>> recovery lock = "/mnt/gluster/ctdb/.ctdb.lock"
> [...]
>
>> дек 27 19:46:54 samba1 systemd[1]: Starting CTDB...
>> дек 27 19:46:54 samba1 ctdbd[25430]: CTDB logging to location
>> file:/var/log/log.ctdb
>> дек 27 19:46:54 samba1 systemd[1]: Started CTDB.
>>
>> [root at samba1 ctdb]# tail -f /var/log/log.ctdb
>> [...]
>> 2024-12-27T19:47:24.597406+05:00 samba1 ctdb-recoverd[25443]: Start election
>> 2024-12-27T19:47:24.598139+05:00 samba1 ctdb-recoverd[25443]: Attempting to take cluster lock ("/mnt/gluster/ctdb/.ctdb.lock")
>> 2024-12-27T19:47:24.611930+05:00 samba1 ctdbd[25431]: ctdb_mutex_fcntl_helper: Unable to open "/mnt/gluster/ctdb/.ctdb.lock" - (No such file or directory)
>> 2024-12-27T19:47:24.612311+05:00 samba1 ctdb-recoverd[25443]: Unable to take cluster lock - unknown error
>> [...]
>> Why?
> Good question!  This took me a few guesses... because it certainly isn't
> obvious! :-)
>
> You don't need the double-quotes around the recovery lock setting.
> They are not inserted by the log messages, so must be maintained from
> ctdb.conf and not stripped.  Samba (and CTDB) log files can handle
> values containing whitespace, so quoting isn't required.
>
> If your Linux distribution included the quotes in the sample
> configuration file then you should report a bug because the example is
> misleading.  If not, you're just unlucky because you did something sane
> but it didn't work.
>
> peace & happiness,
> martin

So simple! ;)))) Thanks!

I can make a next step.

[root at samba1 ~] systemctl start ctdb

[root at samba2 ~] systemctl start ctdb

[root at samba2 ~]# ctdb status
Number of nodes:2
pnn:0 192.168.160.151  OK
pnn:1 192.168.160.152  OK (THIS NODE)
Generation:1076686332
Size:2
hash:0 lmaster:0
hash:1 lmaster:1

[root at samba1 ctdb]# tail -f /var/log/log.ctdb

[root at samba1 ~]# tail -f /var/log/log.ctdb
2025-01-14T12:02:34.731080+05:00 samba1 ctdbd[6033]: Freeze db: 
account_policy.tdb
2025-01-14T12:02:34.735230+05:00 samba1 ctdbd[6033]: Freeze db: 
group_mapping.tdb
2025-01-14T12:02:34.739394+05:00 samba1 ctdbd[6033]: Freeze db: 
share_info.tdb
2025-01-14T12:02:34.743590+05:00 samba1 ctdbd[6033]: Freeze db: 
registry.tdb
2025-01-14T12:02:34.747753+05:00 samba1 ctdbd[6033]: Set runstate to 
SETUP (2)
2025-01-14T12:02:34.763874+05:00 samba1 ctdbd[6033]: Keepalive 
monitoring has been started
2025-01-14T12:02:34.764010+05:00 samba1 ctdbd[6033]: Set runstate to 
FIRST_RECOVERY (3)
2025-01-14T12:02:34.764520+05:00 samba1 ctdb-recoverd[6045]: 
monitor_cluster starting
2025-01-14T12:02:35.764460+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:36.765546+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:37.766605+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:38.767048+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:39.767644+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:39.768239+05:00 samba1 ctdb-recoverd[6045]: Leader 
broadcast timeout
2025-01-14T12:02:39.768764+05:00 samba1 ctdb-recoverd[6045]: Start election
2025-01-14T12:02:39.769124+05:00 samba1 ctdb-recoverd[6045]: Attempting 
to take cluster lock (/mnt/gluster/ctdb.lock)
2025-01-14T12:02:39.769310+05:00 samba1 ctdb-recoverd[6045]: Set cluster 
mutex helper to "/usr/lib/ctdb/ctdb_mutex_fcntl_helper"
2025-01-14T12:02:39.789394+05:00 samba1 ctdb-recoverd[6045]: Cluster 
lock taken successfully
2025-01-14T12:02:39.789657+05:00 samba1 ctdb-recoverd[6045]: Took 
cluster lock, leader=0
2025-01-14T12:02:39.790853+05:00 samba1 ctdb-recoverd[6045]: Node:0 was 
in recovery mode. Start recovery process
2025-01-14T12:02:39.791059+05:00 samba1 ctdb-recoverd[6045]: 
../../ctdb/server/ctdb_recoverd.c:1314 Starting do_recovery
2025-01-14T12:02:39.791262+05:00 samba1 ctdb-recoverd[6045]: 
../../ctdb/server/ctdb_recoverd.c:1358 Recovery initiated due to problem 
with node 0
2025-01-14T12:02:39.791741+05:00 samba1 ctdb-recoverd[6045]: 
../../ctdb/server/ctdb_recoverd.c:1388 Recovery - updated flags
2025-01-14T12:02:39.791946+05:00 samba1 ctdb-recoverd[6045]: Set 
recovery_helper to "/usr/lib/ctdb/ctdb_recovery_helper"
2025-01-14T12:02:39.800805+05:00 samba1 ctdb-recovery[6051]: Set 
recovery mode to ACTIVE
2025-01-14T12:02:39.801001+05:00 samba1 ctdbd[6033]: Recovery has started
2025-01-14T12:02:39.801254+05:00 samba1 ctdb-recovery[6051]: 
start_recovery event finished
2025-01-14T12:02:39.801421+05:00 samba1 ctdb-recovery[6051]: updated VNNMAP
2025-01-14T12:02:39.801492+05:00 samba1 ctdb-recovery[6051]: recover 
database 0x7132c184
2025-01-14T12:02:39.801571+05:00 samba1 ctdb-recovery[6051]: recover 
database 0x3ef19640
2025-01-14T12:02:39.801678+05:00 samba1 ctdb-recovery[6051]: recover 
database 0x2ca251cf
2025-01-14T12:02:39.801742+05:00 samba1 ctdb-recovery[6051]: recover 
database 0xa1413774
2025-01-14T12:02:39.801800+05:00 samba1 ctdb-recovery[6051]: recover 
database 0xc3078fba
2025-01-14T12:02:39.801862+05:00 samba1 ctdb-recovery[6051]: recover 
database 0x6cf2837d
2025-01-14T12:02:39.802221+05:00 samba1 ctdbd[6033]: Freeze db: 
secrets.tdb frozen
2025-01-14T12:02:39.802309+05:00 samba1 ctdbd[6033]: Freeze db: 
passdb.tdb frozen
2025-01-14T12:02:39.802351+05:00 samba1 ctdbd[6033]: Freeze db: 
account_policy.tdb frozen
2025-01-14T12:02:39.802379+05:00 samba1 ctdbd[6033]: Freeze db: 
group_mapping.tdb frozen
2025-01-14T12:02:39.802403+05:00 samba1 ctdbd[6033]: Freeze db: 
share_info.tdb frozen
2025-01-14T12:02:39.802488+05:00 samba1 ctdbd[6033]: Freeze db: 
registry.tdb frozen
2025-01-14T12:02:40.652978+05:00 samba1 ctdbd[6033]: Thaw db: 
secrets.tdb generation 1827381565
2025-01-14T12:02:40.653088+05:00 samba1 ctdbd[6033]: Release freeze 
handle for db secrets.tdb
2025-01-14T12:02:40.653189+05:00 samba1 ctdbd[6033]: Thaw db: passdb.tdb 
generation 1827381565
2025-01-14T12:02:40.653226+05:00 samba1 ctdbd[6033]: Release freeze 
handle for db passdb.tdb
2025-01-14T12:02:40.653283+05:00 samba1 ctdbd[6033]: Thaw db: 
account_policy.tdb generation 1827381565
2025-01-14T12:02:40.653323+05:00 samba1 ctdbd[6033]: Release freeze 
handle for db account_policy.tdb
2025-01-14T12:02:40.653397+05:00 samba1 ctdbd[6033]: Thaw db: 
group_mapping.tdb generation 1827381565
2025-01-14T12:02:40.653441+05:00 samba1 ctdbd[6033]: Release freeze 
handle for db group_mapping.tdb
2025-01-14T12:02:40.653660+05:00 samba1 ctdbd[6033]: Thaw db: 
share_info.tdb generation 1827381565
2025-01-14T12:02:40.653730+05:00 samba1 ctdbd[6033]: Release freeze 
handle for db share_info.tdb
2025-01-14T12:02:40.655336+05:00 samba1 ctdbd[6033]: Thaw db: 
registry.tdb generation 1827381565
2025-01-14T12:02:40.655398+05:00 samba1 ctdbd[6033]: Release freeze 
handle for db registry.tdb
2025-01-14T12:02:40.658394+05:00 samba1 ctdb-recovery[6051]: 6 of 6 
databases recovered
2025-01-14T12:02:40.658577+05:00 samba1 ctdbd[6033]: Recovery mode set 
to NORMAL
2025-01-14T12:02:40.658666+05:00 samba1 ctdbd[6033]: Set cluster mutex 
helper to "/usr/lib/ctdb/ctdb_mutex_fcntl_helper"
2025-01-14T12:02:40.677257+05:00 samba1 ctdb-recovery[6051]: Set 
recovery mode to NORMAL
2025-01-14T12:02:40.677419+05:00 samba1 ctdbd[6033]: Recovery has finished
2025-01-14T12:02:40.678242+05:00 samba1 ctdbd[6033]: Set runstate to 
STARTUP (4)
2025-01-14T12:02:40.678505+05:00 samba1 ctdb-recovery[6051]: recovered 
event finished
2025-01-14T12:02:40.679156+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run starting
2025-01-14T12:02:40.679362+05:00 samba1 ctdb-recoverd[6045]: Set 
takeover_helper to "/usr/lib/ctdb/ctdb_takeover_helper"
2025-01-14T12:02:40.692782+05:00 samba1 ctdb-takeover[6053]: No nodes 
available to host public IPs yet
2025-01-14T12:02:40.693404+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run completed successfully
2025-01-14T12:02:40.693665+05:00 samba1 ctdb-recoverd[6045]: 
../../ctdb/server/ctdb_recoverd.c:1406 Recovery complete
2025-01-14T12:02:40.693874+05:00 samba1 ctdb-recoverd[6045]: Resetting 
ban count to 0 for all nodes
2025-01-14T12:02:40.694146+05:00 samba1 ctdb-recoverd[6045]: Just 
finished a recovery. New recoveries will now be suppressed for the 
rerecovery timeout (10 seconds)
2025-01-14T12:02:40.694263+05:00 samba1 ctdb-recoverd[6045]: Disabling 
recoveries for 10 seconds
2025-01-14T12:02:40.767822+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:40.767888+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:40.792535+05:00 samba1 ctdb-recoverd[6045]: Initial 
interface fetched
2025-01-14T12:02:40.792732+05:00 samba1 ctdb-recoverd[6045]: Trigger 
takeoverrun
2025-01-14T12:02:40.792908+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run starting
2025-01-14T12:02:40.805878+05:00 samba1 ctdb-takeover[6054]: No nodes 
available to host public IPs yet
2025-01-14T12:02:40.806450+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run completed successfully
2025-01-14T12:02:41.768559+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:41.768659+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:42.769690+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:42.769777+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:43.769866+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:43.769971+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:44.770295+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:44.770385+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:45.770756+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:45.770847+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:46.771018+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:46.771115+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:47.771442+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:47.771529+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:48.772105+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:48.772185+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:49.772721+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:49.772832+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:50.695560+05:00 samba1 ctdb-recoverd[6045]: Reenabling 
recoveries after timeout
2025-01-14T12:02:50.773839+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:50.773907+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:51.774280+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:51.774384+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:52.774463+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:52.774558+05:00 samba1 ctdbd[6033]: 
../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. 
Wait one more second.
2025-01-14T12:02:53.775182+05:00 samba1 ctdbd[6033]: 
CTDB_WAIT_UNTIL_RECOVERED
2025-01-14T12:02:53.775367+05:00 samba1 ctdbd[6033]: 
ctdb_recheck_persistent_health: OK[6] FAIL[0]
2025-01-14T12:02:53.775413+05:00 samba1 ctdbd[6033]: Running the 
"startup" event.
2025-01-14T12:02:53.775832+05:00 samba1 ctdbd[6033]: startup event OK - 
enabling monitoring
2025-01-14T12:02:53.775876+05:00 samba1 ctdbd[6033]: Set runstate to 
RUNNING (5)
2025-01-14T12:02:55.777187+05:00 samba1 ctdbd[6033]: monitor event OK - 
node re-enabled
2025-01-14T12:02:55.778083+05:00 samba1 ctdbd[6033]: Node became 
HEALTHY. Ask recovery master to reallocate IPs
2025-01-14T12:02:55.802465+05:00 samba1 ctdb-recoverd[6045]: Unassigned 
IP 192.168.160.180 can be served by this node
2025-01-14T12:02:55.802591+05:00 samba1 ctdb-recoverd[6045]: Trigger 
takeoverrun
2025-01-14T12:02:55.802770+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run starting
2025-01-14T12:02:55.813236+05:00 samba1 ctdbd[6033]: Takeover of IP 
192.168.160.180/24 on interface ens19
2025-01-14T12:02:55.819797+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run completed successfully
2025-01-14T12:02:56.802871+05:00 samba1 ctdb-recoverd[6045]: Assigned IP 
192.168.160.180 not on an interface
2025-01-14T12:02:56.802933+05:00 samba1 ctdb-recoverd[6045]: Trigger 
takeoverrun
2025-01-14T12:02:56.803143+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run starting
2025-01-14T12:02:56.813856+05:00 samba1 ctdbd[6033]: Takeover of IP 
192.168.160.180/24 on interface ens19
2025-01-14T12:02:56.824962+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run completed successfully
2025-01-14T12:02:57.803911+05:00 samba1 ctdb-recoverd[6045]: Assigned IP 
192.168.160.180 not on an interface
2025-01-14T12:02:57.803980+05:00 samba1 ctdb-recoverd[6045]: Trigger 
takeoverrun
2025-01-14T12:02:57.804181+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run starting
2025-01-14T12:02:57.815061+05:00 samba1 ctdbd[6033]: Takeover of IP 
192.168.160.180/24 on interface ens19
2025-01-14T12:02:57.823951+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run completed successfully
2025-01-14T12:02:58.804912+05:00 samba1 ctdb-recoverd[6045]: Assigned IP 
192.168.160.180 not on an interface
2025-01-14T12:02:58.804980+05:00 samba1 ctdb-recoverd[6045]: Trigger 
takeoverrun
2025-01-14T12:02:58.805180+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run starting
2025-01-14T12:02:58.816689+05:00 samba1 ctdbd[6033]: Takeover of IP 
192.168.160.180/24 on interface ens19
2025-01-14T12:02:58.822922+05:00 samba1 ctdb-recoverd[6045]: Takeover 
run completed successfully
.....

and again and again repeated last lines (assign ip ..)

[root at samba2 ~]# ctdb ip
Public IPs on node 1
192.168.160.180 0

but 192.168.160.180 not assign to ens19

[root at samba1 ~]#  ip -4 -f inet a show ens19
2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
state UP group default qlen 1000
    altname enp0s19
    inet 192.168.160.151/24 brd 192.168.160.255 scope global 
noprefixroute ens19
       valid_lft forever preferred_lft forever

[root at samba1 ~]# cat /etc/ctdb/public_addresses
192.168.160.180/24 ens19

[root at samba2 ~]# cat /etc/ctdb/public_addresses
192.168.160.180/24 ens19

[root at samba2 ~]# ip -4 -f inet a show ens19
2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
state UP group default qlen 1000
    altname enp0s19
    inet 192.168.160.152/24 brd 192.168.160.255 scope global 
noprefixroute ens19
       valid_lft forever preferred_lft forever

why 192.168.160.180 not assigned ?

-- 
Anton