[Samba] ctdb + gluster9 = not working

Anton Shevtsov shevtsovay at basealt.ru
Tue Jan 14 07:19:58 UTC 2025


i fixed))


[root at samba1 ~]#  ctdb event script list legacy
   00.ctdb
   01.reclock
   05.system
   06.nfs
   10.interface
   11.natgw
   11.routing
   13.per_ip_routing
   20.multipathd
   31.clamd
   40.vsftpd
   41.httpd
   48.netbios
   49.winbind
   50.samba
   60.nfs
   70.iscsi
   91.lvs

[root at samba1 ~]# ctdb event script enable legacy 10.interface

[root at samba1 ~]# ctdb event script list legacy
   00.ctdb
   01.reclock
   05.system
   06.nfs
* 10.interface
   11.natgw
   11.routing
   13.per_ip_routing
   20.multipathd
   31.clamd
   40.vsftpd
   41.httpd
   48.netbios
   49.winbind
   50.samba
   60.nfs
   70.iscsi
   91.lvs

2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
state UP group default qlen 1000
    altname enp0s19
    inet 192.168.160.151/24 brd 192.168.160.255 scope global 
noprefixroute ens19
       valid_lft forever preferred_lft forever
    inet 192.168.160.180/24 brd 192.168.160.255 scope global secondary 
ens19
       valid_lft forever preferred_lft forever

14.01.2025 12:15, Anton Shevtsov via samba пишет:
>
> 29.12.2024 04:25, Martin Schwenke via samba пишет:
>> Hi Anton,
>>
>> On Fri, 27 Dec 2024 20:04:07 +0500, Anton Shevtsov via samba
>> <samba at lists.samba.org> wrote:
>>
>>> Hi, i tried make a samba cluster (ctdb) with gluster9 (do not use vfs
>>> module)
>>>
>>> But i get an errors.
>>>
>>> [root at samba1 ctdb]# rpm -q samba samba-ctdb
>>> samba-4.19.9-alt3.x86_64
>>> samba-ctdb-4.19.9-alt3.x86_64
>>>
>>> [root at samba1 ctdb]# grep -v '^#' /etc/ctdb/ctdb.conf
>>> [cluster]
>>> recovery lock = "/mnt/gluster/ctdb/.ctdb.lock"
>> [...]
>>
>>> дек 27 19:46:54 samba1 systemd[1]: Starting CTDB...
>>> дек 27 19:46:54 samba1 ctdbd[25430]: CTDB logging to location
>>> file:/var/log/log.ctdb
>>> дек 27 19:46:54 samba1 systemd[1]: Started CTDB.
>>>
>>> [root at samba1 ctdb]# tail -f /var/log/log.ctdb
>>> [...]
>>> 2024-12-27T19:47:24.597406+05:00 samba1 ctdb-recoverd[25443]: Start 
>>> election
>>> 2024-12-27T19:47:24.598139+05:00 samba1 ctdb-recoverd[25443]: 
>>> Attempting to take cluster lock ("/mnt/gluster/ctdb/.ctdb.lock")
>>> 2024-12-27T19:47:24.611930+05:00 samba1 ctdbd[25431]: 
>>> ctdb_mutex_fcntl_helper: Unable to open 
>>> "/mnt/gluster/ctdb/.ctdb.lock" - (No such file or directory)
>>> 2024-12-27T19:47:24.612311+05:00 samba1 ctdb-recoverd[25443]: Unable 
>>> to take cluster lock - unknown error
>>> [...]
>>> Why?
>> Good question!  This took me a few guesses... because it certainly isn't
>> obvious! :-)
>>
>> You don't need the double-quotes around the recovery lock setting.
>> They are not inserted by the log messages, so must be maintained from
>> ctdb.conf and not stripped.  Samba (and CTDB) log files can handle
>> values containing whitespace, so quoting isn't required.
>>
>> If your Linux distribution included the quotes in the sample
>> configuration file then you should report a bug because the example is
>> misleading.  If not, you're just unlucky because you did something sane
>> but it didn't work.
>>
>> peace & happiness,
>> martin
>
> So simple! ;)))) Thanks!
>
> I can make a next step.
>
> [root at samba1 ~] systemctl start ctdb
>
> [root at samba2 ~] systemctl start ctdb
>
> [root at samba2 ~]# ctdb status
> Number of nodes:2
> pnn:0 192.168.160.151  OK
> pnn:1 192.168.160.152  OK (THIS NODE)
> Generation:1076686332
> Size:2
> hash:0 lmaster:0
> hash:1 lmaster:1
>
> [root at samba1 ctdb]# tail -f /var/log/log.ctdb
>
> [root at samba1 ~]# tail -f /var/log/log.ctdb
> 2025-01-14T12:02:34.731080+05:00 samba1 ctdbd[6033]: Freeze db: 
> account_policy.tdb
> 2025-01-14T12:02:34.735230+05:00 samba1 ctdbd[6033]: Freeze db: 
> group_mapping.tdb
> 2025-01-14T12:02:34.739394+05:00 samba1 ctdbd[6033]: Freeze db: 
> share_info.tdb
> 2025-01-14T12:02:34.743590+05:00 samba1 ctdbd[6033]: Freeze db: 
> registry.tdb
> 2025-01-14T12:02:34.747753+05:00 samba1 ctdbd[6033]: Set runstate to 
> SETUP (2)
> 2025-01-14T12:02:34.763874+05:00 samba1 ctdbd[6033]: Keepalive 
> monitoring has been started
> 2025-01-14T12:02:34.764010+05:00 samba1 ctdbd[6033]: Set runstate to 
> FIRST_RECOVERY (3)
> 2025-01-14T12:02:34.764520+05:00 samba1 ctdb-recoverd[6045]: 
> monitor_cluster starting
> 2025-01-14T12:02:35.764460+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:36.765546+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:37.766605+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:38.767048+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:39.767644+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:39.768239+05:00 samba1 ctdb-recoverd[6045]: Leader 
> broadcast timeout
> 2025-01-14T12:02:39.768764+05:00 samba1 ctdb-recoverd[6045]: Start 
> election
> 2025-01-14T12:02:39.769124+05:00 samba1 ctdb-recoverd[6045]: 
> Attempting to take cluster lock (/mnt/gluster/ctdb.lock)
> 2025-01-14T12:02:39.769310+05:00 samba1 ctdb-recoverd[6045]: Set 
> cluster mutex helper to "/usr/lib/ctdb/ctdb_mutex_fcntl_helper"
> 2025-01-14T12:02:39.789394+05:00 samba1 ctdb-recoverd[6045]: Cluster 
> lock taken successfully
> 2025-01-14T12:02:39.789657+05:00 samba1 ctdb-recoverd[6045]: Took 
> cluster lock, leader=0
> 2025-01-14T12:02:39.790853+05:00 samba1 ctdb-recoverd[6045]: Node:0 
> was in recovery mode. Start recovery process
> 2025-01-14T12:02:39.791059+05:00 samba1 ctdb-recoverd[6045]: 
> ../../ctdb/server/ctdb_recoverd.c:1314 Starting do_recovery
> 2025-01-14T12:02:39.791262+05:00 samba1 ctdb-recoverd[6045]: 
> ../../ctdb/server/ctdb_recoverd.c:1358 Recovery initiated due to 
> problem with node 0
> 2025-01-14T12:02:39.791741+05:00 samba1 ctdb-recoverd[6045]: 
> ../../ctdb/server/ctdb_recoverd.c:1388 Recovery - updated flags
> 2025-01-14T12:02:39.791946+05:00 samba1 ctdb-recoverd[6045]: Set 
> recovery_helper to "/usr/lib/ctdb/ctdb_recovery_helper"
> 2025-01-14T12:02:39.800805+05:00 samba1 ctdb-recovery[6051]: Set 
> recovery mode to ACTIVE
> 2025-01-14T12:02:39.801001+05:00 samba1 ctdbd[6033]: Recovery has started
> 2025-01-14T12:02:39.801254+05:00 samba1 ctdb-recovery[6051]: 
> start_recovery event finished
> 2025-01-14T12:02:39.801421+05:00 samba1 ctdb-recovery[6051]: updated 
> VNNMAP
> 2025-01-14T12:02:39.801492+05:00 samba1 ctdb-recovery[6051]: recover 
> database 0x7132c184
> 2025-01-14T12:02:39.801571+05:00 samba1 ctdb-recovery[6051]: recover 
> database 0x3ef19640
> 2025-01-14T12:02:39.801678+05:00 samba1 ctdb-recovery[6051]: recover 
> database 0x2ca251cf
> 2025-01-14T12:02:39.801742+05:00 samba1 ctdb-recovery[6051]: recover 
> database 0xa1413774
> 2025-01-14T12:02:39.801800+05:00 samba1 ctdb-recovery[6051]: recover 
> database 0xc3078fba
> 2025-01-14T12:02:39.801862+05:00 samba1 ctdb-recovery[6051]: recover 
> database 0x6cf2837d
> 2025-01-14T12:02:39.802221+05:00 samba1 ctdbd[6033]: Freeze db: 
> secrets.tdb frozen
> 2025-01-14T12:02:39.802309+05:00 samba1 ctdbd[6033]: Freeze db: 
> passdb.tdb frozen
> 2025-01-14T12:02:39.802351+05:00 samba1 ctdbd[6033]: Freeze db: 
> account_policy.tdb frozen
> 2025-01-14T12:02:39.802379+05:00 samba1 ctdbd[6033]: Freeze db: 
> group_mapping.tdb frozen
> 2025-01-14T12:02:39.802403+05:00 samba1 ctdbd[6033]: Freeze db: 
> share_info.tdb frozen
> 2025-01-14T12:02:39.802488+05:00 samba1 ctdbd[6033]: Freeze db: 
> registry.tdb frozen
> 2025-01-14T12:02:40.652978+05:00 samba1 ctdbd[6033]: Thaw db: 
> secrets.tdb generation 1827381565
> 2025-01-14T12:02:40.653088+05:00 samba1 ctdbd[6033]: Release freeze 
> handle for db secrets.tdb
> 2025-01-14T12:02:40.653189+05:00 samba1 ctdbd[6033]: Thaw db: 
> passdb.tdb generation 1827381565
> 2025-01-14T12:02:40.653226+05:00 samba1 ctdbd[6033]: Release freeze 
> handle for db passdb.tdb
> 2025-01-14T12:02:40.653283+05:00 samba1 ctdbd[6033]: Thaw db: 
> account_policy.tdb generation 1827381565
> 2025-01-14T12:02:40.653323+05:00 samba1 ctdbd[6033]: Release freeze 
> handle for db account_policy.tdb
> 2025-01-14T12:02:40.653397+05:00 samba1 ctdbd[6033]: Thaw db: 
> group_mapping.tdb generation 1827381565
> 2025-01-14T12:02:40.653441+05:00 samba1 ctdbd[6033]: Release freeze 
> handle for db group_mapping.tdb
> 2025-01-14T12:02:40.653660+05:00 samba1 ctdbd[6033]: Thaw db: 
> share_info.tdb generation 1827381565
> 2025-01-14T12:02:40.653730+05:00 samba1 ctdbd[6033]: Release freeze 
> handle for db share_info.tdb
> 2025-01-14T12:02:40.655336+05:00 samba1 ctdbd[6033]: Thaw db: 
> registry.tdb generation 1827381565
> 2025-01-14T12:02:40.655398+05:00 samba1 ctdbd[6033]: Release freeze 
> handle for db registry.tdb
> 2025-01-14T12:02:40.658394+05:00 samba1 ctdb-recovery[6051]: 6 of 6 
> databases recovered
> 2025-01-14T12:02:40.658577+05:00 samba1 ctdbd[6033]: Recovery mode set 
> to NORMAL
> 2025-01-14T12:02:40.658666+05:00 samba1 ctdbd[6033]: Set cluster mutex 
> helper to "/usr/lib/ctdb/ctdb_mutex_fcntl_helper"
> 2025-01-14T12:02:40.677257+05:00 samba1 ctdb-recovery[6051]: Set 
> recovery mode to NORMAL
> 2025-01-14T12:02:40.677419+05:00 samba1 ctdbd[6033]: Recovery has 
> finished
> 2025-01-14T12:02:40.678242+05:00 samba1 ctdbd[6033]: Set runstate to 
> STARTUP (4)
> 2025-01-14T12:02:40.678505+05:00 samba1 ctdb-recovery[6051]: recovered 
> event finished
> 2025-01-14T12:02:40.679156+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run starting
> 2025-01-14T12:02:40.679362+05:00 samba1 ctdb-recoverd[6045]: Set 
> takeover_helper to "/usr/lib/ctdb/ctdb_takeover_helper"
> 2025-01-14T12:02:40.692782+05:00 samba1 ctdb-takeover[6053]: No nodes 
> available to host public IPs yet
> 2025-01-14T12:02:40.693404+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run completed successfully
> 2025-01-14T12:02:40.693665+05:00 samba1 ctdb-recoverd[6045]: 
> ../../ctdb/server/ctdb_recoverd.c:1406 Recovery complete
> 2025-01-14T12:02:40.693874+05:00 samba1 ctdb-recoverd[6045]: Resetting 
> ban count to 0 for all nodes
> 2025-01-14T12:02:40.694146+05:00 samba1 ctdb-recoverd[6045]: Just 
> finished a recovery. New recoveries will now be suppressed for the 
> rerecovery timeout (10 seconds)
> 2025-01-14T12:02:40.694263+05:00 samba1 ctdb-recoverd[6045]: Disabling 
> recoveries for 10 seconds
> 2025-01-14T12:02:40.767822+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:40.767888+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:40.792535+05:00 samba1 ctdb-recoverd[6045]: Initial 
> interface fetched
> 2025-01-14T12:02:40.792732+05:00 samba1 ctdb-recoverd[6045]: Trigger 
> takeoverrun
> 2025-01-14T12:02:40.792908+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run starting
> 2025-01-14T12:02:40.805878+05:00 samba1 ctdb-takeover[6054]: No nodes 
> available to host public IPs yet
> 2025-01-14T12:02:40.806450+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run completed successfully
> 2025-01-14T12:02:41.768559+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:41.768659+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:42.769690+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:42.769777+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:43.769866+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:43.769971+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:44.770295+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:44.770385+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:45.770756+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:45.770847+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:46.771018+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:46.771115+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:47.771442+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:47.771529+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:48.772105+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:48.772185+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:49.772721+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:49.772832+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:50.695560+05:00 samba1 ctdb-recoverd[6045]: 
> Reenabling recoveries after timeout
> 2025-01-14T12:02:50.773839+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:50.773907+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:51.774280+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:51.774384+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:52.774463+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:52.774558+05:00 samba1 ctdbd[6033]: 
> ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to 
> end. Wait one more second.
> 2025-01-14T12:02:53.775182+05:00 samba1 ctdbd[6033]: 
> CTDB_WAIT_UNTIL_RECOVERED
> 2025-01-14T12:02:53.775367+05:00 samba1 ctdbd[6033]: 
> ctdb_recheck_persistent_health: OK[6] FAIL[0]
> 2025-01-14T12:02:53.775413+05:00 samba1 ctdbd[6033]: Running the 
> "startup" event.
> 2025-01-14T12:02:53.775832+05:00 samba1 ctdbd[6033]: startup event OK 
> - enabling monitoring
> 2025-01-14T12:02:53.775876+05:00 samba1 ctdbd[6033]: Set runstate to 
> RUNNING (5)
> 2025-01-14T12:02:55.777187+05:00 samba1 ctdbd[6033]: monitor event OK 
> - node re-enabled
> 2025-01-14T12:02:55.778083+05:00 samba1 ctdbd[6033]: Node became 
> HEALTHY. Ask recovery master to reallocate IPs
> 2025-01-14T12:02:55.802465+05:00 samba1 ctdb-recoverd[6045]: 
> Unassigned IP 192.168.160.180 can be served by this node
> 2025-01-14T12:02:55.802591+05:00 samba1 ctdb-recoverd[6045]: Trigger 
> takeoverrun
> 2025-01-14T12:02:55.802770+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run starting
> 2025-01-14T12:02:55.813236+05:00 samba1 ctdbd[6033]: Takeover of IP 
> 192.168.160.180/24 on interface ens19
> 2025-01-14T12:02:55.819797+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run completed successfully
> 2025-01-14T12:02:56.802871+05:00 samba1 ctdb-recoverd[6045]: Assigned 
> IP 192.168.160.180 not on an interface
> 2025-01-14T12:02:56.802933+05:00 samba1 ctdb-recoverd[6045]: Trigger 
> takeoverrun
> 2025-01-14T12:02:56.803143+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run starting
> 2025-01-14T12:02:56.813856+05:00 samba1 ctdbd[6033]: Takeover of IP 
> 192.168.160.180/24 on interface ens19
> 2025-01-14T12:02:56.824962+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run completed successfully
> 2025-01-14T12:02:57.803911+05:00 samba1 ctdb-recoverd[6045]: Assigned 
> IP 192.168.160.180 not on an interface
> 2025-01-14T12:02:57.803980+05:00 samba1 ctdb-recoverd[6045]: Trigger 
> takeoverrun
> 2025-01-14T12:02:57.804181+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run starting
> 2025-01-14T12:02:57.815061+05:00 samba1 ctdbd[6033]: Takeover of IP 
> 192.168.160.180/24 on interface ens19
> 2025-01-14T12:02:57.823951+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run completed successfully
> 2025-01-14T12:02:58.804912+05:00 samba1 ctdb-recoverd[6045]: Assigned 
> IP 192.168.160.180 not on an interface
> 2025-01-14T12:02:58.804980+05:00 samba1 ctdb-recoverd[6045]: Trigger 
> takeoverrun
> 2025-01-14T12:02:58.805180+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run starting
> 2025-01-14T12:02:58.816689+05:00 samba1 ctdbd[6033]: Takeover of IP 
> 192.168.160.180/24 on interface ens19
> 2025-01-14T12:02:58.822922+05:00 samba1 ctdb-recoverd[6045]: Takeover 
> run completed successfully
> .....
>
> and again and again repeated last lines (assign ip ..)
>
> [root at samba2 ~]# ctdb ip
> Public IPs on node 1
> 192.168.160.180 0
>
> but 192.168.160.180 not assign to ens19
>
> [root at samba1 ~]#  ip -4 -f inet a show ens19
> 2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
> state UP group default qlen 1000
>    altname enp0s19
>    inet 192.168.160.151/24 brd 192.168.160.255 scope global 
> noprefixroute ens19
>       valid_lft forever preferred_lft forever
>
> [root at samba1 ~]# cat /etc/ctdb/public_addresses
> 192.168.160.180/24 ens19
>
> [root at samba2 ~]# cat /etc/ctdb/public_addresses
> 192.168.160.180/24 ens19
>
> [root at samba2 ~]# ip -4 -f inet a show ens19
> 2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
> state UP group default qlen 1000
>    altname enp0s19
>    inet 192.168.160.152/24 brd 192.168.160.255 scope global 
> noprefixroute ens19
>       valid_lft forever preferred_lft forever
>
> why 192.168.160.180 not assigned ?
>
-- 
basealt logo *Шевцов Антон Юрьевич*
Эксперт по инфраструктуре
Дирекция по проектам ООО Базальт СПО
мобильный : +79222651692
telegram : @anton_shevtsov


More information about the samba mailing list