[Samba] Connection problem due to tdb_lock failed

[Oren Kishon]

We are using Samba 4.6.15. We have users not being able to login. winbindd log shows this:


[2018/06/14 14:48:44.300203,  0, pid=3228] ../source3/rpc_client/cli_pipe.c:3292(cli_rpc_pipe_open_schannel_with_creds)
  netlogon_creds_cli_get returned NT_STATUS_UNSUCCESSFUL
[2018/06/14 14:53:43.896350,  0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
  tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 beyond eof at 16384
[2018/06/14 14:53:43.896430,  0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
  tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 beyond eof at 16384
[2018/06/14 14:53:43.896465,  0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
  tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 beyond eof at 16384
[2018/06/14 14:53:43.896500,  0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
  tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 825242227 beyond eof at 16384
[2018/06/14 14:53:43.896523,  1, pid=3228] ../source3/lib/g_lock.c:193(g_lock_trylock)
  rec->store failed: NT_STATUS_UNSUCCESSFUL
[2018/06/14 14:53:43.896586,  0, pid=3228] ../source3/rpc_client/cli_pipe.c:3292(cli_rpc_pipe_open_schannel_with_creds)
  netlogon_creds_cli_get returned NT_STATUS_UNSUCCESSFUL
[2018/06/14 14:54:23.899114,  1, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
  tdb(/var/vol/3/.ctera/samba/lock/mutex.tdb): tdb_lock failed on list 77 ltype=1 (Interrupted system call)
[2018/06/14 14:54:23.899181,  0, pid=3228] ../source3/lib/util_tdb.c:497(tdb_chainlock_with_timeout_internal)
  tdb_chainlock_with_timeout_internal: alarm (40) timed out for key R-DC-CLIENT.COM in tdb /var/vol/3/.ctera/samba/lock/mutex.tdb
[2018/06/14 14:54:23.899204,  1, pid=3228] ../source3/lib/server_mutex.c:97(grab_named_mutex)
  Could not get the lock for R-DC-CLIENT.COM
[2018/06/14 14:54:23.899276,  0, pid=3228] ../source3/winbindd/winbindd_cm.c:1022(cm_prepare_connection)
  cm_prepare_connection: mutex grab failed for R-DC-CLIENT.COM
[2018/06/14 14:54:23.899298,  1, pid=3228] ../source3/winbindd/winbindd_cm.c:1253(cm_prepare_connection)
  Failed to prepare SMB connection to R-DC-CLIENT.COM: NT_STATUS_POSSIBLE_DEADLOCK

The issue can be reproduced manually by using "wbinfo --ping-dc" on the server machine, which produces output:

wbinfo --ping-dc checking the NETLOGON for domain[CLIENT] dc connection to "" failed wbcPingDc2(CLIENT): error code was NT_STATUS_UNSUCCESSFUL (0xc0000001)


Seems like g_lock.tdb is corrupted and indeed tdbdump on it returns same "beyond eof" error as in the above log.

Restarting Samba fixes this and about 3 days later it would happen again.
What could be the cause of the file corruption?
Is there any way to bypass this flow?

Thank you,
Oren Kishon