Hitting Could not claim version: NT_STATUS_IO_TIMEOUT

[Richard Sharpe]

Hi folks,

I am hitting the following (in 4.7.1) in a CTDB-setup, and am looking
for hints on how to debug it.

--------------------
[2019/05/16 18:49:24.648838,  5, pid=21261, effective(0, 0), real(0,
0)] ../lib/dbwrap/dbwrap.c:160(dbwrap_check_lock_order)
  check lock order 3 for g_lock.tdb
[2019/05/16 18:49:24.648880, 10, pid=21261, effective(0, 0), real(0,
0)] ../lib/dbwrap/dbwrap.c:115(debug_lock_order)
  lock order:  1:<none> 2:<none> 3:g_lock.tdb
[2019/05/16 18:49:24.648908, 10, pid=21261, effective(0, 0), real(0,
0)] ../source3/lib/dbwrap/dbwrap_ctdb.c:1112(fetch_locked_internal)
  Locking db 1294615339 key 73616D62615F76657273
[2019/05/16 18:49:24.648935,  5, pid=21261, effective(0, 0), real(0,
0)] ../lib/dbwrap/dbwrap.c:128(dbwrap_lock_order_state_destructor)
  release lock order 3 for g_lock.tdb
[2019/05/16 18:49:24.648958, 10, pid=21261, effective(0, 0), real(0,
0)] ../lib/dbwrap/dbwrap.c:115(debug_lock_order)
  lock order:  1:<none> 2:<none> 3:<none>
[2019/05/16 18:49:24.648963, 10, pid=21261, effective(0, 0), real(0,
0)] ../source3/lib/dbwrap/dbwrap_ctdb.c:980(db_ctdb_record_destr)
  Unlocking db 1294615339 key 73616D62615F76657273
[2019/05/16 18:49:24.648977,  1, pid=21261, effective(0, 0), real(0,
0)] ../source3/smbd/server.c:1497(smbd_claim_version)
  smbd_claim_version: g_lock_lock(G_LOCK_READ) failed: NT_STATUS_IO_TIMEOUT
[2019/05/16 18:49:24.649000,  1, pid=21261, effective(0, 0), real(0,
0)] ../source3/smbd/server.c:2024(main)
  main: Could not claim version: NT_STATUS_IO_TIMEOUT
---------------------

This is a four-node ctdb setup which was upgraded recently, but as far
as I can see all nodes are at the same version and the problem is
grabbing the lock.

This piece of code took the error path:

        status = g_lock_lock(ctx, name, G_LOCK_READ,
                             (struct timeval) { .tv_sec = 60 });
        if (!NT_STATUS_IS_OK(status)) {
                DBG_WARNING("g_lock_lock(G_LOCK_READ) failed: %s\n",
                            nt_errstr(status));
                TALLOC_FREE(ctx);
                return status;
        }

Anyone have any idea what that is about or how to debug it further?

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)(传说杜康是酒的发明者)