[Samba] secrets.tdb locking fun!

[Mac]

Hi all,

	It seems to have happened again.

Here's some sample log entries:-

[2007/07/06 14:24:33, 1, pid=6128] smbd/service.c:make_connection_snum(950)
  212.219.219.243 (212.219.219.243) connect to service favorites initially as user tjowett (uid=1860, gid=270) (pid 6128)
[2007/07/06 14:24:33, 0, pid=3074] tdb/tdbutil.c:tdb_log(783)
 tdb(/usr/local/samba/private/secrets.tdb): tdb_lock failed on list 2 ltype=2 (Interrupted system call)
[2007/07/06 14:24:33, 0, pid=3074] tdb/tdbutil.c:tdb_chainlock_with_timeout_internal(82)
  tdb_chainlock_with_timeout_internal: alarm (10) timed out for key replay cache mutex in tdb /usr/local/samba/private/secrets.tdb
[2007/07/06 14:24:33, 1, pid=3074] libads/kerberos_verify.c:ads_verify_ticket(357)
  ads_verify_ticket: unable to protect replay cache with mutex.
[2007/07/06 14:24:33, 1, pid=3074] smbd/sesssetup.c:reply_spnego_kerberos(202)
  Failed to verify incoming ticket!

>I mean the one in the non-TERM'able state. That's the one that's
>blocking the others.

We've used a 'pkill -TERM' to try and track down the "wedged" process.
Last time this occured two processes did not die after a SIGTERM. We
suspected one of them of being the culprit.

Not entirely sure that we've succeeded this time.  The process that
(appeared) not to have died with '-TERM' had a stack backtrace of :-

(gdb) bt
#0  0x00078760 in get_valid_user_struct ()
#1  0x00078c0c in invalidate_vuid ()
#2  0x00078f34 in invalidate_all_vuids ()
#3  0x004a1868 in exit_server_common ()
#4  0x004a1dc0 in exit_server_cleanly ()
#5  0x00123c2c in async_processing ()
#6  0x001244ec in receive_message_or_smb ()
#7  0x00127d04 in smbd_process ()
#8  0x004a306c in main ()
(gdb) detach
Detaching from program: /usr/local/samba/3.0.24/sbin/smbd process 17813
(gdb) quit


But by the time we'd exited gdb and had a further look around, the
process (17813) appeared to have exited cleanly (with no need for a
'-KILL').  This was not the observed behaviour last time.



We're working on scripting the 'gdb bt' so that we can run it across all
the smbd processes, when this next happens.  We're also trying to work
out if we can (sanely) use truss (or similar) to track what on earth is
going on.



                               Mac
          Assistant Systems Administrator @nibsc.ac.uk
                           mac at nibsc.ac.uk
   Work: +44 1707 641565          Everything else: +44 7956 237670 (anytime)