locking issue with samba-2.2.1a-cvs010809 on Solaris 8

[Scott Moomaw]

I'm beginning to look closely at a locking problem that we've been having
with the Samba 2.2.x series on our Solaris 8 system.  We're having
processes get stuck in an infinite cycle of fcntl calls while trying to
open a file.  It appears to me that things are cycling inside of
open.c/open_mode_check.  Next time things go awry, I want to grab more
info, but I have some details that might help others looking into this
problem.

In the case that I'm looking at from last night, I have several tidbits
that may prove helpful.  As a first debugging step, I inserted an extra
DEBUG line in smbd/open.c just inside the for loop to show file open
attempts in open_mode_check.  I'm going to work more with this very soon,
but my first work did give me a clue regarding at least one circumstance.

[2001/08/17 19:57:45, 0] smbd/open.c:open_mode_check(491)
  scott_open_mode_check: oplock_request = 0, breaking oplock (4) on file
mclark/Recent/TEAM BUILDING.lnk, dev = 740040, inode = 35429
[2001/08/17 19:57:45, 0] smbd/open.c:open_mode_check(491)
  scott_open_mode_check: oplock_request = 0, breaking oplock (4) on file
mclark/Recent/TEAM BUILDING.lnk, dev = 740040, inode = 35429

The above snippet from the log file shows a client continually trying to
open a file.  (These lines are repeated 10000s of times before samba was
restarted).  In this case, it appears to me that a process (PID 3426)
opened the file in question using a LEVEL_II oplock.  I don't have enough
info available to determine why, but something happened (maybe a sudden
client reboot?) that originated a new process for that client (PID 3435).
Below is the info from smbstatus related to this system.

profiles     mclark   bcstaff   3426   football1 (147.138.140.222) Fri Aug
17 19:16:57 2001
IPC$         xguest   guest     3426   football1 (147.138.140.222) Fri Aug
17 19:15:57 2001
IPC$         xguest   guest     3426   football1 (147.138.140.222) Fri Aug
17 19:16:50 2001
IPC$         xguest   guest     3426   football1 (147.138.140.222) Fri Aug
17 19:16:57 2001
profiles     mclark   bcstaff   3435   football1 (147.138.140.222) Fri Aug
17 19:18:15 2001

3435   DENY_NONE  RDONLY     EXCLUSIVE+BATCH
/export/profiles/mclark/Recent/TIGER.lnk   Fri Aug 17 19:18:15 2001
3435   DENY_NONE  RDONLY     EXCLUSIVE+BATCH
/export/profiles/mclark/Recent/Book1.lnk   Fri Aug 17 19:18:15 2001
3426   DENY_NONE  RDONLY     NONE
/export/profiles/mclark/Recent/sunday august 19th 2001.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_WRITE RDONLY     NONE
/export/profiles/mclark/Recent/sunday august 19th 2001.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_NONE  RDONLY     NONE
/export/profiles/mclark/Recent/Football.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_WRITE RDONLY     NONE
/export/profiles/mclark/Recent/Football.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_WRITE RDONLY     EXCLUSIVE+BATCH
/export/profiles/mclark/Recent/TEAM ROSTER.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_NONE  RDONLY     NONE
/export/profiles/mclark/Recent/summer ball 2001.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_WRITE RDONLY     NONE
/export/profiles/mclark/Recent/summer ball 2001.lnk   Fri Aug 17 19:17:03 2001
3426   DENY_WRITE RDONLY     LEVEL_II
/export/profiles/mclark/Recent/TEAM BUILDING.lnk   Fri Aug 17 19:17:03 2001

As I noted, I don't have enough info to confirm what happened to 3426 or
3435, but I do know that the client in question was continually trying to
open /export/profiles/mclark/Recent/TEAM BUILDING.lnk which has been
opened by a LEVEL_II oplock by PID 3426.

Tracing downward through the code, inside oplock.c/request_oplock_break, a
break message is sent to the client via sendto and the function returns
successfully when the oplock help is LEVEL_II.  Unless I'm
misunderstanding sendto, its errorcode is not dependent upon the
destination actually receiving the sent message; thus, a succsesful send
can occur without a client available to act upon it.  In the case where
the client doesn't react to the invalid oplock break message correctly,
the request_oplock_break function doesn't free the LEVEL_II oplock so the
open_mode_check function will just keep iterating.  Am I missing something
that will prevent this from occurring, or is this a condition that isn't
being fully handled by samba?

I'm going to attempt to further diagnose this and provide more info as I
acquire it.  I'd appreciate any help on solving this issue.  Processes
tripping over oplocks and getting stuck in infinite loops (fcntl calls)
represent a big problem for us.

Scott
------------------------------------------------------------------------
 Scott Moomaw, Network Administrator              Scott at Bridgewater.edu
 Bridgewater College, IT Center
 Bridgewater, VA  22812
 Phone (540) 828 - 8000  x5437              FAX:  (540) 828 - 5493