Treating share modes as advisory when set by Mac clients

[Jeremy Allison]

On Mon, Mar 12, 2012 at 03:27:01PM -0400, Brian Campbell wrote:
> I work on network attached storage for video editing. Our customers use
> both Mac and Windows workstations, and on the Mac, they generally use
> Thursby Software's DAVE as a client, as it works much better than the
> built-in smbfs on Mac OS X.
> 
> On Mac OS X, when opening a file, you can request a shared or exclusive
> lock on the file using the O_SHLOCK or O_EXLOCK flags in the open() call.
> On local filesystems, these are implemented as locks with flock() semantics.
> 
> Thursby recently added support in DAVE to implement these flags using share
> modes (DENY_WRITE/DENY_ALL). They apparently did this because Microsoft
> Word depends on these locks working, and was getting confused when two
> people could edit the same file at the same time. This also appears to be
> how smbfs on Mac OS X implements these open() flags.
> 
> This has caused a problem for our customers using Final Cut (and possibly
> some other video editing software). When Final Cut writes files out in
> certain situations (renders, and log and transfer), it opens the file with
> the O_EXLOCK flag, and holds the file open with that exclusive lock even
> after it's done writing, until you close out the entire project. Locally,
> that doesn't cause any problems, since the lock is advisory and most
> software that needs to read the file doesn't try to obtain any locks on the
> file.
> 
> With DAVE now implementing these locks as share modes, which are mandatory
> and enforced by the server, we're running into trouble. When Final Cut
> writes the file out, and keeps the file open with the lock, this means the
> file continues to have the share mode DENY_ALL on the server, which
> prevents anyone else from reading the file.
> 
> I was wondering about whether it would be possible to recognize this
> situation on the server, and when we see a share mode being set by DAVE or
> Mac OS X's smbfs, to interpret that as an advisory lock, and not a
> mandatory lock.
> 
> How to do this is a bit of an open question. You would probably still want
> share modes to be mandatory for Windows clients. But you would want them to
> be advisory for Mac clients. I don't believe other Unix clients have locks
> that can be set on open(), and I haven't seen any other clients that
> actually implement flock(). I was thinking that you could add an extra bit
> somewhere in share_mode_entry indicating whether a share mode is mandatory
> or advisory. Windows clients would open files with a mandatory share mode.
> Mac clients would open them with an advisory share mode. If a file has a
> mandatory share mode that is incompatible with your requested access, it
> would fail. If a file has an advisory share mode, opening a file will only
> fail if you are requesting a mandatory or advisory share mode that is
> incompatible with the advisory share mode, but will not fail if you are
> opening the file with a share mode of DENY_NONE even if the type of access
> you are requesting would be incompatible with the share mode were it
> mandatory.
> 
> Does this make any sense? Does it sound like something that would be
> reasonable to implement? Are there any gotchas here that I'm missing? I'm
> completely new the the Samba code base and SMB, so please feel free to
> educate me if I'm missing something.

So there is a very old parameter called "share modes = [yes|no]" that
is share-specific that was designed to do what you want (i.e. you'd
redirect all the Mac clients to a specific share with this set).

Having said that it hasn't really been tested or used for many
a year, and now only exists as one check inside smbd/open.c
that always causes open_mode_check() to return true (i.e. don't
prevent opens) - which is almost certainly wrong and will break
other logic.

What you'd need is to resurrect that code correctly.

The real question is why is Dave keeping files open beyond
their application open lifetime ? That would cause problems
for lots of software.

Jeremy.