Treating share modes as advisory when set by Mac clients

[Brian Campbell]

I work on network attached storage for video editing. Our customers use
both Mac and Windows workstations, and on the Mac, they generally use
Thursby Software's DAVE as a client, as it works much better than the
built-in smbfs on Mac OS X.

On Mac OS X, when opening a file, you can request a shared or exclusive
lock on the file using the O_SHLOCK or O_EXLOCK flags in the open() call.
On local filesystems, these are implemented as locks with flock() semantics.

Thursby recently added support in DAVE to implement these flags using share
modes (DENY_WRITE/DENY_ALL). They apparently did this because Microsoft
Word depends on these locks working, and was getting confused when two
people could edit the same file at the same time. This also appears to be
how smbfs on Mac OS X implements these open() flags.

This has caused a problem for our customers using Final Cut (and possibly
some other video editing software). When Final Cut writes files out in
certain situations (renders, and log and transfer), it opens the file with
the O_EXLOCK flag, and holds the file open with that exclusive lock even
after it's done writing, until you close out the entire project. Locally,
that doesn't cause any problems, since the lock is advisory and most
software that needs to read the file doesn't try to obtain any locks on the
file.

With DAVE now implementing these locks as share modes, which are mandatory
and enforced by the server, we're running into trouble. When Final Cut
writes the file out, and keeps the file open with the lock, this means the
file continues to have the share mode DENY_ALL on the server, which
prevents anyone else from reading the file.

I was wondering about whether it would be possible to recognize this
situation on the server, and when we see a share mode being set by DAVE or
Mac OS X's smbfs, to interpret that as an advisory lock, and not a
mandatory lock.

How to do this is a bit of an open question. You would probably still want
share modes to be mandatory for Windows clients. But you would want them to
be advisory for Mac clients. I don't believe other Unix clients have locks
that can be set on open(), and I haven't seen any other clients that
actually implement flock(). I was thinking that you could add an extra bit
somewhere in share_mode_entry indicating whether a share mode is mandatory
or advisory. Windows clients would open files with a mandatory share mode.
Mac clients would open them with an advisory share mode. If a file has a
mandatory share mode that is incompatible with your requested access, it
would fail. If a file has an advisory share mode, opening a file will only
fail if you are requesting a mandatory or advisory share mode that is
incompatible with the advisory share mode, but will not fail if you are
opening the file with a share mode of DENY_NONE even if the type of access
you are requesting would be incompatible with the share mode were it
mandatory.

Does this make any sense? Does it sound like something that would be
reasonable to implement? Are there any gotchas here that I'm missing? I'm
completely new the the Samba code base and SMB, so please feel free to
educate me if I'm missing something.

-- Brian Campbell
EditShare