Better interop for NFS/SMB file share mode/reservation

[Amir Goldstein]

[dropping linux-fsdevel]

On Thu, Apr 25, 2019 at 9:11 PM Amir Goldstein <amir73il at gmail.com> wrote:
>
> On Thu, Mar 7, 2019 at 1:04 PM Stefan Metzmacher <metze at samba.org> wrote:
> >
> > Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical:
> > >
> > > Jeremy Allison wrote:
> > >> On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
> > >>> On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
> > >>>>
> > >>>> Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation
> > >>>> API?  Is that close to what's needed?
> > >>>>
> > >>>
> > >>> Here's the C headers for that stuff:
> > >>>
> > >>>    https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/src/include/cephfs/libcephfs.h#L1734
> > >>>
> > >>> It's simple enough and works for us in ganesha, and I think we can
> > >>> probably adapt it to samba without too much difficulty. The callback
> > >>> doesn't seem like it'll do for a kernel API though -- you'd almost
> > >>> certainly need to do something different there (signals? inotify?).
> > >>
> > >> SMB3 leases have R/RW and Handle-based leases.
> > >
> > > Just to be precise: SMB2.1+ has R, RH, RW and RWH leases.
> > >
> > >> Handle leases allow multiple opens of the same pathname
> > >> that get different handles to share the lease, allowing
> > >> a client redirector to delay opens or closes locally
> > >> so long as it has a handle lease.
> > >
> > > That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request
> > >
> > >>
> > >> Here are the semantics:
> > >>
> > >> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943d-6ad7-4b30-9f58-96ae90fc6204
> > >>
> > >> I'm not sure a simple file-descriptor based API is
> > >> enough for us. Can he have a uuid or token based
> > >> API instead where the server can chose what fd's
> > >> to cover with a token ?
> > >
> > > Yes, that would be ideal.
>
> Getting back to this.
> Thanks all for the valuable inputs.
>
> Next week is LSF/MM and I was assigned a 30 minute slot on filesystems track
> to discuss "NFS/SMB file share".
>
> So let me try to echo what I read on this thread and how I understand what APIs
> samba needs from the kernel.
>
> >
> > If we want to design an useful API, we also need to think about
> > all features:
> > - file oplock/leases
>
> Kernel can have a flavor of leases which are not broken
> by opens from threads of the process holding the lease.
> Bruce has some patches along those lines for knfsd and SMB R/RW
> leases could use this flavor if it was exported to userspace?
>
> For SMB RH/RWH leases and Ganesha delegations, server
> could keep track of its own handles/clients and break leases within the
> same process without involving the kernel.
> Am I wrong?
>
> > - directory leases
>
> I have WIP on fsnotify directory pre modification hooks.
> There is opposition from fsnotify maintainer to add new userspace
> APIs that can create kernel->user->kernel deadlocks, like the
> deadlocks currently reported with fanotify permission events.
>
> Need to see if we can find a middle ground between
> "post modification notifications" and "pre modification permission"
> API, somewhere along the lines of regular file lease breaking API.
>
> > - share modes
>
> Volker told me he thinks samba can enforce share modes by
> a single daemon policing all opens in the system with fanotify.
> I think he is right. If anyone thinks differently please speak up.
>
> > - disconnected handles (for durable and persistent handles),
> >   which exists within the kernel for a while and can be reattached
> >   to process, using some kind of cookie and the same euid
>
> So this interface exists in the kernel.
> Nothing more required from the kernel API. Right?
>
> > - the API needs ways to use epoll in order to do async opens
> >   and lease breaks. For opens the model of async socket connects
> >   could be used. Leases could have a signalfd-style api.
>
> I should hope that the new AIO API (http://kernel.dk/io_uring.pdf)
> would solve those problems as well as other issues that
> samba has w.r.t dispatching AIO.
>
> >
> > We may not need everything at once, but we should have the full picture
> > in mind. And we need working code in kernel and userspace that passes
> > all tests (we may need to add additional test). Otherwise the kernel
> > creates new syscalls, which wouldn't be used by Samba in the end.
> >
>
> Tested interfaces - good idea ;-)
>
> If anyone has any comments about my view of required new interfaces,
> or important things that I missed, please say so before Tuesday!
>

Hello Samba-team,

Some of you may have already seen the reports from my session at LSF/MM
on Samba/NFS interop: https://lwn.net/Articles/788335/

It should not be a surprise to anyone here to know that I have had interesting
and productive conversations with NFS folks about improving samba interop.
It should not be a surprise to anyone here to know that the rest of the audience
was, generally speaking, uninterested in the problem.

Which provides a re-enforcement to the point I was trying to make in session -
The path of least resistance for NFS-Samba interop is the communicate with
each other (both human and software wise) and try to leave VFS out of the
discussion for as much as possible (hence dropping linux-fsdevel from
this thread).

An idea that has already been thrown around is to use some samba daemon as
an arbitrator for opening files and locks. Of course, this would be an
opt-in feature
for NFS servers.

For example, can we use fanotify permission hooks to delegate access control
checks from knfsd to a daemon? Right now, the information in
permission events is
rather minimal, but as an fanotify developer, I can assure you, that
we can enrich the
information passed by knfsd on open permission events if that is deemed useful.

I will be attending SambaXP, so if any of the Samba guys would like to, we could
find a slot in the Hallway track or at a local bar to discuss those options.

Thanks,
Amir.