Best practices: Adding Persistent Handle support

[Christopher R. Hertel]

Responding to a few intertwined threads here...

Tom:
>From one pedant to another, your clear definitions of Continued Availability
and (vs.) Persistent Handles is quite helpful and appreciated.  From a
practical perspective, it seems to me that if the SMB3 server does not have
any shares that provide Continuous Availability, then it should not send
SMB2_GLOBAL_CAP_PERSISTENT_HANDLES since it cannot actually provide
Persistent Handles.

Metze:
I am leaving Resilient handles off of my immediate "To Do" list because, as
Tom explained, they are not quite the same as Durable and Persistent handles
in their behavior.  Still, any work done to implement Persistent handles
should move us closer to being able to do Resilient handles as well.

Thanks for your breakdown of the different levels.  I want to expose
Persistent handles in the VFS layer because different underlying file
systems may have different features that could be leveraged to support
Persistent Handles.

I'm going to study the existing Durable handle support to see if those VFS
functions can be overloaded to expose both Durable and Persistent handles.
That would be instead of adding a new brace of VFS functions to separately
implement Persistent handles, although I might still need to add an UPGRADE
function as you outlined.  This approach could turn out to be a bad idea,
but I also want to see if we can make Durable handles work across a cluster.

I will take a look first at the "top level" you described and see what needs
to be done to ensure that we can send SMB2_GLOBAL_CAP_PERSISTENT_HANDLES and
SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY in their appropriate places.  I still
think that the VFS module itself (the one supporting CA) should be in charge
of enabling these flags on the server.

Something to consider:  If the Persistent (and Durable) handles are exposed
in the VFS layer, and we can collect and maintain the required handle state
outside of the "glue layer" you described, then when we try to re-establish
a Persistent handle after a node crash we *should* be able to do so by
resynchronizing the "glue layer".  We should be able to do that during the
recovery phase.

Christoph:
This is the difficult bit of implementing behaviors that are outside of the
POSIX or Linux range of "normal".  We've got to negotiate how we're going to
do these things so that we cause the minimum amount of trouble for all of
the other stuff that people are trying to accomplish.

In this case, we're building on top of the SMB2_CREATE message which, in
turn, is built on top of a handful of Windows system calls plus some
features of the SMB3 protocol that are implemented by the protocol stack.
Persistent handles are one such feature.  If the SMB3 server is a member of
a CA (Continuous Availability) cluster and it crashes, then the client
should be able to connect to another member of that CA cluster and
re-establish the handle.  This, as Metze pointed out, requires maintaining
additional state.

Chris -)-----


On 06/02/2017 09:49 AM, Tom Talpey wrote:
>> -----Original Message-----
>> From: samba-technical [mailto:samba-technical-bounces at lists.samba.org] On
>> Behalf Of Stefan Metzmacher via samba-technical
>> Sent: Friday, June 2, 2017 7:25 AM
>> To: Christoph Hellwig <hch at infradead.org>; Christopher R. Hertel
>> <crh at samba.org>
>> Cc: Samba Technical <samba-technical at lists.samba.org>
>> Subject: Re: Best practices: Adding Persistent Handle support
>>
>> Hi Christoph,
>>
>>> On Thu, Jun 01, 2017 at 09:26:45PM -0500, Christopher R. Hertel via samba-
>> technical wrote:
>>>> I guess the reason my wording seemed convoluted is that I was
>>>> thinking of the situation in which, for example, you had three shares
>>>> on XFS file systems and another two shares on GlusterFS (or some
>>>> such).  In that case, you want to be specific about which VFS modules
>>>> you load for which particular shares.  You don't want to load a
>>>> Persistent Handle module on the XFS shares, for instance.
>>>
>>> For XFS shares (or in fact any Linux file system) you should use the
>>> open_by_handle_at and name_to_handle_at syscalls.  Of course I'm not
>>> too familar with the detailed rules of Samba structure, but as a Linux
>>> user and developer I'd expect these to be just available on a Linux
>>> setup without having to load magic modules.
>>
>> I guess 'struct file_handle' of open_by_handle_at() and
>> name_to_handle_at() is something difference, it's something like "open by a
>> unique number" instead of "open by path", is that correct?
>>
>> Persistent handles in SMB3 are more persistent "struct file"
>> (file description) instances, so all information about an open file is persistent,
>> e.g. the current seek pointer, all locks and all other properties.
> 
> I feel I should add that formally speaking, the persistent handle attribute is a property of the server, not the share (SMB2_GLOBAL_CAP_PERSISTENT_HANDLES). In fact, it's also a property of the client, both sides implement and advertise the capability.
> 
> Continuous availablility, however, is a property of the share (SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY). CA has more to do with the recovery scenario than the handle properties themselves. When a server advertises a share as CA, clients typically will request persistent handles during create. So both properties are needed to light it up.
> 
> Resilient handles, by the way, are more of an explicit client request. Resiliency is requested *after* handle creation, by an FSCTL_LMR_REQUEST_RESILIENCY. Once a handle is made resilient, in fact, durability is turned off (MS-SMB2 section 3.3.5.15.9). Resilient handles don't confer the same lock preservation semantics as durable/persistent, and they're sort of discouraged in the SMB3 age.
> 
> Tom.
> 
>>
>> So if the SMB3 client gets disconnected for a while, the open handle (which is
>> basically a file descriptor referencing a file description) is still open and the
>> client can resume this "open", without the need of re-adding locks or doing
>> other recover actions.
>>
>> What Samba will require from the kernel will be:
>> - a call that takes an fd and a timeout value and hands out some kind of
>>   cookie/id that references the file description of the given fd
>>   and makes sure that the file description remains at for the given
>>   timeout without having a remaining file descriptor referencing the
>>   file description.
>> - a call to "resume" that takes the cookie/id and returns an fd
>>   that references the file description and resets the expiration timer.
>>
>> In SMB2/3 there're a few different flavors of these "disconnected"
>> handles, we have:
>>
>> - durable handles (which require an exclusive oplock),
>>   these are best effort by the server, but if a 2nd client
>>   opens the file, the disconnected handle is closed,
>>   so the cookie/id would be invalidated.
>>
>> - resilient handles (which are explicitly requested by the application
>>   layer): The still running server (kernel) would keep the file
>>   description valid for the specified timeout, even if another
>>   client (not knowing the cookie/id) will open the file.
>>
>> - persistent handles (are requested if the share is marked as
>>   "continuously available"): These are like resilient handles,
>>   but they're also persistent across reboots.
>>
>>
>> A possible design for new syscalls will require thinking about much more
>> details, but I hope this a useful overview.
>>
>> metze
>