Work on SMB3 persistent handles

Thu Oct 26 18:37:21 UTC 2017

On 10/25/2017 11:48 AM, Albert Guo wrote:
> Hi Chris,

Hi, Albert,

> Are you going to implement persistent file handles based on some interfaces
> provided by cluster file system or based on CTDB?

As Michael mentioned, my own goal is to expose the interfaces so that you,
or I, or some other developer can adapt the solution to their platform with
ease.

> VMware only concerns about the former one.

Unfortunately, much of the work that others have put into this has been
quite focused on CTDB.  That's one reason that it is useful to you and me
and others to separate out the protocol changes from the changes to the
backing store.  If we can get the protocol bits right, then we can tackle
the backing store separately.

To that point, it is worth noting that Persistent Handles *can be tested*
without having a working persistent backing store.  Persistent Handles (as
you have already discovered) can be tested by breaking the network
connection.  The client will attempt to re-establish the connection and will
request the restoration of the Persistent Handle.

> If the cluster file system provides a small fast POSIX file system which is
> used to store the TDBs, would that be sufficient for supporting persistent
> file handles?

That depends entirely on how you want to implement the persistent backing
store.  I am aware of one solution (which, alas, was never released) that
worked like this:

For each server IP address (aka "public address" in CTDB-speak), there was a
TDB file stored on cluster-shared storage.  Only the node serving on the
given IP address had the database open so no cross-cluster locking was needed.

In the event of a node failure, the IP address(es) would move to active
server nodes, and so would access to the TDB files.

> As regard to the fast POSIX file system, it could be a memory based file
> system which is replicated to at least 2 nodes, or it can be on NVMe or SSD etc.

These are implementation details so I'll put them aside, except to say that
they illustrate exactly why it is that we believe it is really important to
be able to expose the backing store.

> In our architecture, different nodes serve different shares, so the same TDB
> won’t be accessed/competed by different hosts simultaneously. Hopefully the
> performance won’t be very bad.

:-)

> In order to ensure any SMB request are processed transactionally. i.e. Samba
> can recover itself to a consistent state whenever one node crashes. I think
> we’ll need the following guarantee for entries related to persistent file
> handles:
> 
> ·          Update to a single TDB is transactional.  Looks like current TDB
> only support a single transaction, making it transactional might downgrade
> its performance dramatically.
> 
> ·          It’s possible to update multiple TDBs when process a single SMB
> request like open. We need to ensure the update to those multiple TDBs are
> transactional as well. Probably we may need to introduce an assistant TDB
> for ensuring the transactional semantic of processing a single SMB request.
> 
> Thoughts?

Several.  Hopefully they're helpful.

- Below you mention that you removed TDB_CLEAR_IF_FIRST from the open
  calls to locking.tdb and smbXsrv_open_global.tdb.  I did the same as
  a test, but I also removed TDB_CLEAR_IF_FIRST from the
  byte-range-locking and leases databases.  These two additional
  databases are also used to maintain handle state.

  Unfortunately, the change resulted in random core dumps related to
  libtevent.

- My current goal is to merge my changes with the changes Michael
  posted and update the merged set with master.  Given the fixes that
  Michael has provided, we should be able to get full protocol support
  for Persistent Handles.  Then we only need to work on the backing
  store.

- Regarding the backing store...  All four of the databases we've
  identified are opened and accessed via the DBwrap API.  DBwrap hides
  the differences between CTDB and TDB, and also provides for
  "transactions".  The problem is that DBwrap is somewhat specific to
  [C]TDB.  We may be able to work around that, though.

- My hope had been that we could expose the relevant calls via
  Samba's VFS layer.  I still think that's a worthy goal but we may
  need to get there incrementally.

Chris -)-----

> On 10/26/17, 12:09 AM, "Albert Guo" <aguo at vmware.com> wrote:
> 
>  
> 
>     Hi Chris,
> 
>     I’ve already checked out your code from the CA branch and played with it.
> 
>    
> 
>     I made a POC based on your git repository with some hack.  This is test
> I did:
> 
>     1.                Copy a large file to a CA enabled share from Windows
> 2012 R2 client
> 
>     2.                Restart Samba daemons
> 
>     The persistent file handle is reconnected successfully, and the file
> copy continues without any error.
> 
>     The purpose of the POC is to identify the gaps we have for supporting
> persistent file handles.
> 
>    
> 
>     I attached my patch in the attachment. The patch is very simple:
> 
>     1.                Remove TDB_CLEAR_IF_FIRST from open path of
> locking.tdb and smbXsrv_open_global.tdb
> 
>     2.                Disable the code for checking whether server_id is
> disconnected.
> 
>     
> 
>     I assume you’re trying to store server_id when Samba deamon is shutdown
> so that persistent file handle can be reconnected successfully?
> 
>    
> 
>     --
> 
>     Thanks!
> 
>     Albert
> 
>     
> 
>     
> 
>     On 10/25/17, 2:17 AM, "Christopher R. Hertel" <crh at samba.org> wrote:
> 
>    
> 
>         On 10/24/2017 10:29 AM, Michael Adam wrote:
> 
>         :
> 
>         :
> 
>         > Chris, feel free to use / merge / discard as you see fit.
> 
>         > (I'd guess Metze would agree).
> 
>        
> 
>         I will do my best to merge the solutions so that they can be applied
> against
> 
>         master.
> 
>        
> 
>         > Also I'd be interested to see any WIP patches of yours,
> 
>         > no matter how incomplete.
> 
>        
> 
>         git://git.samba.org/crh/continuousavailability.git
> 
>        
> 
>         That code is testable as-is.  It has no negative impact on the rest of
> 
>         Samba, and succeeds at negotiating Persistent Handles (per Session) and
> 
>         Continuous Availability (per share).  It does not yet have the changes
> 
>         needed to actually store the Persistent Handle data in a recoverable
> way,
> 
>         but that's not necessary for initial testing.
> 
>        
> 
>         Persistent Handles should be recoverable even without persistent backing
> 
>         store.  In a network outage, a Persistent Handle would work the same
> way as
> 
>         a Durable Handle, so a test that simply drops the TCP connection and
> then
> 
>         reconnects should get correct Persistent Handle behavior.
> 
>        
> 
>         I think that Ralph had some smbtorture code that could do some of
> this testing.
> 
>        
> 
>         What I am working on now is making sure that the code in my repository
> 
>         provides a correct response to a Persistent Handle reconnect.  If I
> can get
> 
>         that part working, then the only part missing will be the actual
> persistent
> 
>         data storage/recovery.
> 
>        
> 
>         Chris -)-----
> 
>        
> 
>     
> 
>     
> 