Fake Failover with Durable Handles

[Christopher R. Hertel]

On 08/09/2017 10:46 AM, Stefan Metzmacher via samba-technical wrote:

> 
> Hi Chris,
> 
> I just had the chance to listen the recording of your
> SambaXP talk:
> https://sambaxp.org/archive_data/SambaXP2017-AUDIO/Day3/Track1/Can%20we%2=
> 0fake%20a%20Failover.mp3
> https://sambaxp.org/archive_data/SambaXP2017-SLIDES/Day3/Track1/Can%20we%=
> 20Fake%20a%20Failover%20-%20Christopher%20Hertel.pdf
> 
> Do you realize that durable handle failover in Samba also
> works in a ctdb cluster? For Samba there's no difference.

That would explain why the Durable Cookie data is NDR encoded.
...and no, I was not aware that we could do that.  Thanks for letting me
know.  That's good news.

> This works fine for the case you just have network problems
> and the client reconnects (even to another node).

Okay...

> The things which doesn't work are reconnects if
> the connected nodes dies or ctdbd terminates.

Okay, that makes sense too.

> In order to do maintenance of a node you need to
> remove all public address away from a node, then
> wait for the durable timeout to pass in order to allow
> the reconnects while ctdbd is still running on the node.
> Then you can terminate ctdbd.

Very good to know.

> What kind of thing does your "Fake Failover" handle beyond
> this things we already have?

I had discussed Persistent Handles and Durable Handles with my employer, and
the idea was presented that we could create "Fake Failover" using Durable
Handles.  If the Durable Handle data is stored in backing store in the
cluster, then we could (in theory) recover that handle even after a node
crash.  That was the theory.  It's a bit more difficult because the timing
is tighter.

So I started digging into the Durable Handle support in Samba to see if we
could simply copy critical data in the FSP and Durable Cookie structures
into backing store, so that it could be recovered after a single node crash.
 The hardest part there was going to be figuring out which data needed to be
stored and which could be re-created from available information.  For
example, the cookie stores a lot of stat data that doesn't need to be in
backing store because it's already available as meta-data.

I did my presentation based on that info.  I still believe that we could do
something along those lines, but before I could go further down that path I
had a long discussion with the other folks at my company and we decided it
would be better to aim at Persistent Handles first.

So...  Nothing new on recoverable Durable Handles yet.

In theory, however, if there is a backing store which is write-only while
the cluster is running and is only used to rebuild the Durable Handle data
after a failover, we might be able to squeeze into the timeout and provide
"fake failover".

Chris -)-----

PS.  Yes, I know that the code I've submitted so far is only enough to set
the bits on the wire.  It's a necessary first step, however, and I wanted to
get it out there for review, or for anyone else who wants to work on this
stuff.  :-)