Some thoughts on the external recovery lock helper

[Ira Cooper]

On Fri, Jun 17, 2016 at 11:16 AM, Richard Sharpe <
realrichardsharpe at gmail.com> wrote:

> On Fri, Jun 17, 2016 at 8:10 AM, Ira Cooper <ira at wakeful.net> wrote:
> >
> >
> > On Fri, Jun 17, 2016 at 10:56 AM, Richard Sharpe
> > <realrichardsharpe at gmail.com> wrote:
> >>
> >>
> >> OK, so this part is the harder part. If the parent crashes then the
> >> helper needs to exit. It is not impossible, but a design where the
> >> helper is called as a (shared) library function gets this for free. We
> >> have lots of experience with this sort of thing in Samba.
> >>
> >> However, it is not impossible.
> >>
> >
> > Shouldn't we be able to detect that the pipe/socket that we are writing
> to
> > has closed?
>
> (Replied on-list this time.)
>
> Maybe so, however, what about a failure where the ctdb process is in
> an infinite loop somewhere and not responding to election requests,
> but is holding the lock. The external process thus does not get
> informed to release the lock.
>
> Of course, we could add keep-alives :-)
>

"What would we do today?"  The problem should be identical, if I understand
it correctly?

BTW:

Your API also doesn't cover the problem, you need a refresh call from CTDB
to confirm it wants to hold the lock.

Otherwise the library likely will have an infinite lease on the key, or
it'll have a thread refreshing it.  (which won't be in the infinite spin
probably ;) )

The infinite lease is the easiest implementation, and probably the one I'd
use personally.  I guess we need a keep alive.

Don't think about only one third party implementing this API.  I can see...
3-4 implementations off the top of my head?

etcd
consul
zookeeper
Ceph MON (Possibly, I have talked to people about it.  Nothing concrete
yet.  But they have PAXOS, there.)

Possibly Gluster could have its own API.  But I tend to doubt that today,
based on what I'm seeing.

In each case, the language of choice may be different.

In the case of etcd, this discussion actually is making me wonder if I
should write the "locking" code in go, instead of Python.

I chose Python because it is one of the languages we use in Samba.  But is
that a good reason not to use go here?

---

That said, if this integration grows deeper, I may change my mind, quite
easily.

The potential for pushing our "persistent" databases up into etcd and
friends is also quite real, and something we may want to think about as we
do this.

Cheers,

-Ira