[PATCH] Add CTDB_REQ_TUNNEL for new protocol

[Amitay Isaacs]

On Fri, Oct 6, 2017 at 1:24 AM, Volker Lendecke <Volker.Lendecke at sernet.de>
wrote:

> On Fri, Oct 06, 2017 at 01:01:31AM +1100, Amitay Isaacs wrote:
> > Are you talking about source3/lib/messages_ctdb.[ch]?
>
> That would be the Samba-side user of what I mean, sure. On a more
> basic level we do
>
> r.hdr.operation  = CTDB_REQ_MESSAGE;
>
> to send messages.
>
> > As you might have guessed, the tunnel implementation is pretty similar
> > to srvid mesages.  One of the main differences is you can register
> > an endpoint with unique id only once on a node.  It's essential that
> > there is only one "service" daemon (e.g. database daemon) running
> > on a node.
>
> Oh, that would be a very worthwhile addition to
> CTDB_CONTROL_REGISTER_SRVID :-)
>

I think that's the wrong approach.  The current design of SRVID based
messaging is designed to handle multiple recipients of the message.
It's used in few places to handle "notifications" style broadcast.

Samba has overloaded the same mechanism by having unique
srvid to create uniquely distinguishable endpoints in the cluster.
We should have used SERVER_ID based registrations for that
purpose.  Unfortunately SERVER_ID system was never utilized
in samba/ctdb, so I have dropped it.

We can still resurrect SERVER_ID based registration if we need
a better mechanism to register unique endpoints in a cluster.
There are sufficient details in server_id structure to even
uniquely identify multiple CTDB connections from a single client.


> > Also, you cannot tunnel a packet between two different
> > tunnel ids.
>
> What do you mean by that exactly?
>

It simply means that when you send a tunnel packet, it will be
delivered to and endpoint on destination node with matching tunnel_id.
That means you cannot tunnel messages between "service" daemons.
For example, database daemon can only talk to other database
daemons using a tunnel and not to cluster manager.


> One thing for example the srvid based messaging misses is the ability
> to listen for dying clients with a srvid. This could contribute to a
> socket-like interface based on srvid messaging too I guess.
>

Well we can definitely support that.  Referring to my earlier
blurb on SRVID versus SERVER_ID, if we use SERVER_ID for
endpoint registrations and SRVID as purely mechanism for
messaging (and notifications), then a client going away can
be sent easily as a broadcast message.  We can still do that
in the current framework.  It just needs a dedicated SRVID for
"client gone away" messages.

May be we need to sit down and work out the requirements
for cluster-wide messaging.  Since we are  planning to separate
various services, this might be ideal time to discuss messaging.


> > I feel this will ensure that the design of a "service" will
> > remain coherent and there will be cleaner interfaces between services.
> > That's why I decided to avoid srvid based messaging to forgo the
> > temptation of violating inter-service interfaces.
>
> It's your design, your code, of course. But I feel that we have a
> transport that can carry anything already, and if there's one thing
> ctdb does not need is more complexity :-)
>

I agree with you regarding not needing more complexity.  However,
in the current design, there are many special cases and short-cuts
taken to end up with a spaghetti bowl.  The tunnel is conceived
only as an interim measure to keep the new code design clean
and not mix with the existing mess.  It's supposed to be replaced
with new transport once we get rid of the monolithic ctdb daemon.
That's the reason Martin and I have been aggressively removing
unused protocol elements (mainly controls) to keep support effort
for "legacy" protocol to a minimum going forward.

I am open to suggestions.  We definitely should discuss
this if we can avoid unnecessary re-designs.

Amitay.