[PATCH] Add CTDB_REQ_TUNNEL for new protocol

[Volker Lendecke]

On Fri, Oct 06, 2017 at 12:18:26PM +1100, Amitay Isaacs wrote:
> I think that's the wrong approach.  The current design of SRVID based
> messaging is designed to handle multiple recipients of the message.
> It's used in few places to handle "notifications" style broadcast.

I meant that we could add an option to the registration API to only
allow unique SRVID registrations when requested.

> 
> Samba has overloaded the same mechanism by having unique
> srvid to create uniquely distinguishable endpoints in the cluster.
> We should have used SERVER_ID based registrations for that
> purpose.  Unfortunately SERVER_ID system was never utilized
> in samba/ctdb, so I have dropped it.

That might be because I (and possibly others) never really understood
the difference or the SERVER_ID API properly. This persists until
today :-)

> We can still resurrect SERVER_ID based registration if we need
> a better mechanism to register unique endpoints in a cluster.
> There are sufficient details in server_id structure to even
> uniquely identify multiple CTDB connections from a single client.

Can you explain SRVID vs SERVER_ID in a bit more detail?

> It simply means that when you send a tunnel packet, it will be
> delivered to and endpoint on destination node with matching tunnel_id.
> That means you cannot tunnel messages between "service" daemons.
> For example, database daemon can only talk to other database
> daemons using a tunnel and not to cluster manager.

I may be stupid, but I still don't get the distinction between a
tunnel and SRVID style messages. Are there any restrictions or ACLs
for registring tunnel IDs? Or are you missing a central ID registry?
That would be easy to add as a convention on SRVIDs (which we already
have by means of the CTDB_SRVID_*_RANGE allocations.

Maybe we should use one of the next confcalls for a quick discussion
in person about this :-)

> > One thing for example the srvid based messaging misses is the ability
> > to listen for dying clients with a srvid. This could contribute to a
> > socket-like interface based on srvid messaging too I guess.
> >
> 
> Well we can definitely support that.  Referring to my earlier
> blurb on SRVID versus SERVER_ID, if we use SERVER_ID for
> endpoint registrations and SRVID as purely mechanism for
> messaging (and notifications), then a client going away can
> be sent easily as a broadcast message.  We can still do that
> in the current framework.  It just needs a dedicated SRVID for
> "client gone away" messages.
> 
> May be we need to sit down and work out the requirements
> for cluster-wide messaging.  Since we are  planning to separate
> various services, this might be ideal time to discuss messaging.

For me messaging, naming and id registration are just very tightly
coupled. In Samba we have the struct server_id (a PID on steroids) as
both a messaging target and existence check. I don't see the benefit
having two namespaces for those tasks. But it might really be that I
don't see the all requirements we have right now.

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de