[PATCH] Add CTDB_REQ_TUNNEL for new protocol

[Amitay Isaacs]

On Fri, Oct 6, 2017 at 7:14 PM, Volker Lendecke <Volker.Lendecke at sernet.de>
wrote:

> On Fri, Oct 06, 2017 at 12:18:26PM +1100, Amitay Isaacs wrote:
> > I think that's the wrong approach.  The current design of SRVID based
> > messaging is designed to handle multiple recipients of the message.
> > It's used in few places to handle "notifications" style broadcast.
>
> I meant that we could add an option to the registration API to only
> allow unique SRVID registrations when requested.
>
> >
> > Samba has overloaded the same mechanism by having unique
> > srvid to create uniquely distinguishable endpoints in the cluster.
> > We should have used SERVER_ID based registrations for that
> > purpose.  Unfortunately SERVER_ID system was never utilized
> > in samba/ctdb, so I have dropped it.
>
> That might be because I (and possibly others) never really understood
> the difference or the SERVER_ID API properly. This persists until
> today :-)
>
> > We can still resurrect SERVER_ID based registration if we need
> > a better mechanism to register unique endpoints in a cluster.
> > There are sufficient details in server_id structure to even
> > uniquely identify multiple CTDB connections from a single client.
>
> Can you explain SRVID vs SERVER_ID in a bit more detail?
>

Well as I understood from ctdb code, SRVID is wrongly named. It
should have been msgid (or something similar) since it's used to
send and receive messages.  In CTDB, SRVID is used strictly to
refer to a particular message.  So many endpoints (including
smbd) can register for specific messages using SRVID.  The
whole api involving SRVID is designed to distribute messages
to multiple endpoints across the cluster that have registered
for the specific SRVID.

As you correctly mentioned, SERVER_ID is PID on steroids.  We
can uniquely identify each endpoint in the cluster using "vnn" and
"pid" alone.  If we want to identify multiple connections to CTDB
from a single endpoint, we can use "task_id".  There is really
no need for yet another mechanism to uniquely identify an
endpoint.

One of the reasons for using "unique_id" would be to identify
an endpoint using well known ids.  Of course this requires
cooperation of all CTDB apps since unique ids will have to be
pre-defined.

Another reason is to uniquely identify a process in time,
given that pids are re-used.

Also, another reason for using "unique_id" would be to send
messages to specific endpoints using already established SRVID
messaging mechanism.Here is where unique_id would be the same
as SRVID from the current use.  In that case, the "unique_id"
needs to be assigned from a specific range for each app.
In this case, SRVID is NOT really used as a message id, but
purely re-using the existing mechanism for sending messages.
One can send different messages to an endpoint and establish
a sub-protocol.  This is the point of contention.

There is a defined range of SRVID for smbd (even though
I think it should be renamed a PREFIX rather than a RANGE)
CTDB_SRVID_SAMBA_RANGE, it isn't used.  smbd generates
unique ids with generate_random_buffer() which can potentially
generate any of the pre-defined SRVIDs.  There is no attempt
to ensure that it's actually unique across the cluster.


> It simply means that when you send a tunnel packet, it will be
> > delivered to and endpoint on destination node with matching tunnel_id.
> > That means you cannot tunnel messages between "service" daemons.
> > For example, database daemon can only talk to other database
> > daemons using a tunnel and not to cluster manager.
>
> I may be stupid, but I still don't get the distinction between a
> tunnel and SRVID style messages. Are there any restrictions or ACLs
> for registring tunnel IDs? Or are you missing a central ID registry?
> That would be easy to add as a convention on SRVIDs (which we already
> have by means of the CTDB_SRVID_*_RANGE allocations.
>

Conceptually there is no difference between tunnel and SRVID style
messages.  It's simply a way to avoid ever sending protocol packets
between service daemons to smbd.  I strongly feel that the
daemon-to-daemon (ctdb-ctdb) protocol should have been separate
from daemon-to-client (ctdb-smbd) protocol.  It unnecessarily
exposes inner working of ctdb and requires understanding of inner
working to use ctdb.  This is another design issue I want to
consider when separating service daemons.  Even though this
may result in some duplication of protocol, it's better than to
avoid introducing hacks and special cases to distinguish between
daemon-daemon and daemon-client communication.


> Maybe we should use one of the next confcalls for a quick discussion
> in person about this :-)
>
> > > One thing for example the srvid based messaging misses is the ability
> > > to listen for dying clients with a srvid. This could contribute to a
> > > socket-like interface based on srvid messaging too I guess.
> > >
> >
> > Well we can definitely support that.  Referring to my earlier
> > blurb on SRVID versus SERVER_ID, if we use SERVER_ID for
> > endpoint registrations and SRVID as purely mechanism for
> > messaging (and notifications), then a client going away can
> > be sent easily as a broadcast message.  We can still do that
> > in the current framework.  It just needs a dedicated SRVID for
> > "client gone away" messages.
> >
> > May be we need to sit down and work out the requirements
> > for cluster-wide messaging.  Since we are  planning to separate
> > various services, this might be ideal time to discuss messaging.
>
> For me messaging, naming and id registration are just very tightly
> coupled. In Samba we have the struct server_id (a PID on steroids) as
> both a messaging target and existence check. I don't see the benefit
> having two namespaces for those tasks. But it might really be that I
> don't see the all requirements we have right now.
>
>
There is no namespace separation. They are closely related. An endpoint is
uniquely identified in cluster using server_id (or subset of it), srvid
alone is
not sufficient. Even though smbd only uses srvid for registration, ctdb
converts that internally to information similar to struct server_id by
querying pid using the socket.  SRVID alone is just for messaging.

The only reason for introducing tunnels is to overcome the limitation
in the srvid based registration.  If we resurrect server_id based
registration with improvements, and both smbd & service daemons
are using the same registration mechanism, then we stick to using
srvid based messaging and sub-protocol definition.

Amitay.