Corrected diagram and description for SMB-Direct (RDMA) support in Samba
obnox at samba.org
Mon May 19 03:04:12 MDT 2014
After talking to Metze, I have to add one additional
When the rdma-proxy-daemon has received a connection,
instead of sending a message to main smbd, and
have c2 establish a connection to the rdma-proxy,
upon receiving the rdma-connection, the proxy
creates a socket-pair (or even two) and passes
it via fd-passing message to main smbd, which
inherits it via fort to c2, which passes it on
to c1 after getting ClientGUID from negprot.
This is much more elegant and direct.
Cheers - Michael
On 2014-05-19 at 07:46 +0200, Michael Adam wrote:
> Hi Richard,
> this is a good diagram, thank you!
> I think we have a common understanding now.
> Just one detail regarding #7 in your list:
> Like with multi-channel, our plan was to have the
> temporary child do as little processing as possible,
> i.e. pass the connection fd over to the client's
> already existing smbd as early as possible.
> The clientGUID is already available in the
> NegProt request, so we can look for the client's
> smbd by clientGUID already here, and have c1 do
> the session setup.
> This has the advantage of requiring less state
> to be transferred an re-integrated. It has the
> (rather theoretical?) disadvantage if a client
> does multiple independent smb sessions with the
> same clientGUID, then these will be handled by
> one instead of two smbd processes.
> Let me add the process to the wiki, and
> (with your permission) also add the diagram.
> Cheers - Michael
> On 2014-05-18 at 16:29 -0700, Richard Sharpe wrote:
> > Hi folks,
> > I have redrawn my diagram and it has gotten more complex. It is attached.
> > The steps are:
> > 0. The master smbd starts.
> > 1. It does a fork and exec or otherwise starts the smbdirectd. This is
> > the only process that uses RDMA via librdmacm, libibverbs, etc.
> > 2. A tcp connection from a client arrives at the master smbd.
> > 3. The master smbd forks a child smbd 'c1' to handle the connection.
> > The child handles all client SMB requests, including, most likely, an
> > FSCTL to return the interfaces. The response to this would include the
> > RDMA interface
> > 4. An RDMA connection comes in to the smbdirect daemon, perhaps in
> > response to the client connected to smbd 'c1' finding out that it has
> > an RDMA interface in common with the server. This is accepted by the
> > smbdirect daemon.
> > 5. When the SMB Direct channel is established and the first SMB
> > arrives, the smbdirect daemon informs the master smbd and provides it
> > with the name of a UNIX domain socket to use.
> > 6. The master smbd now forks a new child smbd, 'c2', which uses the
> > supplied UNIX domain socket to communicate with the smbddirect daemon.
> > 7. The child smbd, c2, now handles the first part of the SMB protocol
> > over the supplied UNIX domain socket. This includes the
> > NegotiateProtocol and SessionSetup request. Upon receiving the
> > SessionSetup request, c2 uses the clientGuid to determine which child
> > smbd should handle the connection. In this case, it is c1. It might
> > find this mapping by looking in the connections TDB, for example.
> > 8. The child smbd, c2, now transfers the UNIX domain socket to c1 and exits.
> > After that, c1 will handle both connections, one directly, and one via
> > the UNIX domain socket connection it has to the smbdirect daemon.
> > For those SMB requests that use RDMA READ/WRITE requests (, ie, for
> > which it has descriptors) it uses a shared-memory area established by
> > the smbdirect daemon for this purpose.
> > If Michael and I are now on the same page, we can move on to other issues.
> > --
> > Regards,
> > Richard Sharpe
> > (何以解憂？唯有杜康。--曹操)
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 198 bytes
Desc: Digital signature
More information about the samba-technical