Corrected diagram and description for SMB-Direct (RDMA) support in Samba
realrichardsharpe at gmail.com
Sun May 18 23:58:17 MDT 2014
On Sun, May 18, 2014 at 10:46 PM, Michael Adam <obnox at samba.org> wrote:
> Hi Richard,
> this is a good diagram, thank you!
> I think we have a common understanding now.
> Just one detail regarding #7 in your list:
> Like with multi-channel, our plan was to have the
> temporary child do as little processing as possible,
> i.e. pass the connection fd over to the client's
> already existing smbd as early as possible.
> The clientGUID is already available in the
> NegProt request, so we can look for the client's
> smbd by clientGUID already here, and have c1 do
> the session setup.
Ummm, that is true. I have checked the SMB2 document. That being the
case, we really should sanction the small layering violation and have
the SMBDirect daemon look in the NegProt request and send the
connection to the correct smbd. Why mess around with starting a
temporary smbd for one request/response pair?
> This has the advantage of requiring less state
> to be transferred an re-integrated. It has the
> (rather theoretical?) disadvantage if a client
> does multiple independent smb sessions with the
> same clientGUID, then these will be handled by
> one instead of two smbd processes.
> Let me add the process to the wiki, and
> (with your permission) also add the diagram.
> Cheers - Michael
> On 2014-05-18 at 16:29 -0700, Richard Sharpe wrote:
>> Hi folks,
>> I have redrawn my diagram and it has gotten more complex. It is attached.
>> The steps are:
>> 0. The master smbd starts.
>> 1. It does a fork and exec or otherwise starts the smbdirectd. This is
>> the only process that uses RDMA via librdmacm, libibverbs, etc.
>> 2. A tcp connection from a client arrives at the master smbd.
>> 3. The master smbd forks a child smbd 'c1' to handle the connection.
>> The child handles all client SMB requests, including, most likely, an
>> FSCTL to return the interfaces. The response to this would include the
>> RDMA interface
>> 4. An RDMA connection comes in to the smbdirect daemon, perhaps in
>> response to the client connected to smbd 'c1' finding out that it has
>> an RDMA interface in common with the server. This is accepted by the
>> smbdirect daemon.
>> 5. When the SMB Direct channel is established and the first SMB
>> arrives, the smbdirect daemon informs the master smbd and provides it
>> with the name of a UNIX domain socket to use.
>> 6. The master smbd now forks a new child smbd, 'c2', which uses the
>> supplied UNIX domain socket to communicate with the smbddirect daemon.
>> 7. The child smbd, c2, now handles the first part of the SMB protocol
>> over the supplied UNIX domain socket. This includes the
>> NegotiateProtocol and SessionSetup request. Upon receiving the
>> SessionSetup request, c2 uses the clientGuid to determine which child
>> smbd should handle the connection. In this case, it is c1. It might
>> find this mapping by looking in the connections TDB, for example.
>> 8. The child smbd, c2, now transfers the UNIX domain socket to c1 and exits.
>> After that, c1 will handle both connections, one directly, and one via
>> the UNIX domain socket connection it has to the smbdirect daemon.
>> For those SMB requests that use RDMA READ/WRITE requests (, ie, for
>> which it has descriptors) it uses a shared-memory area established by
>> the smbdirect daemon for this purpose.
>> If Michael and I are now on the same page, we can move on to other issues.
>> Richard Sharpe
More information about the samba-technical