SMB3 and RDMA support

[Richard Sharpe]

On Sun, May 18, 2014 at 11:39 AM, Michael Adam <obnox at samba.org> wrote:
>> Is there a strong reason for that extra daemon?
>
> This is the rdma-proxy-daemon (I called smbd-d, i.e.
> smb-direct-daemon for want of a better name).
>
> With the design we discussed, this smbd-d is only process that
> listens to rdma and answers rdma requests.
>
> We would like to omit that and treat rdma connections like other
> smbd-child processes, i.e. have main-smbd listen on rdma and fork
> a child process when a new rdma connection is accepted and
> fd-pass them to the already existing smbd process for the client.

I think that this is possible ... see below ...

> But we assumed up to now that the rdma client libs don't
> support forking with established connections.  Hence the need
> for a single daemon (not forking children) to listen for RDMA
> connections and do the acual rdma traffic with all clients:
> the rdma-proxy-daemon.

This is possible, I believe with ibv_fork_init. Since we would only
perform RDMA requests in the one daemon (the master, under this
approach) this should work.

> In theory, I guess the main smbd could also take this task,
> but we wanted to make it a separate daemon so that the main
> smbd would not be prevented from doing its accept
> connection/fork business. And also, this separate rdma-proxy
> could be a fully multi-threaded daemon, thereby partly
> compensating for the lack of forking.

It could still be multi-threaded if need be.

> Now instead of having this bottleneck single proxy daemon, we
> would like to have a daemon that forks children for new
> connections and the fork function that might help here.
> (Btw, a prefork model might also just do the trick!!!...)
>
> But since we can make the single rdma-proxy-daemon
> multi-threaded, I am not certain how much going multi-process
> gains us if we can't fd-pass the rdma connection to
> another smbd for further treatment. Forking without fd-passing
> gives us one rdma-proxy-daemon per rdma-connection. The ideal
> solution would (or might) be the solution where there is no
> proxy daemon at all any more and the rdma traffic is done by
> the initial smbd process for the client.

I don't understand the comments about one RDMA deamon per smbd. I was
not suggesting that and it cannot be supported by the user-level RDMA
support in Linux or FreeBSD in any case.

> So let me now comment on the steps you listed:
>
> 0. main smbd is started.
>> 1. Not longer after starting, the master smbd forks an SMB Direct
>> daemon, smb-dd.
>
> Well the smbd-d might also be an independent daemon process,
> maybe started by main smbd, not certain yet. The smbd-d should
> definitely be a small special-purpose process, not a full
> smbd-child.

I now think that this is unnecessary and the RDMA work can be handled
by the master smbd. It might also work better with ibv_fork_init().

> Now the setup is this:
>
> - smbd-d listens for rdma connections.
> - smbd listens for tcp connections on smb ports.
>
>> 2. A TCP connection comes in to the master
>>
>> 3. The master forks
>
> child smbd process "c1"
>
>> after it accepts and protocol processing happens
>> in the child. The TCP connection is closed in the master but kept open
>> in the child, so in some sense it is transferred to the child.
>>
>> 4. An RDMA connection comes into the smb-dd. It is accepted and the
>> SMB Direct protocol starts.
>>
>> 5. The smbd-d forks an smbd (the smbd-d is really just an smbd
>> anyhow), and communicates with this child via a UNIX-domain socket
>> (transfers the SMB requests and responses via this UNIX-domain
>> socket.)
>
> No! The smbd-d does not fork a child. (At least not in the
> approach we discussed since we assumed that libibverbs can't
> cope with forking established rdma connections.)

That was not the intent. The child process would do no RDMA requests,
and would have been protected by ibv_fork_init().

In any event, I am no longer sure if it is the way to go.

> Instead, it sends a message with the information about the new
> rdma client connection to main-smbd. From here on it is a
> little different:

Well, that is a little complex ... but anyway.

> 6. main smbd forks a new child "c2" when receiving the
>    message from the smbd-d.
>    c2 establishes a unix domain socket connection to smbd-d
>    this is a communication and proxy channel.
>    Over this communication channel, it gets the smb
>    information from the connected rdma client, including the
>    client GUID.
>
>    c2 finds that c1 already serves the client GUID and
>    then transfers (even befor an rdma session bind)
>    the unix socket (connecting to smbd-d) to c1 and
>    then c2 dies.
>
> 7. c1 completes protocol processing (session bind, ...)
>    and also establishes a mmap area with smbd-d.
>
> 8. Thereafter smb-direct-requests are proxied through
>    the proxy-unix-domain connection, i.e. rmda
>    send/recv calls are send over the socket from
>    smbd-d to c2, which sends the answer back.
>    And rdma write/read requests are treated as
>    proper rdma requests which use the mmap area
>    to access memory from c1.
>
> Makes sense?

Most of it does ... now it is just a matter of some code.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)