SMB3 and RDMA support
realrichardsharpe at gmail.com
Sun May 18 14:00:57 MDT 2014
On Sun, May 18, 2014 at 11:39 AM, Michael Adam <obnox at samba.org> wrote:
>> Is there a strong reason for that extra daemon?
> This is the rdma-proxy-daemon (I called smbd-d, i.e.
> smb-direct-daemon for want of a better name).
> With the design we discussed, this smbd-d is only process that
> listens to rdma and answers rdma requests.
> We would like to omit that and treat rdma connections like other
> smbd-child processes, i.e. have main-smbd listen on rdma and fork
> a child process when a new rdma connection is accepted and
> fd-pass them to the already existing smbd process for the client.
I think that this is possible ... see below ...
> But we assumed up to now that the rdma client libs don't
> support forking with established connections. Hence the need
> for a single daemon (not forking children) to listen for RDMA
> connections and do the acual rdma traffic with all clients:
> the rdma-proxy-daemon.
This is possible, I believe with ibv_fork_init. Since we would only
perform RDMA requests in the one daemon (the master, under this
approach) this should work.
> In theory, I guess the main smbd could also take this task,
> but we wanted to make it a separate daemon so that the main
> smbd would not be prevented from doing its accept
> connection/fork business. And also, this separate rdma-proxy
> could be a fully multi-threaded daemon, thereby partly
> compensating for the lack of forking.
It could still be multi-threaded if need be.
> Now instead of having this bottleneck single proxy daemon, we
> would like to have a daemon that forks children for new
> connections and the fork function that might help here.
> (Btw, a prefork model might also just do the trick!!!...)
> But since we can make the single rdma-proxy-daemon
> multi-threaded, I am not certain how much going multi-process
> gains us if we can't fd-pass the rdma connection to
> another smbd for further treatment. Forking without fd-passing
> gives us one rdma-proxy-daemon per rdma-connection. The ideal
> solution would (or might) be the solution where there is no
> proxy daemon at all any more and the rdma traffic is done by
> the initial smbd process for the client.
I don't understand the comments about one RDMA deamon per smbd. I was
not suggesting that and it cannot be supported by the user-level RDMA
support in Linux or FreeBSD in any case.
> So let me now comment on the steps you listed:
> 0. main smbd is started.
>> 1. Not longer after starting, the master smbd forks an SMB Direct
>> daemon, smb-dd.
> Well the smbd-d might also be an independent daemon process,
> maybe started by main smbd, not certain yet. The smbd-d should
> definitely be a small special-purpose process, not a full
I now think that this is unnecessary and the RDMA work can be handled
by the master smbd. It might also work better with ibv_fork_init().
> Now the setup is this:
> - smbd-d listens for rdma connections.
> - smbd listens for tcp connections on smb ports.
>> 2. A TCP connection comes in to the master
>> 3. The master forks
> child smbd process "c1"
>> after it accepts and protocol processing happens
>> in the child. The TCP connection is closed in the master but kept open
>> in the child, so in some sense it is transferred to the child.
>> 4. An RDMA connection comes into the smb-dd. It is accepted and the
>> SMB Direct protocol starts.
>> 5. The smbd-d forks an smbd (the smbd-d is really just an smbd
>> anyhow), and communicates with this child via a UNIX-domain socket
>> (transfers the SMB requests and responses via this UNIX-domain
> No! The smbd-d does not fork a child. (At least not in the
> approach we discussed since we assumed that libibverbs can't
> cope with forking established rdma connections.)
That was not the intent. The child process would do no RDMA requests,
and would have been protected by ibv_fork_init().
In any event, I am no longer sure if it is the way to go.
> Instead, it sends a message with the information about the new
> rdma client connection to main-smbd. From here on it is a
> little different:
Well, that is a little complex ... but anyway.
> 6. main smbd forks a new child "c2" when receiving the
> message from the smbd-d.
> c2 establishes a unix domain socket connection to smbd-d
> this is a communication and proxy channel.
> Over this communication channel, it gets the smb
> information from the connected rdma client, including the
> client GUID.
> c2 finds that c1 already serves the client GUID and
> then transfers (even befor an rdma session bind)
> the unix socket (connecting to smbd-d) to c1 and
> then c2 dies.
> 7. c1 completes protocol processing (session bind, ...)
> and also establishes a mmap area with smbd-d.
> 8. Thereafter smb-direct-requests are proxied through
> the proxy-unix-domain connection, i.e. rmda
> send/recv calls are send over the socket from
> smbd-d to c2, which sends the answer back.
> And rdma write/read requests are treated as
> proper rdma requests which use the mmap area
> to access memory from c1.
> Makes sense?
Most of it does ... now it is just a matter of some code.
More information about the samba-technical