SMB Direct Implementation via Linux Device Driver for Samba Design Doc, final?

[Richard Sharpe]

On Mon, Aug 26, 2013 at 4:29 AM, Or Gerlitz <ogerlitz at mellanox.com> wrote:
> On 26/08/2013 01:45, Richard Sharpe wrote:
>>
>> IOCTLS
>>
>> The following IOCTLS are needed:
>>
>> 1. SET_SMBD_PARAMETERS
>>
>> This IOCTL sets the set of parameters that SMB Direct operates under:
>>
>>    - ReceiveCreditMax
>>    - SendCreditMax
>>    - MaxSendSize
>>    - MaxFragmentSize
>>    - MaxReceiveSize
>>    - KeepaliveInterval
>>    - The initial security blob required to handle the SMB3 Negotiate
>> response.
>>
>> 2. SET_SMBD_SESSION_ID
>>
>> This ioctl tells the smbd driver the session ID in use by the current smbd
>> process and thus allows connections over RDMA using this session id.
>>
>> 3. GET_MEM_PARAMS
>>
>> This ioctl is used to retrieve important memory parameters established
>> when an
>> smbd opens the device. Each open after the first open allocates memory
>> that
>> will be used to receive and send PDUs as well as buffers to be used for
>> RDMA READs and WRITES.
>>
>> The information retrieved by this IOCTL includes the size of the memory
>> area
>> that the smbd should mmap against the device.
>>
>> 4. GET_SMBD_EVENT
>>
>> This ioctl is used by the smbd to retrieve the latest events from the
>> driver.
>> Events can be of the following type:
>>
>> a. PDU received
>> b. PDU sent
>> c. RDMA READ/WRITE complete and thus the buffers can be reused.
>>
>> A list of events is provided for the smbd to deal with.
>>
>> When PDUs received events are handled, the PDU will be copied into memory
>> pointed to by the event array passed in. The reason for this copy is to
>> allow
>> the SMB Direct protocol engine to turn its internal buffers around and
>> return
>> credits to the client. The cost of copying these PDUs is small in return
>> for
>> getting more requests in.
>>
>> The device will remain in a POLLIN state if there are outstanding events
>> to be handled.
>
>
> For getting nice IOPS numbers, its important to put elements into the design
> which will basically allow for
> cost amortization of user/kernel travels. E.g if multiple PDUs has arrives,
> the client should issue multiple
> IOCTLs to consume them all before it goes back it it epoll/select on the
> char-device fd.
>
> So maybe add an IOCTL that tells you how many events are there?

Hi Or,

These days (since about 2.6.12 or so) unlocked_ioctl does not run
under the BKL. I was planning on using that.

However, I was thinking of taking an approach like kqueue under
FreeBSD, where you can pass in an array of objects and ask the driver
to return as many as it can along with a count of the remaining events
etc available.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)