[PATCH v4] smb: server: allocate enough space for RW WRs and ib_drain_qp()

[Stefan Metzmacher]

Am 17.10.25 um 08:34 schrieb Namjae Jeon:
> On Thu, Oct 16, 2025 at 11:21 PM Stefan Metzmacher <metze at samba.org> wrote:
>>
>> Make use of rdma_rw_mr_factor() to calculate the number of rw
>> credits and the number of pages per RDMA RW operation.
>>
>> We get the same numbers for iWarp connections, tested
>> with siw.ko and irdma.ko (in iWarp mode).
>>
>> siw:
>>
>> CIFS: max_qp_rd_atom=128, max_fast_reg_page_list_len = 256
>> CIFS: max_sgl_rd=0, max_sge_rd=1
>> CIFS: responder_resources=32 max_frmr_depth=256 mr_io.type=0
>> CIFS: max_send_wr 384, device reporting max_cqe 3276800 max_qp_wr 32768
>> ksmbd: max_fast_reg_page_list_len = 256, max_sgl_rd=0, max_sge_rd=1
>> ksmbd: device reporting max_cqe 3276800 max_qp_wr 32768
>> ksmbd: Old sc->rw_io.credits: max = 9, num_pages = 256
>> ksmbd: New sc->rw_io.credits: max = 9, num_pages = 256, maxpages=2048
>> ksmbd: Info: rdma_send_wr 27 + max_send_wr 256 = 283
>>
>> irdma (in iWarp mode):
>>
>> CIFS: max_qp_rd_atom=127, max_fast_reg_page_list_len = 262144
>> CIFS: max_sgl_rd=0, max_sge_rd=13
>> CIFS: responder_resources=32 max_frmr_depth=2048 mr_io.type=0
>> CIFS: max_send_wr 384, device reporting max_cqe 1048574 max_qp_wr 4063
>> ksmbd: max_fast_reg_page_list_len = 262144, max_sgl_rd=0, max_sge_rd=13
>> ksmbd: device reporting max_cqe 1048574 max_qp_wr 4063
>> ksmbd: Old sc->rw_io.credits: max = 9, num_pages = 256
>> ksmbd: New sc->rw_io.credits: max = 9, num_pages = 256, maxpages=2048
>> ksmbd: rdma_send_wr 27 + max_send_wr 256 = 283
>>
>> This means that we get the different correct numbers for ROCE,
>> tested with rdma_rxe.ko and irdma.ko (in RoCEv2 mode).
>>
>> rxe:
>>
>> CIFS: max_qp_rd_atom=128, max_fast_reg_page_list_len = 512
>> CIFS: max_sgl_rd=0, max_sge_rd=32
>> CIFS: responder_resources=32 max_frmr_depth=512 mr_io.type=0
>> CIFS: max_send_wr 384, device reporting max_cqe 32767 max_qp_wr 1048576
>> ksmbd: max_fast_reg_page_list_len = 512, max_sgl_rd=0, max_sge_rd=32
>> ksmbd: device reporting max_cqe 32767 max_qp_wr 1048576
>> ksmbd: Old sc->rw_io.credits: max = 9, num_pages = 256
>> ksmbd: New sc->rw_io.credits: max = 65, num_pages = 32, maxpages=2048
>> ksmbd: rdma_send_wr 65 + max_send_wr 256 = 321
>>
>> irdma (in RoCEv2 mode):
>>
>> CIFS: max_qp_rd_atom=127, max_fast_reg_page_list_len = 262144,
>> CIFS: max_sgl_rd=0, max_sge_rd=13
>> CIFS: responder_resources=32 max_frmr_depth=2048 mr_io.type=0
>> CIFS: max_send_wr 384, device reporting max_cqe 1048574 max_qp_wr 4063
>> ksmbd: max_fast_reg_page_list_len = 262144, max_sgl_rd=0, max_sge_rd=13
>> ksmbd: device reporting max_cqe 1048574 max_qp_wr 4063
>> ksmbd: Old sc->rw_io.credits: max = 9, num_pages = 256,
>> ksmbd: New sc->rw_io.credits: max = 159, num_pages = 13, maxpages=2048
>> ksmbd: rdma_send_wr 159 + max_send_wr 256 = 415
>>
>> And rely on rdma_rw_init_qp() to setup ib_mr_pool_init() for
>> RW MRs. ib_mr_pool_destroy() will be called by rdma_rw_cleanup_mrs().
>>
>> It seems the code was implemented before the rdma_rw_* layer
>> was fully established in the kernel.
>>
>> While there also add additional space for ib_drain_qp().
>>
>> This should make sure ib_post_send() will never fail
>> because the submission queue is full.
>>
>> Fixes: ddbdc861e37c ("ksmbd: smbd: introduce read/write credits for RDMA read/write")
>> Fixes: 4c564f03e23b ("smb: server: make use of common smbdirect_socket")
>> Fixes: 177368b99243 ("smb: server: make use of common smbdirect_socket_parameters")
>> Fixes: 95475d8886bd ("smb: server: make use smbdirect_socket.rw_io.credits")
>> Cc: Namjae Jeon <linkinjeon at kernel.org>
>> Cc: Steve French <smfrench at gmail.com>
>> Cc: Tom Talpey <tom at talpey.com>
>> Cc: linux-cifs at vger.kernel.org
>> Cc: samba-technical at lists.samba.org
>> Signed-off-by: Stefan Metzmacher <metze at samba.org>
> Have you run checkpatch.pl before submitting the patch ?

No, sorry, I only have it in my script that submits more than
one patch, I need to add to the send one patch script.

> WARNING: quoted string split across lines
> #337: FILE: fs/smb/server/transport_rdma.c:2046:
> + pr_err("Possible CQE overrun: max_send_wr %d, "
> +        "device reporting max_cqe %d max_qp_wr %d\n",
> 
> WARNING: quoted string split across lines
> #350: FILE: fs/smb/server/transport_rdma.c:2059:
> + pr_err("Possible CQE overrun: rdma_send_wr %d + max_send_wr %d = %d, "
> +        "device reporting max_cqe %d max_qp_wr %d\n",
> 
> WARNING: quoted string split across lines
> #362: FILE: fs/smb/server/transport_rdma.c:2071:
> + pr_err("Possible CQE overrun: max_recv_wr %d, "
> +        "device reporting max_cpe %d max_qp_wr %d\n",
> 
> total: 0 errors, 3 warnings, 305 lines checked
I'll update these.
metze