Samba with multichannel and io_uring
Stefan Metzmacher
metze at samba.org
Fri Oct 16 11:49:41 UTC 2020
Hi Jens,
> Thanks for sending this, very interesting! As per this email, I took a
> look at the NUMA bindings. If you can, please try this one-liner below.
> I'd be interested to know if that removes the fluctuations you're seeing
> due to bad locality.
>
> Looks like kthread_create_on_node() doesn't actually do anything (at
> least in terms of binding).
>
>
> diff --git a/fs/io-wq.c b/fs/io-wq.c
> index 74b84e8562fb..7bebb198b3df 100644
> --- a/fs/io-wq.c
> +++ b/fs/io-wq.c
> @@ -676,6 +676,7 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index)
> kfree(worker);
> return false;
> }
> + kthread_bind_mask(worker->task, cpumask_of_node(wqe->node));
>
> raw_spin_lock_irq(&wqe->lock);
> hlist_nulls_add_head_rcu(&worker->nulls_node, &wqe->free_list);
>
I no longer have access to that system, but I guess it will help, thanks!
With this:
worker->task = kthread_create_on_node(io_wqe_worker, worker, wqe->node,
"io_wqe_worker-%d/%d", index, wqe->node);
I see only "io_wqe_worker-0" and "io_wqe_worker-1" in top, without '/0' or '/1' at the end,
this is because set_task_comm() truncates to 15 characters.
As developer I think 'io_wqe' is really confusing, just from reading I thought it
means "work queue entry", but it's a per numa node worker pool container...
'struct io_wq_node *wqn' would be easier to understand for me...
Would it make sense to give each io_wq a unique identifier and use names like this:
(fdinfo of the io_uring fd could also include the io_wq id)
"io_wq-%u-%u%c", wq->id, wqn->node, index == IO_WQ_ACCT_BOUND ? 'B' : 'U')
io_wq-500-M
io_wq-500-0B
io_wq-500-0B
io_wq-500-1B
io_wq-500-0U
io_wq-200-M
io_wq-200-0B
io_wq-200-0B
io_wq-200-1B
io_wq-200-0U
I'm not sure how this interacts with workers moving between bound and unbound
and maybe a worker id might also be useful (or we rely on their pid)
I just found that proc_task_name() handles PF_WQ_WORKER special
and cat /proc/$pid/comm can expose something like:
kworker/u17:2-btrfs-worker-high
ps and top still truncate, but that can be fixed.
Some workqueues also expose their details under /sys/bus/workqueue/.
I guess there's a lot of potential to optimize the details exposed to the admins
and (userspace) developers.
metze
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20201016/c1de51b9/signature.sig>
More information about the samba-technical
mailing list