[Samba] Large amount of 'messaging_out_queue_recv returned Connection timed out' warnings

[Thomas de Zeeuw]

Hello readers,

I'm having a bit of an issue with the write performance of Samba. I think I’m having two issues, but one might be the cause of the other.

I found the the writes per second to be not great. I'm testing using fio (https://github.com/axboe/fio) doing a very simple sequal write of a 1G file, I've attached the fio job file. Running locally this on the server I'm getting 3936MiB/s, while doing this on macOS using SMB I get 6880KiB/s. I of course didn't expect the same numbers, but I feel the difference between the two is quite large. This let me down a little performance tuning/debugging hole which let me to the second problem.

While debugging the first issue I found a large number of 'messaging_out_queue_recv returned Connection timed out' warnings. I'm not sure what this means, other then something took too long, but while writing the 1GB file it was logged 56250 times, which seems like a lot (no idea if that is true or not). I noticed that fio reported 100% progress, but then was blocked for a good amount of time (~30 seconds) while (seemingly) not to make any further progress.

I tried searching online for the time out message, but found little to nothing (only the source code, it’s lib/messaging/messages_dgm.c line 704 btw), so perhaps this mailing list can help me.

Some information about the environment/setup:
Server:
NixOS 22.05
uname -a: Linux 5.19.16 #1-NixOS SMP PREEMPT_DYNAMIC Sat Oct 15 06:01:46 UTC 2022 x86_64 GNU/Linux
smbd --version: Version 4.15.9

Client:
macOS Montery 12.6
I'm using the builtin Finder to mount the SMB share, I don't know how to get the version for this.

I’m using a wireless network between the two, achieving a fairly consist <3ms latency. I’ve also previously achieved 300MB/s+ using the same WiFI connection, so I don’t think this a limiting factor (at least not yet).

I've attached my configuration and the fio job file I'm using. I also wanted to attache the log generated (with log level = 2), but that became to large (even compressed) so I’ve uploaded it as a GitHub Gist and is available at https://gist.githubusercontent.com/Thomasdezeeuw/91f43adc74abe4d9e7ddbea2e522bb13/raw/5ed6d3de865cbaa967eb64ac3cccd0c5bcdb8a94/samba.log.

Some notes for the log.
* The test is writing to the 'fio-seq-write' file.
* It's opened on line 246, Oct 23 19:53:16.
* It's closed on line 125254, Oct 23 19:57:42.
* 'messaging_out_queue_recv returned Connection timed out' appears 56250 times.

Any help with this would be appreciated. If I need to share more information please let me.


-------------- next part --------------



-- 
Regards,

Thomas de Zeeuw