[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
awl1
awl1 at mnet-online.de
Mon Jul 3 10:08:53 UTC 2017
Hello one more time, Jeremy & fellow Samba experts/developers,
over the weekend, I have done some more reading about wireshark
tooling/statistics - the answer to
https://ask.wireshark.org/questions/58970/analysing-performance-issues-with-storage-smb2
was very helpful - and am now able to provide very clear and simple
proof of the performance regression that I am seeing between SMB/1.5 in
Samba 3.5.16 and SMB2/3.1.1 in Samba 4.6.5, using Wireshark's
"Statistics -> Service Response Times -> SMB(2)" tool.
I really hope that someone from the development team is now interested
in taking over and starts looking into this with me. Please rest assured
that I will be happy to do everything I can to support the analysis and
testing process. Please get back to me, and I will be sending you the
access information and password for the respective wireshark PCAPNG
traces ZIP file.
My test scenario: Win 10 client, copying files from/to share with Total
Commander, Thecus N4200pro NAS (Linux kernel 2.6.33) with 3 GB RAM and
either Thecus original Samba 3.5.16 or self-compiled (using gcc-5.2)
Samba 4.6.5, exact same smb.conf for both versions, definitely no other
load on /access to the NAS during my testing.
A) "Write" Scenario: Win 10 client copying/storing ~ 1000 small files
onto a Samba share on Thecus NAS
A1) smb15_write: Samba share running Thecus Samba 3.5.16 (writing 1024
small files to share)
=================================================================================================
SMB Service Response Time Statistics - smb15_write:
Index Procedure Calls Min SRT (s) Max SRT (s)
Avg SRT (s) Sum SRT (s)
-------------------------------------------------------------------------------------------------
SMB Commands
Close 4 2791 0.000210 0.020178
0.000831 2.320021
Negotiate Protocol 114 1 0.001986 0.001986
0.001986 0.001986
NT Create AndX 162 3066 0.000650 0.019631
0.003122 9.570728
Session Setup AndX 115 2 0.001048 0.005596
0.003322 0.006644
Tree Connect AndX 117 1 0.012328 0.012328
0.012328 0.012328
Write AndX 47 1024 0.000473 0.350664
0.001528 1.564348
Transaction2 Sub-Commands
FIND_FIRST2 1 2042 0.001599 0.017518
0.003191 6.515678
QUERY_FILE_INFO 7 3071 0.000260 0.007440
0.000421 1.292491
QUERY_FS_INFO 3 1024 0.000259 0.017036
0.000363 0.371787
QUERY_PATH_INFO 5 2 0.000437 0.001040
0.000739 0.001477
SET_FILE_INFO 8 2037 0.001218 0.008207
0.001812 3.691841
NT Transaction Sub-Commands
-------------------------------------------------------------------------------------------------
Grand Total Sum of SRT (s) 25.349329
A2) smb31_write: Samba share running self-compiled Samba 4.6.5 (writing
1015 small files to share)
========================================================================================
SMB2 Service Response Time Statistics - smb31_write:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT
(s) Sum SRT (s)
----------------------------------------------------------------------------------------
SMB2
Close 6 3059 0.000496 0.590329
0.004469 13.670456
Create 5 3061 0.001683 0.023491
0.005686 17.405265
Find 14 1607 0.001383 0.746684
0.193413 310.814294
GetInfo 16 46 0.000424 0.009404
0.001441 0.066298
Ioctl 11 1 0.000510 0.000510
0.000510 0.000510
Negotiate Protocol 0 1 0.043381 0.043381
0.043381 0.043381
Session Setup 1 2 0.001535 0.037150
0.019343 0.038685
SetInfo 17 2033 0.000757 0.007015
0.001111 2.259172
Tree Connect 3 2 0.001591 0.031307
0.016449 0.032898
Tree Disconnect 4 1 0.004142 0.004142
0.004142 0.004142
Write 9 1015 0.000590 0.358855
0.001798 1.825072
----------------------------------------------------------------------------------------
Grand Total Sum of SRT
(s) 346.160173
As you can easily see, the big difference here is in the "Find" Operation:
A2a) While in Samba 4.6.5, 1607 calls to SMB2 "Find" take 310.81
seconds, in Samba 3.5.16 2042 calls to SMB "FIND_FIRST2" take only 6.51
seconds.
The obvious consequence is that the overall grand total sum of service
response time for copying ~ 1000 small files goes up from 25 to a really
annoying 346 seconds.
B) "Read" Scenario: Win 10 client copying/reading ~ 2000 small files
from a Samba share on Thecus NAS
B1) smb15_read: Samba share running Thecus Samba 3.5.16 (reading 2040
small files from share)
=================================================================================================
SMB Service Response Time Statistics - smb15_read:
Index Procedure Calls Min SRT (s) Max SRT (s)
Avg SRT (s) Sum SRT (s)
-------------------------------------------------------------------------------------------------
SMB Commands
Close 4 1630 0.000234 0.051417
0.001525 2.486313
Negotiate Protocol 114 1 0.001941 0.001941
0.001941 0.001941
NT Create AndX 162 2040 0.009347 0.170227
0.012715 25.938098
Read AndX 46 2185 0.000267 0.008728
0.000842 1.838953
Session Setup AndX 115 2 0.001074 0.005605
0.003340 0.006679
Tree Connect AndX 117 1 0.012529 0.012529
0.012529 0.012529
Transaction2 Sub-Commands
FIND_FIRST2 1 10 0.014211 0.817830
0.218505 2.185050
FIND_NEXT2 2 117 0.167130 0.868053
0.547014 64.000643
QUERY_FILE_INFO 7 10200 0.000267 0.003532
0.000363 3.706283
QUERY_FS_INFO 3 2048 0.000254 0.017356
0.000334 0.683642
QUERY_PATH_INFO 5 54 0.000424 0.055262
0.007493 0.404625
NT Transaction Sub-Commands
-------------------------------------------------------------------------------------------------
Grand Total Sum of SRT (s) 101.264756
B2) smb31_read: Samba share running self-compiled Samba 4.6.5 (reading
2033 small files from share)
========================================================================================
SMB2 Service Response Time Statistics - smb31_read:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT
(s) Sum SRT (s)
----------------------------------------------------------------------------------------
SMB2
Close 6 1754 0.000443 0.188892
0.006119 10.732004
Create 5 2038 0.001203 2.296105
0.026880 54.782150
Find 14 52 0.000435 2.540043
1.351859 70.296650
GetInfo 16 6091 0.000656 0.066154
0.001229 7.485929
Negotiate Protocol 0 1 0.032793 0.032793
0.032793 0.032793
Read 8 2033 0.000300 0.071186
0.000742 1.507737
Session Setup 1 2 0.001522 0.036670
0.019096 0.038192
Tree Connect 3 1 0.030543 0.030543
0.030543 0.030543
----------------------------------------------------------------------------------------
Grand Total Sum of SRT
(s) 144.905998
In this case, the difference in the overall grand total sum of response
times is "only" 101 seconds to 145 seconds, which is much less drastic,
but still pretty much noticeable.
Unfortunately, it seems less clear here what the main culprit is - the
difference rather seems to come from three different SMB(2) calls:
B2a) While we have 10+117 = 127 calls to "FIND_FIRST2"/"FIND_NEXT2" in
SMB that take ~ 66 seconds, only 52 calls to SMB2 "Find" take ~ 70 seconds.
B2b) SMB2 "Close" takes ~ 10 seconds for 1754 calls, while SMB "Close"
takes only ~ 2.5 seconds for 1630 calls.
B2c) SMB2 "Create" takes ~ 55 seconds for 2038 calls, while SMB "NT
Create AndX" takes only ~ 26 seconds for 2040 calls.
I truly hope we will be able to improve general Samba 4.6 / SMB2
performance for the "huge number of small files" scenario as a result of
this exercise...
Many thanks in advance for your kind help & best regards
Andreas
Am 29.06.2017 um 11:03 schrieb awl1:
> Hello again, Jeremy and other Samba experts,
>
> I'm sorry to be such a pain in your neck(s), but I still need your
> help in looking for help trying to find out why SMB2/3.1.1 in Samba
> 4.6.5 performs so much worse than SMB/1.5 in Samba 3.5.16 in scenarios
> with a huge number of small files.
>
> As requested by Jeremy, I have done wireshark "pcapng" captures of the
> four scenarios as described in my original post below:
>
> * smb311_write - Win 10 client storing ~ 1000 small files onto Samba
> 4.6.5/Thecus NAS
> * smb15_write - Win 10 client storing ~ 1000 small files on Samba
> 3.5.16/Thecus NAS
> * smb311_read - Win 10 client reading ~ 2000 small files from Samba
> 4.6.5/Thecus NAS
> * smb15_read - Win 10 client reading ~ 2000 small files from Samba
> 3.5.16/Thecus NAS
>
> These recordings do indeed contain confidential data from my machine,
> which is why I have so far only sent a download link to Jeremy via
> private mail.
>
> In case others from the Samba team would also like to look into the
> wireshark capture traces, please get back to me directly and request
> access: I will then also send you a download link/password to the
> capture files ZIP via private mail.
>
> I would really appreciate to be able to switch this old Thecus
> N4200PRO NAS away from Thecus' outdated 3.5.16 version (prone to
> "SambaCry") to a self-compiled, but secure 4.6.x version asap.
>
> Many thanks one more time for your kind help with this & best regards
> Andreas
>
> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>>> Can you get comparitive wireshark traces for the two cases ?
>>>
>>> That would help discover what the bottleneck is.
More information about the samba
mailing list