[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

awl1 awl1 at mnet-online.de
Mon Jul 3 10:08:53 UTC 2017


Hello one more time, Jeremy & fellow Samba experts/developers,

over the weekend, I have done some more reading about wireshark 
tooling/statistics - the answer to

https://ask.wireshark.org/questions/58970/analysing-performance-issues-with-storage-smb2

was very helpful - and am now able to provide very clear and simple 
proof of the performance regression that I am seeing between SMB/1.5 in 
Samba 3.5.16 and SMB2/3.1.1 in Samba 4.6.5, using Wireshark's 
"Statistics -> Service Response Times -> SMB(2)" tool.

I really hope that someone from the development team is now interested 
in taking over and starts looking into this with me. Please rest assured 
that I will be happy to do everything I can to support the analysis and 
testing process. Please get back to me, and I will be sending you the 
access information and password for the respective wireshark PCAPNG 
traces ZIP file.

My test scenario: Win 10 client, copying files from/to share with Total 
Commander, Thecus N4200pro NAS (Linux kernel 2.6.33) with 3 GB RAM and 
either Thecus original Samba 3.5.16 or self-compiled (using gcc-5.2) 
Samba 4.6.5, exact same smb.conf for both versions, definitely no other 
load on /access to the NAS during my testing.


A) "Write" Scenario: Win 10 client copying/storing ~ 1000 small files 
onto a Samba share on Thecus NAS

A1) smb15_write: Samba share running Thecus Samba 3.5.16 (writing 1024 
small files to share)

=================================================================================================
SMB Service Response Time Statistics - smb15_write:
Index                        Procedure  Calls  Min SRT (s)  Max SRT (s)  
Avg SRT (s)  Sum SRT (s)
-------------------------------------------------------------------------------------------------
SMB Commands
Close                                4   2791 0.000210     0.020178     
0.000831     2.320021
Negotiate Protocol                 114      1 0.001986     0.001986     
0.001986     0.001986
NT Create AndX                     162   3066 0.000650     0.019631     
0.003122     9.570728
Session Setup AndX                 115      2 0.001048     0.005596     
0.003322     0.006644
Tree Connect AndX                  117      1 0.012328     0.012328     
0.012328     0.012328
Write AndX                          47   1024 0.000473     0.350664     
0.001528     1.564348
Transaction2 Sub-Commands
FIND_FIRST2                          1   2042 0.001599     0.017518     
0.003191     6.515678
QUERY_FILE_INFO                      7   3071 0.000260     0.007440     
0.000421     1.292491
QUERY_FS_INFO                        3   1024 0.000259     0.017036     
0.000363     0.371787
QUERY_PATH_INFO                      5      2 0.000437     0.001040     
0.000739     0.001477
SET_FILE_INFO                        8   2037 0.001218     0.008207     
0.001812     3.691841
NT Transaction Sub-Commands
-------------------------------------------------------------------------------------------------
Grand Total Sum of SRT (s)    25.349329

A2) smb31_write: Samba share running self-compiled Samba 4.6.5 (writing 
1015 small files to share)

========================================================================================
SMB2 Service Response Time Statistics - smb31_write:
Index               Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT 
(s)  Sum SRT (s)
----------------------------------------------------------------------------------------
SMB2
Close                       6   3059     0.000496 0.590329     
0.004469    13.670456
Create                      5   3061     0.001683 0.023491     
0.005686    17.405265
Find                       14   1607     0.001383 0.746684     
0.193413   310.814294
GetInfo                    16     46     0.000424 0.009404     
0.001441     0.066298
Ioctl                      11      1     0.000510 0.000510     
0.000510     0.000510
Negotiate Protocol          0      1     0.043381 0.043381     
0.043381     0.043381
Session Setup               1      2     0.001535 0.037150     
0.019343     0.038685
SetInfo                    17   2033     0.000757 0.007015     
0.001111     2.259172
Tree Connect                3      2     0.001591 0.031307     
0.016449     0.032898
Tree Disconnect             4      1     0.004142 0.004142     
0.004142     0.004142
Write                       9   1015     0.000590 0.358855     
0.001798     1.825072
----------------------------------------------------------------------------------------
                                                  Grand Total Sum of SRT 
(s)   346.160173

As you can easily see, the big difference here is in the "Find" Operation:
A2a) While in Samba 4.6.5, 1607 calls to SMB2 "Find" take 310.81 
seconds, in Samba 3.5.16 2042 calls to SMB "FIND_FIRST2" take only 6.51 
seconds.
The obvious consequence is that the overall grand total sum of service 
response time for copying ~ 1000 small files goes up from 25 to a really 
annoying 346 seconds.


B) "Read" Scenario: Win 10 client copying/reading ~ 2000 small files 
from a Samba share on Thecus NAS

B1) smb15_read: Samba share running Thecus Samba 3.5.16 (reading 2040 
small files from share)

=================================================================================================
SMB Service Response Time Statistics - smb15_read:
Index                        Procedure  Calls  Min SRT (s)  Max SRT (s)  
Avg SRT (s)  Sum SRT (s)
-------------------------------------------------------------------------------------------------
SMB Commands
Close                                4   1630 0.000234     0.051417     
0.001525     2.486313
Negotiate Protocol                 114      1 0.001941     0.001941     
0.001941     0.001941
NT Create AndX                     162   2040 0.009347     0.170227     
0.012715    25.938098
Read AndX                           46   2185 0.000267     0.008728     
0.000842     1.838953
Session Setup AndX                 115      2 0.001074     0.005605     
0.003340     0.006679
Tree Connect AndX                  117      1 0.012529     0.012529     
0.012529     0.012529
Transaction2 Sub-Commands
FIND_FIRST2                          1     10 0.014211     0.817830     
0.218505     2.185050
FIND_NEXT2                           2    117 0.167130     0.868053     
0.547014    64.000643
QUERY_FILE_INFO                      7  10200 0.000267     0.003532     
0.000363     3.706283
QUERY_FS_INFO                        3   2048 0.000254     0.017356     
0.000334     0.683642
QUERY_PATH_INFO                      5     54 0.000424     0.055262     
0.007493     0.404625
NT Transaction Sub-Commands
-------------------------------------------------------------------------------------------------
Grand Total Sum of SRT (s)   101.264756

B2) smb31_read: Samba share running self-compiled Samba 4.6.5 (reading 
2033 small files from share)

========================================================================================
SMB2 Service Response Time Statistics - smb31_read:
Index               Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT 
(s)  Sum SRT (s)
----------------------------------------------------------------------------------------
SMB2
Close                       6   1754     0.000443 0.188892     
0.006119    10.732004
Create                      5   2038     0.001203 2.296105     
0.026880    54.782150
Find                       14     52     0.000435 2.540043     
1.351859    70.296650
GetInfo                    16   6091     0.000656 0.066154     
0.001229     7.485929
Negotiate Protocol          0      1     0.032793 0.032793     
0.032793     0.032793
Read                        8   2033     0.000300 0.071186     
0.000742     1.507737
Session Setup               1      2     0.001522 0.036670     
0.019096     0.038192
Tree Connect                3      1     0.030543 0.030543     
0.030543     0.030543
----------------------------------------------------------------------------------------
                                                  Grand Total Sum of SRT 
(s)   144.905998

In this case, the difference in the overall grand total sum of response 
times is "only" 101 seconds to 145 seconds, which is much less drastic, 
but still pretty much noticeable.
Unfortunately, it seems less clear here what the main culprit is - the 
difference rather seems to come from three different SMB(2) calls:
B2a) While we have 10+117 = 127 calls to "FIND_FIRST2"/"FIND_NEXT2" in 
SMB that take ~ 66 seconds, only 52 calls to SMB2 "Find" take ~ 70 seconds.
B2b) SMB2 "Close" takes ~ 10 seconds for 1754 calls, while SMB "Close" 
takes only ~ 2.5 seconds for 1630 calls.
B2c) SMB2 "Create" takes ~ 55 seconds for 2038 calls, while SMB "NT 
Create AndX" takes only ~ 26 seconds for 2040 calls.


I truly hope we will be able to improve general Samba 4.6 / SMB2 
performance for the "huge number of small files" scenario as a result of 
this exercise...

Many thanks in advance for your kind help & best regards
Andreas


Am 29.06.2017 um 11:03 schrieb awl1:
> Hello again, Jeremy and other Samba experts,
>
> I'm sorry to be such a pain in your neck(s), but I still need your 
> help in looking for help trying to find out why SMB2/3.1.1 in Samba 
> 4.6.5 performs so much worse than SMB/1.5 in Samba 3.5.16 in scenarios 
> with a huge number of small files.
>
> As requested by Jeremy, I have done wireshark "pcapng" captures of the 
> four scenarios as described in my original post below:
>
>  * smb311_write - Win 10 client storing ~ 1000 small files onto Samba
>    4.6.5/Thecus NAS
>  * smb15_write - Win 10 client storing ~ 1000 small files on Samba
>    3.5.16/Thecus NAS
>  * smb311_read - Win 10 client reading ~ 2000 small files from Samba
>    4.6.5/Thecus NAS
>  * smb15_read - Win 10 client reading ~ 2000 small files from Samba
>    3.5.16/Thecus NAS
>
> These recordings do indeed contain confidential data from my machine, 
> which is why I have so far only sent a download link to Jeremy via 
> private mail.
>
> In case others from the Samba team would also like to look into the 
> wireshark capture traces, please get back to me directly and request 
> access: I will then also send you a download link/password to the 
> capture files ZIP via private mail.
>
> I would really appreciate to be able to switch this old Thecus 
> N4200PRO NAS away from Thecus' outdated 3.5.16 version (prone to 
> "SambaCry") to a self-compiled, but secure 4.6.x version asap.
>
> Many thanks one more time for your kind help with this & best regards
> Andreas
>
> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>>> Can you get comparitive wireshark traces for the two cases ?
>>>
>>> That would help discover what the bottleneck is. 




More information about the samba mailing list