[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
awl1
awl1 at mnet-online.de
Sat Jul 15 23:16:08 UTC 2017
Hello Jeremy,
ouch - indeed, when using a Linux client (mount.cifs --vers=3.0),
client-side behaviour in terms of find requests is completely different:
Exact same setup, Linux Ubuntu 16.04 from exact same PC hardware that
was previously running Win10, copying the same 1031 files onto the NAS
results in the following:
===========================================================================
SMB2 Service Response Time Statistics - new_linux:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close 6 4180 0.000496 0.004940 0.001097 4.586340
Create 5 5272 0.001412 0.023465 0.004757 25.081164
Find 14 4 0.000524 0.008630 0.004097 0.016389
GetInfo 16 1084 0.000525 0.009067 0.000664 0.719405
SetInfo 17 4122 0.000436 0.006486 0.001091 4.496511
Write 9 1031 0.000813 0.064120 0.001197 1.234281
---------------------------------------------------------------------------
which means a Grand Total Sum of SRT of roundabout 36 seconds (as
opposed to roundabout 112 seconds in the latest attempt from Windows!!!).
Note that the Linux mount.cifs vers=3.0 client does only four (!!!) Find
requests for the whole scenario... So why does Windows do 2140 find
requests (1036 times "SMB2_FIND_ID_BOTH_DIRECTORY_INFO, Pattern: *" plus
1104 times SMB2_FIND_NAME_INFO Pattern: <file name>).
While even the Linux SMB2 client is still slower than the Windows SMB1
client, I tend to think that the remaining difference from 25 seconds
with SMB 1.5 in Win10 to 36 seconds with SMB2 3.0 in Linux (44%) might
be tolerable - what do you think? ;-)
I have then also compared the Win10 Total Commander client to a Win10
Explorer client and a Win10 command line prompt executing "xcopy /s",
and the results for Win10 Explorer are again interesting:
For the default Windows explorer:
===========================================================================
SMB2 Service Response Time Statistics - explorer:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close 6 2966 0.000364 0.214817 0.002725
8.082934
Create 5 3118 0.000853 0.021998 0.005782
18.026942
Find 14 1607 0.001326 0.255691 0.052544
84.438903
GetInfo 16 1053 0.000285 0.009104 0.000563
0.593040
Ioctl 11 6 0.000308 0.004001 0.002039
0.012235
Read 8 1 0.000594 0.000594 0.000594
0.000594
SetInfo 17 2063 0.000757 0.006111 0.001136
2.344231
Tree Connect 3 1 0.001563 0.001563 0.001563
0.001563
Write 9 1034 0.000303 0.305642 0.001412
1.460132
---------------------------------------------------------------------------
Grand Total Sum of SRT here: roundabout 115 seconds - pretty much the
same as from Total Commander, while the Win10 Command Prompt "xcopy /s"
is by far worst:
===========================================================================
SMB2 Service Response Time Statistics - xcopy:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close 6 8047 0.000433 0.201613 0.001869 15.040585
Create 5 8268 0.001659 0.251251 0.015277 126.313894
Find 14 3741 0.002241 0.251251 0.047638 178.213167
GetInfo 16 31 0.000461 0.002741 0.001239 0.038413
SetInfo 17 4127 0.000755 0.016472 0.003262 13.463437
Write 9 1038 0.000713 0.083296 0.001479 1.535621
---------------------------------------------------------------------------
Grand Total Sum of SRT for xcopy: roundabout 335 seconds - note that
xcopy does no less than 3741 find requests in order to copy 1038
files... Unbelieveable... :-(:-(:-(
Looking forward to your comments, especially on how to proceed as most
probably it is indeed Microsoft to blame here: Does the Samba team have
contacts inside Microsoft to point them to their SMB2 performance
regressions!? ;-)
Thanks a million one more time & best regards
Andreas
Am 15.07.2017 um 20:06 schrieb awl1 via samba:
> Hmm, in addition I just started wondering about the following:
>
> As it obviously is my Windows client that creates and sends the SMB2
> requests, maybe it is rather Microsoft to blame for choosing to
> sending the "wrong"/suboptimal/poorly performing client-side SMB2 find
> request (namely SMB2_FIND_ID_BOTH_DIRECTORY_INFO with pattern "*"
> instead of SMB2_FIND_NAME_INFO with the particular file name about to
> be written? Can Samba influence the Windows client earlier on in the
> message exchange to send find requests differently?
>
> And in case that is true that this is a shortfall of Microsoft's SMB2
> client as opposed to their SMB1 client, then how could that be fixed?
> Are there any secret registry entries to influence this, or would I
> need to raise an issue with Microsoft and document this?
>
> Really interesting... ;-):-(
>
> I will now move forward and try to execute the same client scenario
> from a Linux environment (will boot the exact same client-side PC
> hardware into Ubuntu 16.04), do another tshark recording from there
> and report back (probably tomorrow or on Monday) what this might tell
> us...
>
> Best regards,
> Andreas
>
>
> Am 15.07.2017 um 19:13 schrieb awl1 via samba:
>> In this case, the Find operation does not look for the particular
>> file name that is about to be written, but seems to try and list the
>> whole current directory's content with a pattern of "*". Note that,
>> looping through the 2000 files to be written, the length of the Find
>> Response above increases (significantly!) with every file
>> successfully copied: When copying file number 1000, the Find Response
>> sends back a list of all 999 files that have been successfully copied
>> to this directory before, and this list of 999 file names is not
>> needed for any meaningful purpose, as the goal seems to be to check
>> whether file number 1000 already exists in this list of 999 files
>> (which it of course never does!) or not.
>>
>> The last call to SMB2_FIND_ID_BOTH_DIRECTORY_INFO that is contained
>> in the traces has a response length of about 64kB (containing
>> filenames that have already been written to the target directory but
>> are not needed/helpful in any way) and interestingly does not return
>> "STATUS_NO_MORE_FILES", but "STATUS_INFO_LENGTH_MISMATCH", maybe
>> because the buffer size for the result of the pattern lookup is only
>> 64kB!?
More information about the samba
mailing list