[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

[awl1]

Hello again, Jeremy and other Samba experts,

I'm sorry to be such a pain in your neck(s), but I still need your help 
in looking for help trying to find out why SMB2/3.1.1 in Samba 4.6.5 
performs so much worse than SMB/1.5 in Samba 3.6.15 in scenarios with a 
huge number of small files.

As requested by Jeremy, I have done wireshark "pcapng" captures of the 
four scenarios as described in my original post below:

  * smb311_write - Win 10 client storing ~ 1000 small files onto Samba
    4.6.5/Thecus NAS
  * smb15_write - Win 10 client storing ~ 1000 small files on Samba
    3.6.15/Thecus NAS
  * smb311_read - Win 10 client reading ~ 2000 small files from Samba
    4.6.5/Thecus NAS
  * smb15_read - Win 10 client reading ~ 2000 small files from Samba
    3.6.15/Thecus NAS

These recordings do indeed contain confidential data from my machine, 
which is why I have so far only sent a download link to Jeremy via 
private mail.

In case others from the Samba team would also like to look into the 
wireshark capture traces, please get back to me directly and request 
access: I will then also send you a download link/password to the 
capture files ZIP via private mail.

I would really appreciate to be able to switch this old Thecus N4200PRO 
NAS away from Thecus' outdated 3.6.15 version (prone to "SambaCry") to a 
self-compiled, but secure 4.6.x version asap.

Many thanks one more time for your kind help with this & best regards
Andreas


-------- Weitergeleitete Nachricht --------
Betreff: 	Re: [Samba] Huge number of small files performance regression 
from 3.5.16 to 4.6.5 with identical smb.conf
Datum: 	Tue, 20 Jun 2017 14:30:22 +0200
Von: 	awl1 <awl1 at mnet-online.de>
An: 	Jeremy Allison <jra at samba.org>, samba at lists.samba.org



Hello again, Jeremy,

first of all, I am terribly sorry for my late reply. I tried to send my
posting many times, but my mail has always been silently discarded by
the Samba mail servers due to my main mail provider (GMX - a very large
German mail provider with millions of customers) having been blacklisted
by SORBS.

For the time being, SORBS is still unwilling to delist them for unknown
reasons (which I consider a clear malpractice by SORBS, as GMX has
sophisticated spam/abuse management in place), so I had to switch to
another mail provider just in order to be able to post again on the
Samba list... :-(


Am 13.06.2017 um 19:00 schrieb awl1:
> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>> Can you get comparitive wireshark traces for the two cases ?
>>
>> That would help discover what the bottleneck is.
> I am not at all a network guy, but I hope that - maybe with a little 
> more help from your part once I have tried to do so in practice - I 
> should be able to do so...

OK, so it looks like I have been able to successfully produce Wireshark
capture files for the four scenarios... :-)

As I am almost certain that these packet captures will contain at least
some sensitive information from my environment - such as e.g. user,
share and machine names, IP addresses (possibly in the old SMB dialect
1.5 even the clear-text password for the share?) - I will only send the
link to the captures ZIP file stored in my cloud space to you via
private mail. So please keep the packet dumps confidential, and only
share them with other Samba developers after getting my explicit consent!

The ZIP file contains four Wireshark captures for the two scenarios
(write to and read from share) and the two Samba/SMB versions
(4.6.5/SMB2/dialect 3.1.1 and 3.5.16/SMB/dialect 1.5) in "pcapng" format:

  * smb311_write - Win10 client writing to Samba 4.6.5 using SMB2
    protocol (dialect 3.1.1), copying ~ 1000 files from local hard disk
    onto the share, documenting the issue with very slow throughput of
    below 10 kB/sec (especially in the range of file 300-600, most
    interestingly throughput improved again after some time)
  * smb15_write - Win10 client writing to Samba 3.5.16 using SMB
    protocol (dialect 1.5), copying ~ 1000 files from local hard disk
    onto the share, with much better throughput than in smb311_write

  * smb311_read - Win10 client reading from Samba 4.6.5 using SMB2
    protocol (dialect 3.1.1), copying ~ 2000 files from the share to
    local hard disk, with acceptable throughput, but consistently slower
    than in smb15_read
  * smb15_read - Win10 client reading from Samba 3.5.16 using SMB
    protocol (dialect 1.5), copying ~ 2000 files from the share to local
    hard disk, with consistently better throughput than in smb311_read

Fingers crossed that you will be able to determine why 4.6.5 is slower
in both scenarios, and especially so much slower when writing to the
share (smb311_write) and one more time, thanks a million for digging
into these packet dumps...

Best regards
Andreas