[Samba] Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Tue Jun 13 10:09:41 UTC 2017

Hello Samba experts,

I have just successfully replaced the old default Thecus Samba version 
3.5.16 on my Thecus NAS (32-bit Intel Atom, 3 GB RAM, i686 Thecus kernel 
2.6.33) by a current Samba 4.6.5 build that I have cross-compiled myself 
from scratch.

Note that so far, I am using the unchanged (i.e. exactly identical) old 
3.5.16 smb.conf file for 4.6.5.

Everything seems to work fine in terms of functionality. From Windows 
PowerShell, I have been also able to verify with "Get-SmbConnection" 
that I am now using SMB3 ("dialect 3.1.1"), and I have noticed decent 
performance gains for copying large files as expected:

  * single large file copy, Win10 NTFS client to Thecus NAS ("write"):
    before: Samba 3.5.16/SMB 1.5: ~18 MB per second
    after: Samba 4.6.5/SMB 3.1.1: ~23 MB per second

  * single large file copy, Thecus NAS to Win10 NTFS client ("read"):
    before: Samba 3.5.16/SMB 1.5: ~60 MB per second
    after: Samba 4.6.5/SMB 3.1.1: ~85 MB per second

But unfortunately, I have also run into a severe performance degradation 
for copying a huge number of small files between a Win10 client and the 
NAS in both directions. My test scenario here is copying a single 
directory containing 5400 files of between less than 1kB and max 8kB in 
size in both directions (Win10 -> Thecus and the other way round).

To the best of my knowledge, performance-related parameters in smb.conf 
had already been tuned for 3.5.16 with good results - IIRC, it was 
especially crucial for write performance to switch from

case sensitive = auto

to

case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes

in order to stop smbd from repeatedly looping through those ~5000 trying 
to check whether any new file name was unique in a case insensitive way...

But with the exact same smb.conf as attached (global section), I now 
observe

  * many small files copy, Win10 NTFS client to Thecus NAS ("write"):
    before: Samba 3.5.16/SMB 1.5: ~120 kB per second
    after: Samba 4.6.5/SMB 3.1.1: ~4 (!!!) kB per second (this is
    completely inacceptable: on average, only 1-2 files per second!!!)

  * many small files copy, Thecus NAS to Win10 NTFS client ("read"):
    before: Samba 3.5.16/SMB 1.5: ~400 kB per second
    after: Samba 4.6.5/SMB 3.1.1: ~130 kB per second (even though still
    acceptable, slower by a factor of 3!!!)

Note that also, in both the "write" and the "read" scenario, CPU load 
for the smbd process on the NAS is much higher for the 4.6.5 than the 
3.5.16 scenario - and in the "write" scenario, CPU load even increases 
over time (i.e. with the number of files that are already copied) and 
continuously uses up to about 80% of one of my four (hyper-threaded) 
Atom cores (1.8 GHz)...

Unfortunately, this performance issue makes Samba 4.6.5 pretty much a 
pain to use for any "development" scenarios with huge numbers of small 
files - so for now, I had to revert back my NAS to 3.5.16 to be usable - 
even though I definitely want to upgrade asap in order to remove the 
risk to be affected by the "SambaCry" issue (CVE-2017-7494)... :-(

My questions now are:

  * Am I still doing something wrong in terms of configuration, i.e.
    using inappropriate settings in smb.conf?
  * Am I perhaps hitting an already known performance issue, and if so,
    do you have any plans/timeline for fixing it?
  * And if this issue is indeed new to you, then how can I help the
    Samba team in tracking down the root cause of this and hopefully
    fixing the issue, i.e. can I enable my build to support code
    profiling to see where all that CPU power and time is lost?

Many thanks in advance for looking into this and advising how to 
proceed! :-)

Best regards
Andreas

-------------- next part --------------
[global]
server string = %h
max open files = 100000
deadtime = 15
dead time = 15
hide unreadable = yes
load printers = no
log file = /var/log/samba.%m
max log size = 50
strict locking = no
lock directory = /var/samba
encrypt passwords = yes
case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes
passdb backend = tdbsam
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
aio read size = 1
aio write size = 1
write cache size = 2097152
read raw = yes
write raw = yes
min receivefile size = 0
use sendfile = yes
large readwrite = yes
max xmit = 32768
getwd cache = true
map untrusted to domain = yes
os level = 1
local master = yes
unix extensions = yes
domain master = no
preferred master = no
dns proxy = no
dos charset = cp1252
unix charset = utf8
client ldap sasl wrapping = seal
allow trusted domains = yes
idmap uid = 20000-60000000
idmap gid = 20000-60000000
winbind separator = +
winbind nested groups = yes
winbind enum users = yes
winbind enum groups = yes
create mask = 0644
winbind use default domain = yes
map acl inherit = yes
nt acl support = yes
#map system = yes
bind interfaces only = yes
interfaces = lo,bond*
guest account = nobody
map to guest = Bad User
guest only = yes
follow symlinks = no
block size = 262144
dfree cache time = 5
large readwrite = yes
getwd cache = yes
oplocks = yes
kernel oplocks = yes
veto files = /folder.db/.AppleDouble/.AppleDB/.bin/.AppleDesktop/Network Trash Folder/:2eDS_Store/.DS_Store/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary Items/.AppleDBcnid.lock/.VolumeIcon.icns/.Temporary Items/.Parent/.HSicon/._*/:*/
veto oplock files = /J0*.WMF/*_.GIF/J0*.JPG/*_.WMF/
workgroup = WORKGROUP
password server = *
security = user
auth methods = guest sam_ignoredomain
realm =
idmap backend = rid:WORKGROUP=20000-60000000
wins server = 192.168.2.110
client ntlmv2 auth = no
server signing = disabled
delete veto files = yes