[Samba] Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
awl1
awl1 at mnet-online.de
Fri Jul 14 14:52:40 UTC 2017
Hello again, Jeremy, hello again, Samba experts/developers,
as "all good things come in threes" and "third time is a charm",
following kind advice from Björn Jacke, I do indeed try again to arouse
your interest on this list one more time, giving an even shorter summary
of the issue and having tested with a number of older Samba versions
between 3.5.x and 4.6.x to exactly pinpoint when the issue started.
As I am 99.99% confident that this is not a configuration issue on my
side, I would really appreciate if somebody the Samba team would be
interested in tracking down why - for the specific scenario with a huge
number of small files - performance is (so) much worse with Samba
4.x/SMB2 than it used to be with Samba 3.x/SMB1.
(Please note that, for a small number of larger or even huge files, as
expected, I can also confirm from my observations that Samba 4.x/SMB2 is
typically faster than Samba 3.x/SMB1, sometimes even considerably, so
the issue is NOT with Samba 4.x/SMB2 in general, but seems to be caused
to the specific scenario of a huge number of small files.)
Summary:
* Win10 client using TotalCommander 9.0a to copy files
* Copying files from/to a Samba share running on my Home Office
Thecus NAS
* Thecus N4200pro NAS (Intel(R) Atom(TM) CPU D525, 2 cores/4 HT
threads @ 1.80GHz, Linux kernel 2.6.33, 3 GB RAM) and either
Thecus original Samba 3.5.16 or several self-compiled
(using gcc-5.2) Samba versions:
- Samba 4.6.5, SMB2 dialect 3.1.1
- Samba 4.2.14, SMB2 dialect 3.0
- Samba 4.0.26, SMB2 dialect 3.0
- Samba 3.6.25, SMB2 dialect 2.0.2 (single line
"min protocol = SMB2" added to smb.conf)
- Samba 3.6.25, SMB1 dialect 1.5
- Thecus original Samba 3.5.16, SMB1 dialect 1.5
* Exact same hardware, network, complete software stack for all
cases (except varying Samba version on Thecus NAS)
* Exact same smb.conf for both versions (see attached)
* Definitely no other load on/access to the NAS during my testing
* Recorded Wireshark captures in pcapng format for both Write/Read
scenarios in all above Samba versions
* Looking at Grand Total Sum of Wireshark "Service Response Time
Statistics" (SRT) in seconds for all captures to compare performance
below
A) "Write" Scenario:
Write ~ 1000 Small Files (between <1kB and ~ 20kB) to Samba share on
Thecus NAS, copying from a directory of ~ 5000 files stored on Win10
local NTFS
Samba version SMB/SMB2 dialect Total SRT (sec)
3.5.16 1.5 25
3.6.25 1.5 21
3.6.25 2.0.2 341 (!!!)
4.0.26 3.0 387 (!!!)
4.2.14 3.0 355 (!!!)
4.6.5 3.1.1 346 (!!!)
B) "Read" Scenario:
Read ~ 2000 Small Files (between <1kB and ~ 20kB) from a directory of ~
5000 files from Samba share on Thecus NAS, copy to local NTFS on Win10
Samba version SMB/SMB2 dialect Total SRT (sec)
3.5.16 1.5 101
3.6.25 1.5 100
3.6.25 2.0.2 139 (!)
4.0.26 3.0 152 (!)
4.2.14 3.0 140 (!)
4.6.5 3.1.1 144 (!)
(Note that the read scenario spends most of the time in all cases, i.e.
eben in 3.x/SMB 1.5, determining the whole number of ~5000 files in this
directory, before Total Commander even starts copying the ~ 2000 files.)
Summary of findings:
* For both Write and Read scenario and a huge number of small files,
performance with SMB2/dialect 2.0/3.0/3.1.1 in all Samba versions
>= 3.6 up to most recent 4.6 is (much) worse than SMB performance
with SMB/dialect 1.5 in Samba 3.6 and before.
* While in the Read scenario, performance is "only" worse by a factor
of 40% (which might possibly at least partly be explained by
additional complexity in SMB2), for the Write scenario, performance
is about fourteen times (1400%) worse, a finding which definitely
cannot be explained to be "working as designed".
* While SMB/1.5 performance is still fine in the latest 3.6.25, all
SMB2-capable releases of Samba from the very first SMB2/2.x
implementation in Samba 3.6 onwards seem to be affected by the
performance regression.
I have attached the (anonymized) smb.conf (global section and particular
share definition) as well as an Excel sheet and PDF with the detailed
Wireshark Service Response Time Statistics for Write and Read scenario.
Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
> Can you get comparitive wireshark traces for the two cases ?
> That would help discover what the bottleneck is.
As requested by Jeremy, the Wireshark "pcapng" packet traces/recordings
are available for all Samba versions mentioned above in both Read and
Write scenario. Unfortunately, these recordings do indeed contain
confidential data both from my machine and the share, so please get back
to me directly and request access: I will then send you a download
link/password to the capture files ZIP via private mail.
I hereby promise that I will do everything I can in order to support
your analysis, including running follow-up tests on my
platform/scenario, digging deeper into packet traces or even do source
code investigations based on your instructions.
I truly hope we will be able to improve general Samba 4.x / SMB2
performance for the "huge number of small files" scenario as a result of
this exercise...
Many thanks one more time for your kind help with this!
Best regards,
Andreas
-------------- next part --------------
[global]
server string = %h
max open files = 100000
deadtime = 15
dead time = 15
hide unreadable = yes
load printers = no
log file = /var/log/samba.%m
max log size = 50
strict locking = no
lock directory = /var/samba
encrypt passwords = yes
case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes
passdb backend = tdbsam
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
aio read size = 1
aio write size = 1
write cache size = 2097152
read raw = yes
write raw = yes
min receivefile size = 0
use sendfile = yes
large readwrite = yes
max xmit = 32768
getwd cache = true
map untrusted to domain = yes
os level = 1
local master = yes
unix extensions = yes
domain master = no
preferred master = no
dns proxy = no
dos charset = CP850
unix charset = utf8
client ldap sasl wrapping = seal
allow trusted domains = yes
idmap uid = 20000-60000000
idmap gid = 20000-60000000
winbind separator = +
winbind nested groups = yes
winbind enum users = yes
winbind enum groups = yes
create mask = 0644
winbind use default domain = yes
map acl inherit = yes
nt acl support = yes
#map system = yes
bind interfaces only = yes
interfaces = lo,bond*
guest account = nobody
map to guest = Bad User
guest only = yes
follow symlinks = no
block size = 262144
dfree cache time = 5
large readwrite = yes
getwd cache = yes
oplocks = yes
kernel oplocks = yes
veto files = /folder.db/.AppleDouble/.AppleDB/.bin/.AppleDesktop/Network Trash Folder/:2eDS_Store/.DS_Store/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary Items/.AppleDBcnid.lock/.VolumeIcon.icns/.Temporary Items/.Parent/.HSicon/._*/:*/
veto oplock files = /J0*.WMF/*_.GIF/J0*.JPG/*_.WMF/
workgroup = WORKGROUP
password server = *
security = user
auth methods = guest sam_ignoredomain
realm =
idmap backend = rid:WORKGROUP=20000-60000000
wins server = 192.168.1.1
client ntlmv2 auth = no
server signing = disabled
smb encrypt = disabled
delete veto files = yes
[Work]
comment = Work
browseable = yes
guest only = no
path = /raid0/data/Work
map acl inherit = yes
inherit acls = yes
read only = no
create mask = 0777
force create mode = 0000
inherit permissions = Yes
map archive = yes
map hidden = no
store dos attributes = no
valid users = @smbadmin, at smbwork,user1
invalid users = user2,upload
read list = backup
write list = @smbadmin, at smbwork,user1
More information about the samba
mailing list