[Samba] Improving 30-40MB/sec Sequential Reads

[Brian Moses]

I'm seeing some really poor performance out of my FreeNAS (ver 9.10)
machine, running Samba "4.3.6-GIT-UNKNOWN".  I'm using IOMeter to benchmark
sequential reads, and getting around 35-40 MB/sec, which seems unusual.

Mostly,  I'm hoping someone can point me in the direction of a decent
tuning guide for a SOHO machine running Samba, but if you're inclined, I'll
delve into what I've tried/tinkered with below.

The hardware on My FreeNAS machine is an Avoton C2550, 32 MB ECC RAM, 7x2TB
HDDs in raidz2 config and using a pair of SSDs as cache devices for ZFS.

The motherboard has an onboard Intel Gigabit NIC and I've recently added a
Chelsio T320 10Gbe card as well.

I've benchmarked the local disk read speed and I get around 565MB/sec
reading a file that far-exceeds the NAS' RAM.

I've also used iperf to confirm the network interfaces aren't the
bottleneck. A single instance/thread of iperf pushes 921Mb/sec over the
1Gbe line and 5.07 Gb/sec over the 10Gbe.

There's not much else going on my NAS at the time I'm seeing these slow
transfers.  A peek at top shows the overal CPU utilization staying under
20%,  a peek at the individual cores during the slow transfers doesn't show
any one core ever exceeding around 25% utilization.

Everything so far seems to suggest that there's plenty of disk speed,
available bandwidth (on either interface) and CPU cycles.   Because I'm
seeing the same slow transfers on both interfaces,  I'm suspicious that
there's something about my Samba configuration which is suboptimal and I
need some direction/help in making those changes.

The contents of /usr/local./etc/smb4.conf have been pasted at the end of
the email.

I completely understand I'm unlikely to see the 10Gbe link saturated by a
single session and I don't have extremely lofty expectations.  But do have
250MB/sec picked out as a hopeful goal.

Does anybody have any suggestions on how I can increase performance with my
setup?

smb4.conf
---------------

[global]
    server max protocol = SMB3
    encrypt passwords = yes
    dns proxy = no
    strict locking = no
    oplocks = yes
    deadtime = 15
    max log size = 51200
    max open files = 942104
    logging = file
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    getwd cache = yes
    guest account = nobody
    map to guest = Bad User
    obey pam restrictions = yes
    directory name cache size = 0
    kernel change notify = no
    panic action = /usr/local/libexec/samba/samba-backtrace
    nsupdate command = /usr/local/bin/samba-nsupdate -g
    server string = DrTeeth - FreeNAS Server
    ea support = yes
    store dos attributes = yes
    lm announce = yes
    time server = yes
    acl allow execute always = true
    dos filemode = yes
    multicast dns register = yes
    domain logons = no
    local master = yes
    idmap config *: backend = tdb
    idmap config *: range = 90000001-100000000
    server role = standalone
    netbios name = DRTEETH
    workgroup = LAN
    security = user
    pid directory = /var/run/samba
    create mask = 0666
    directory mask = 0777
    client ntlmv2 auth = yes
    dos charset = CP437
    unix charset = UTF-8
    log level = 1


[homes]
    valid users = %U
    path = /mnt/volume1/homes/%U
    comment = Home Directories
    printable = no
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    vfs objects = zfs_space zfsacl aio_pthread streams_xattr
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare


[Archive]
    path = /mnt/volume1/Archive
    printable = no
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    vfs objects = zfs_space zfsacl aio_pthread streams_xattr
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare


[Backups]
    path = /mnt/volume1/Backups
    comment = Backups Share
    printable = no
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    vfs objects = zfs_space zfsacl aio_pthread streams_xattr
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare