[Samba] Troubleshooting poor (small) random read performance -- serverid.tdb?

[Ray Van Dolson]

Hi everyone;

Have a Windows 2012 based workload that generates many (20K+ PPS) small
reads (Wireshark tells me the individual packets are typically less
than 200 bytes and are Read AndX Request) from eight or so hosts to a
Samba server running on RHEL 6.5.  Samba version is the latest Red Hat
provided 3.6.9 version.

Things are going quite a bit slower than expected (as compared to the
same workload pointed at a Windows 2012 server).

iostat shows not much disk IO activity or iowait going on, but smbd
processes are all maxed out CPU wise.  An strace on them shows *lots*
of activity (almost exclusively) to the serverid.tdb file:

fcntl(16, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=636, len=1}) = 0
fcntl(16, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=636, len=1}) = 0

File handle 16 corresponds with the serverid.tdb file for the PID in
question.

Have been searching around trying to find out what exactly it is that
the serverid.tdb file is used for but hvaen't found a great
explanation.  Am wondering if it has something to do with my use of
security ads and winbind?

I'm hoping that if I can find a way to eliminate or optimize all of
this activity I can get better performance out of Samba and avoid
needing to shift to Windows.

Possibly Samba 4.x would work better?  Haven't yet tried.

Also am unsure if this random read workload with very small
transactions will work well out of the box with Samba.

My config:

[global]
    workgroup = WORKGROUP
    password server = server, *
    realm = realm.com
    security = ads
    idmap uid = 10000-19999
    idmap gid = 10000-19999
    idmap config WORKGROUP:backend = rid
    idmap config WORKGROUP:range = 10000000-19999999
    template shell = /bin/bash
    winbind enum users = no
    winbind enum groups = no
    winbind separator = +
    winbind use default domain = yes
    winbind normalize names = yes
    template homedir = /home/%D/%U
    template shell = /bin/bash
    server string = Samba Server Version %v
    log file = /var/log/samba/log.%m
    log level= 1
    max log size = 50
    passdb backend = tdbsam
    load printers = no
    cups options = raw

    socket options = TCP_NODELAY SO_KEEPALIVE SO_RCVBUF=131072 SO_SNDBUF=131072 IPTOS_LOWDELAY
    # These are defaults, but explicitly set here.
    read raw = yes
    write raw = yes
    strict allocate = no
    # Unsure is this is valid?
    min receivefile size = 16384


Thanks,
Ray