[Samba] long delays with file enumeration & listing in large data storage environment

[CaptainTrips28]

After joining a Samba server to domain using either Winbind or using
ADBridge (PBIS), enumeration time of listing files from windows file
explorer for that share is roughly (9) seconds for a folder containing 30k
files consisting of both large (upto 15GB in size) and small size files (as
low as KB's). Intitially, without setting logging to "0" and disabling smb
max server protocol in smb.conf (which is now defaulting to smb3_11),
listing of these files would take up to 45 seconds. So we have already
drastically cut down this time with smb.conf option corrections.

As ~9 seconds may seem trivial to enumerate 30k files, this is only in a
test scenario with randomly generated junk files. In our production
scenario, we have folders frequently containing upwards of 100k and 250k
files and requirements for folders that may contain upto potentially 1
million files of various sizes and filenames consisting of upper and
lowercase characters. Assuming the average of around 200k files in the prod
scenario, the enumeration time would be around 1min10secs to list all
files. This enumeration delay is causing issues with our end users.

Initially, prior to joining the samba server to domain, using just a local
smb-enabled account, testing access time from windows file explorer to this
share of 30k files for enumeration was instaneous (a second or less).
However, in production, windows AD users are accessing the shares which
have AD owners and groups assigned to the shares folders and permissions
based on their level of access.

We have tried numerous config options and variations of options
enabled/disabled, trying local storage mountpoints on the samba server vs.
remote mountpoints via NFS and SSHFS. We've tried numerous hardware types:
a physical HPE Gen11 DL380 system acting the samba server, a virtual samba
server within vsphere, moving hardware around on network appliances to
reduce hops and improve throughput. Ipv6 has been disabled, all ports are
open as required, no antivirus scan interference. From the windows side,
smb client settings have been tested at various options, confirmed netbios
is disabled, etc. We've also tried limiting the samba server to one DC and
ignoring all irrelevant domains. No matter the test scenario, once domain
joined, the absolute lowest enumeration time we can achieve is the (9)
seconds until file listing is complete and that is pretty universal outcome
in each of the above mentioned testing scenarios.

- Tried both RHEL 8.4 and 8.6 (FIPS disabled, SELinux permissive and
firewalld off)

- Samba versions 4.13.3 (rhel 8.4) and 4.15.5 (rhel 8.6) have both been
tried with identical outcome, all pulled from RH satellite repos)

- We've tried with stigs both applied and unapplied, as well as folder/file
encryption both on and off (no difference in performance).

- PBIS/ADBridge was tried with both versions 22.2.x and 22.3.x (latest)

- Testing share access from Windows 2019 Server Datacenter (Build 17763).
Our user VDA sessions would also being accessing the shares from the same.

- Production environment is enterprise scale; hundreds of users/thousands
of folders/millions of files

- Share enumeration speed is the same if accessing by either mapped drive,
file explorer // access, or symbolically linking to the share

Any suggestions/recommendations on how to reduce or eliminate enumeration
and listing times for these shares is certainly appreciated.




-------------------------------

CURRENT SMB.CONF   (I've had to replace the domain/share/host details for
privacy):

-------------------------------

[global]

        security = ADS

        workgroup = DOMAIN

        realm = DOMAIN

        machine password timeout = 0

        idmap config * : backend = tdb

        idmap config * : range = 3000-7999

        idmap config DOMAIN.COM:backend = ad

        idmap config DOMAIN.COM:schema_mode = rfc2307

        idmap config DOMAIN.COM:range = 1617000000-1617999999

        idmap config DOMAIN.COM:unix_nss_info = yes

        map acl inherit = yes

        store dos attributes = yes

        log level = 0

        dns proxy = yes

        hostname lookups = yes

        kerberos method = system keytab

        log file = /var/log/samba/log.%m

        smb encrypt = yes

        server signing = auto

        client signing = auto

        vfs objects = acl_xattr

        nt acl support = yes

        netbios name = testhost01





[OrgName-OrgGroup-OrgSubgroup-TEST]

        path = /mountpoint/Development/OrgName/OrgGroup/OrgSubgroup/TEST

        writable = yes

        browsable= yes

        read only = no

        create mask = 0770

        force create mode = 0770

        directory mask = 0775

        hide unreadable = yes

#        force group = OrgGroupTest at domain.com

#        valid users = +domain.com\"OrgGroupTest"


---------------------------------------------------------------------------

OTHER SMB.CONF OPTIONS TRIED IN VARIATION:

---------------------------------------------------------------------------

*(not currently applied as these options either made no difference or
decreased performance/enumeration):



(global options we've tried various combinations of. We are aware these
options are not recommended by modern kernel standards, but tried them
regardless)

#        username map = /etc/samba/user.map

#        server multi channel support = yes

#        aio read size = 16384

#        aio write size = 16384

#        aio max threads = 100

#        allocation roundup size = 1048576

#        interfaces = "10.100.99.103;speed=10000000000,capability=RSS"

#        winbind max domain connections = 10

#        winbind expand groups = 1

#        socket options = SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY

#        min receivefile size = 16384

#        use sendfile = true

#        aio read size = 16384

#        aio write size = 16384

#        aio write behind = true


(share options we've tried with no performance improvement even when
testing lowercase files only)

#        case sensitive = true

#        default case = lower

#        preserve case = no

#        short preserve case = no