[Samba] long delays with file enumeration & listing in large data storage environment

[Jeremy Allison]

On Tue, Feb 07, 2023 at 08:07:46PM -0500, CaptainTrips28 via samba wrote:
>
>- Samba versions 4.13.3 (rhel 8.4) and 4.15.5 (rhel 8.6) have both been
>tried with identical outcome, all pulled from RH satellite repos)
>
>- We've tried with stigs both applied and unapplied, as well as folder/file
>encryption both on and off (no difference in performance).
>
>- PBIS/ADBridge was tried with both versions 22.2.x and 22.3.x (latest)
>
>- Testing share access from Windows 2019 Server Datacenter (Build 17763).
>Our user VDA sessions would also being accessing the shares from the same.
>
>- Production environment is enterprise scale; hundreds of users/thousands
>of folders/millions of files
>
>- Share enumeration speed is the same if accessing by either mapped drive,
>file explorer // access, or symbolically linking to the share
>
>Any suggestions/recommendations on how to reduce or eliminate enumeration
>and listing times for these shares is certainly appreciated.

You may be running into this (from the Samba 4.17.x release notes).

NEW FEATURES/CHANGES
====================

SMB Server performance improvements
-----------------------------------

The security improvements in recent releases
(4.13, 4.14, 4.15, 4.16), mainly as protection against symlink races,
caused performance regressions for meta data heavy workloads.

With 4.17 the situation improved a lot again:

- Pathnames given by a client are devided into dirname and basename.
   The amount of syscalls to validate dirnames is reduced to 2 syscalls
   (openat, close) per component. On modern Linux kernels (>= 5.6) smbd
   makes use of the openat2() syscall with RESOLVE_NO_SYMLINKS,
   in order to just use 2 syscalls (openat2, close) for the whole dirname.

- Contended path based operations used to generate a lot of unsolicited
   wakeup events causing thundering herd problems, which lead to masive
   latencies for some clients. These events are now avoided in order
   to provide stable latencies and much higher throughput of open/close
   operations.

I would suggest you try with Samba 4.17.latest to see if you
find an improvement.