[Samba] long delays with file enumeration & listing in large data storage environment

[CaptainTrips28]

Thanks Jeremy, will give it a shot in the morning and update asap.

On Tue, Feb 7, 2023, 8:42 PM Jeremy Allison <jra at samba.org> wrote:

> On Tue, Feb 07, 2023 at 08:07:46PM -0500, CaptainTrips28 via samba wrote:
> >
> >- Samba versions 4.13.3 (rhel 8.4) and 4.15.5 (rhel 8.6) have both been
> >tried with identical outcome, all pulled from RH satellite repos)
> >
> >- We've tried with stigs both applied and unapplied, as well as
> folder/file
> >encryption both on and off (no difference in performance).
> >
> >- PBIS/ADBridge was tried with both versions 22.2.x and 22.3.x (latest)
> >
> >- Testing share access from Windows 2019 Server Datacenter (Build 17763).
> >Our user VDA sessions would also being accessing the shares from the same.
> >
> >- Production environment is enterprise scale; hundreds of users/thousands
> >of folders/millions of files
> >
> >- Share enumeration speed is the same if accessing by either mapped drive,
> >file explorer // access, or symbolically linking to the share
> >
> >Any suggestions/recommendations on how to reduce or eliminate enumeration
> >and listing times for these shares is certainly appreciated.
>
> You may be running into this (from the Samba 4.17.x release notes).
>
> NEW FEATURES/CHANGES
> ====================
>
> SMB Server performance improvements
> -----------------------------------
>
> The security improvements in recent releases
> (4.13, 4.14, 4.15, 4.16), mainly as protection against symlink races,
> caused performance regressions for meta data heavy workloads.
>
> With 4.17 the situation improved a lot again:
>
> - Pathnames given by a client are devided into dirname and basename.
>    The amount of syscalls to validate dirnames is reduced to 2 syscalls
>    (openat, close) per component. On modern Linux kernels (>= 5.6) smbd
>    makes use of the openat2() syscall with RESOLVE_NO_SYMLINKS,
>    in order to just use 2 syscalls (openat2, close) for the whole dirname.
>
> - Contended path based operations used to generate a lot of unsolicited
>    wakeup events causing thundering herd problems, which lead to masive
>    latencies for some clients. These events are now avoided in order
>    to provide stable latencies and much higher throughput of open/close
>    operations.
>
> I would suggest you try with Samba 4.17.latest to see if you
> find an improvement.
>