[Samba] Notifyd process and "Too many open files"

[Devon Johnson]

Hi Samba gurus,
 
Samba 4.19.9 on FreeBSD 14.3 with a ZFS pool for storage.
 
This is storage for roughly 30 million image files that our site displays via a number of Windows 2019 servers running IIS. There are a few other uses, however that is the primary usage.
 
A few weeks back a software update was released for the web site running on those IIS servers and we noticed a marked increase in load averages after this. I do believe its related to these open file messages. 
 
We are seeing errors in our log about too many open files:
Sep 30 11:13:11 XXXXX smbd[1111]: [2025/09/30 11:13:11.346275, 0] ../../lib/util/debug.c:1274(reopen_one_log)
Sep 30 11:13:11 XXXXX smbd[1111]: reopen_one_log: Unable to open new log file '/var/log/samba4/log.smbd': Too many open files
Sep 30 11:13:14 XXXXX smbd[1111]: [2025/09/30 11:13:14.603146, 0] ../../lib/util/debug.c:1274(reopen_one_log)
Sep 30 11:13:14 XXXXX smbd[1111]: reopen_one_log: Unable to open new log file '/var/log/samba4/log.smbd': Too many open files
 
The log file is being written to, however I don't know if process 1111 is writing to it. This is notifyd
[XXXXXXX /var/log]# ps aux |grep 1111
XXXXXX 1111 0.0 0.5 285832 79212 - S Thu22 2:17.17 smbd: notifyd (smbd)
XXXXXX 38195 0.0 0.0 13836 1920 0 S+ 11:16 0:00.00 grep 1111
 
If I run an fstat command to sum up the open files, I see it is hitting the limit set for the process which is 36000.
 
[XXXXXXX /var/log]# fstat|cut -w -f1,2,3|sort |uniq -c|sort -rn | grep smbd 
33603 XXXXXX smbd 1111
 
Yesterday I increased this from 25600 to 33600 and immediately this process hit that limit.

The smbd processes handing the file shares usually stay in the range of 2000-3000 open files per process.

I am trying to figure out what is causing this. The FreeBSD server's load averages are higher than they historically have been. I know the code is C# that the IIS servers run and I do not have access to it. I understand that a new filestream operation was introduced that allowed the browser to pick a 'medium' sized file and these are properly disposed of in the code as I am being told.

I am looking for any suggestions or troubleshooting steps to help identify what is causing these file locks, or what I can do about it besides increasing the limit of open files… eventually that will just overload the system as we discovered when we did try to increase it. The system became unusable at roughly 520,000 open files which happened minutes after smbd was started.

Below is the sanitized smb4.conf * 
[XXXXXXX /var/log]# cat /XXX/smb4.conf
realm = DOMAIN.DOMAIN
workgroup = WORKGROUP
security = ADS
guest account = User
winbind separator = +
winbind enum groups = Yes
winbind enum users = Yes
winbind nss info = rfc2307
winbind offline logon = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY
idmap config * : range = 1000-200000
idmap config * : backend = tdb
template shell = /xxx/xxx/xxx/xxx
template homedir = /home/%D/%U
ntlm auth = yes
passdb backend samba_dsdb
server role = member server
username map script = /bin/echo
log level = 6
follow symlinks = yes
unix extensions = no
nt acl support = yes
inherit acls = no
map acl inherit = yes
deadtime = 10

[share]
path = /path/on/zfs
guest ok = yes
writeable = yes
browsable = yes
inherit permissions = yes
inherit owner = yes
locking = yes
oplocks = yes
kernel oplocks = no
level2 oplocks = true
strict locking = auto

Thank you in advance!