[Samba] Notifyd process and "Too many open files"

[Steve Litt]

Devon Johnson via samba said on Wed, 01 Oct 2025 15:24:22 -0700

>Hi Samba gurus,
> 
>Samba 4.19.9 on FreeBSD 14.3 with a ZFS pool for storage.
> 
>This is storage for roughly 30 million image files that our site
>displays via a number of Windows 2019 servers running IIS. There are a
>few other uses, however that is the primary usage. 

That's a lot of files. Do you have them in a directory hierarchy, and
if so, how deep and how wide?

>A few weeks back a
>software update was released for the web site running on those IIS
>servers and we noticed a marked increase in load averages after this.
>I do believe its related to these open file messages. We are seeing
>errors in our log about too many open files: Sep 30 11:13:11 XXXXX
>smbd[1111]: [2025/09/30 11:13:11.346275, 0]
>../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:11 XXXXX
>smbd[1111]: reopen_one_log: Unable to open new log file
>'/var/log/samba4/log.smbd': Too many open files Sep 30 11:13:14 XXXXX
>smbd[1111]: [2025/09/30 11:13:14.603146, 0]
>../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:14 XXXXX
>smbd[1111]: reopen_one_log: Unable to open new log file
>'/var/log/samba4/log.smbd': Too many open files The log file is being
>written to, however I don't know if process 1111 is writing to it.
>This is notifyd [XXXXXXX /var/log]# ps aux |grep 1111 XXXXXX 1111 0.0
>0.5 285832 79212 - S Thu22 2:17.17 smbd: notifyd (smbd) XXXXXX 38195
>0.0 0.0 13836 1920 0 S+ 11:16 0:00.00 grep 1111 If I run an fstat
>command to sum up the open files, I see it is hitting the limit set
>for the process which is 36000. [XXXXXXX /var/log]# fstat|cut -w
>-f1,2,3|sort |uniq -c|sort -rn | grep smbd 33603 XXXXXX smbd 1111
>Yesterday I increased this from 25600 to 33600 and immediately this
>process hit that limit.

Just for fun, make a shellscript that detects zombie processes and see
how many there are. On my computer, a poorly written Python program
(written by me unfortunately) leaves zombies that hold files open. If
you have a lot of zombies, that problem needs to be fixed whether it's
the root cause or not.

I don't have fstat on my computer so I can't comment on how similar or
different it is than lsof, but lsof redirected to a file can give you a
snapshot of open files and who has them open. This file, which I assume
will be at least 33600 lines long, might give you clues if you look for
patterns within it.

These are a couple easy general diagnostic tests. I don't have the
knowledge to suggest anything further.

SteveT

Steve Litt 

http://444domains.com