[Samba] winbindd: Exceeding 200 client connections, no idle connection found

[Tristan Evans]

Hello! This is my first email to the Samba mailing list :)

We have been having an issue on our servers where the winbind service
begins to utilize 100%~ CPU and logs the following error:

*winbindd: Exceeding 200 client connections, no idle connection found*

We have been running Samba *3.6.3-0.52.5*, however when originally
investigating the problem, it was found that *3.6.3-0.58.1* included a
patch for the following:


*Prune idle or hung connections older than "winbind request
timeout"(bso#3204, bnc#872912)*

*BSO #3204* covers what appears to be the exact problem we're having. The
Bugzilla report is here: https://bugzilla.samba.org/show_bug.cgi?id=3204.

So we've been deploying this patch slowly to all the servers that are
having this problem (around 1500 physical servers running the same image).
I've been keeping a close eye on my email, because the machines will send
an email every time this error occurs. We have deployed it to about 1/3 of
the machines, and today I just got a email from a system that already
installed that patch.

The patch is being deployed via a script that stops the Samba services
(smb, nmb, winbind), installs the RPMs and then starts the services.

*The winbindd service stopped: 2:00:04 AM*

*The patch installed: 2/18 02:00:08 AM*
*The winbindd service started: 2/18 02:00:49 AM*

The error was reported in the logs: 2/18 12:56 PM

Now it could be the service wasn't restarted fully, however it logged as if
it was:

*[2016/02/18 02:00:04.379281,  0]
winbindd/winbindd.c:212(winbindd_sig_term_handler)*
*  Got sig[15] terminate (is_parent=1)*
*[2016/02/18 02:00:49.910364,  0]
winbindd/winbindd_cache.c:3178(initialize_winbindd_cache)*
*  initialize_winbindd_cache: clearing cache and re-creating with version
number 2*

Unfortunately I don't have the system in it's bad state right now (we have
a support team who connects and restarts the service when these emails roll
in, so that our customers do not experience problems).

The next time this happens, I will try to preserve it's state for a bit,
and get stuff like the winbind processes FDs. However, is there anything
else that I should be looking for that would assist with an issue like this?

Thanks.