Winbindd limited by select

[Michael Steffens]

Hi Ken,

Ken Cross wrote:
> I've run into a problem with winbindd in both 2.2.x and 3.0 where it
> just locks up after a while on large, busy networks.
> 
> We finally tracked down the problem to the fact that the C library
> "select" function is limited by default to 256 file descriptors in
> NetBSD (1024 in FreeBSD, 2048 in Linux).  So once 256 (or whatever) smbd
> processes connected to winbindd, it broke pretty badly and was very hard
> to kill.
> 
> This is set at compile-time, not run-time.  This line:
> 
>  #define FD_SETSIZE 2048  /* Max # of winbindd connections */
> 
> must occur before the first invocation of <sys/types>.
> 
> This could be a build option, but it might be much simpler to hard-code
> it in local.h, which is what I did to fix it.
> 
> Can somebody check the implications of this on Solaris, HPUX, etc.?

This will hardly do on HP-UX, because there is a kernel parameter
"maxfiles" controlling the per-process max number of filedescriptors.

It's 60 by default after installation, but is tunable (with reboot).
I would not recommend to set it too high, since it's also a fuse against
single user processes eating up all available file descriptors (controlled
by nfiles).

We have hit the limit *very* quickly on our Winbind production box,
of course, and I have increased maxfiles to 300. Still quite low
when expecting a couple of hundret smbd to become winbind clients.
Each of them consuming two FDs.

The solution (and this should also work on other platforms) was to
have winbindd housekeep its client connections by shutting down
idle connections, and have clients reconnect when required:

   http://lists.samba.org/pipermail/samba-technical/2003-February/042210.html

The threshold was chosen to be 100 active connections, which keeps
winbindd well below 300 FDs. Below 140, actually, including network
sockets and open database and log files.

This only works out well if clients don't connect too frequently,
however, and

   http://lists.samba.org/pipermail/samba-technical/2003-February/042170.html

helped achieving this.

I'm tracking winbindd shutting down sockets for about a week now,
and have extended the DEBUG line in remove_idle_client() to also print
idle time of removal candidates.

With about 100 concurrent smbds (i.e. ~200 client pipes) it
almost always finds connections idle for more than an hour.
I would assume forcing these to reconnect should have no measurable
impact, and the solution should scale to a multitude of its
current load.

It can't be applied directly to 3.0, however. I'm assuming that identifying
idle connections is more complicated there, as both read and write buffers
can be empty while waiting for a request to complete. But it should
nevertheless be possible.

Cheers!
Michael