Proposed patch for Samba-2.0.7 to allow Solaris open more than 1014 (or rlim_fd_max) files

[Eric Boehm]

I would like to submit the following patch to source/lib/util.c to allow
Samba under Solaris to open more than 1014 files -- or whatever the kernel
variable rlim_fd_max is set to.

A little background:
We use Samba to provide Windows clients a way to access ClearCase VOBs
(versioned object bases) hosted by Solaris servers. One limitation has been
that smbd could only open (rlim_fd_max - 10) files. It isn't a good idea to
set rlim_fd_max above 1024 because a higher value will break any system that
uses the select() system call. Solaris 7 and 8 provide a way around this but
it requires that the affected application(s) be recompiled as a 64-bit
application. However, it isn't always possible to get applications recompiled
as 64 bit.

This presents a problems because ClearCase will often want to open more than
1014 files. You can change a setting in ClearCase to reduce the number of
files that can be opened at once but this doesn't always solve the problem.

util.c calls setrlimit to raise the soft limit (rlim_fd_cur) to the hard limit
(rlim_fd_max). setrlimit can also raise the hard limit provided the effective
userid is root. smbd and nmbd usually run as root.

This avoids the need to change rlim_fd_max and break programs that use
select(). It only changes hard limit for smbd/nmbd processes.

This change would allow you to specify any value for "max open files =".

To get the full benefit, you will also need to either

a) compile with sfio (which requires some additional changes to
   source/include/smb.h and source/include/config.h -- from David Boyce
   <dsb at world.std.com) 

OR

b) compile Samba as a 64-bit application

The reason for this is that smbd uses fopen to open the machine account (mac)
file. fopen without sfio or in a 32-bit compilation environment is limited to
256 file descriptors. If an smbd process has > 256 files but < 1014
(rlim_fd_max) files open it can still fail to open the machine account file.

If this patch is acceptable, I can probably provide changes for configure to
handle the changes to source/include/smb.h and source/include/config.h. The
fix for smb.h is not necessary in the Samba 2.2 stream.

--- samba-2.0.7/source/lib/util.c.~1~	Tue Apr 25 19:06:53 2000
+++ samba-2.0.7/source/lib/util.c	Tue Jan 23 08:00:48 2001
@@ -3012,6 +3012,18 @@
 	 * which always returns RLIM_INFINITY for rlp.rlim_max.
 	 */
 
+        if ( geteuid() == 0 ) {
+          
+          if ( rlp.rlim_max < requested_max ) {
+            rlp.rlim_max = requested_max;
+          }
+
+          if(setrlimit(RLIMIT_NOFILE, &rlp)) {
+            DEBUG(0,("set_maxfiles: setrlimit for RLIMIT_NOFILE for %d max files failed with error %s\n", 
+                     (int)rlp.rlim_max, strerror(errno) ));
+          }
+        }
+
 	saved_current_limit = rlp.rlim_cur = MIN(requested_max,rlp.rlim_max);
 
 	if(setrlimit(RLIMIT_NOFILE, &rlp)) {

-- 
Eric M. Boehm                               boehm at nortelnetworks.com