Abnormal system loading boost.

[Jason]

Abnormal system loading boost.

Recently, our samba server always suffers from system loading increasing
abnormally. The system load may be very high to even 30 or 40.
By our observation, when the system loading went up, there are 
several smbd processes 'locking' the smbpasswd file. According to further
check, those smbd processes are for IPC connections.

uptime
 6:00PM  up 5 days, 19:48, 9 users, load averages: 27.65, 33.93, 38.45

smbstatus | grep IPC 
IPC$         nobody   nobody   76288   pc117-11 (192.168.117.11) Wed Jun 18 17:59:11 2003
IPC$         nobody   nobody   75903   pc116-53 (192.168.116.53) Wed Jun 18 17:59:16 2003
IPC$         nobody   nobody   76283   pc116-46 (192.168.116.46) Wed Jun 18 17:59:08 2003
IPC$         nobody   nobody   76006   pc116-09 (192.168.116.9) Wed Jun 18 17:50:27 2003
IPC$         nobody   nobody   76292   pc110-26 (192.168.110.26) Wed Jun 18 17:59:12 2003
IPC$         nobody   nobody   76287   pc117-14 (192.168.117.14) Wed Jun 18 17:59:10 2003
IPC$         nobody   nobody   76010   pc116-14 (192.168.116.14) Wed Jun 18 17:50:28 2003
IPC$         nobody   nobody   76296   pc110-22 (192.168.110.22) Wed Jun 18 17:59:13 2003
IPC$         nobody   nobody   76290   pc116-23 (192.168.116.23) Wed Jun 18 17:59:12 2003
IPC$         nobody   nobody   76300   pc110-24 (192.168.110.24) Wed Jun 18 17:59:14 2003
IPC$         nobody   nobody   75929   pc116-50 (192.168.116.50) Wed Jun 18 17:59:14 2003
IPC$         nobody   nobody   76294   pc110-30 (192.168.110.30) Wed Jun 18 17:59:13 2003
IPC$         nobody   nobody   76304   pc110-27 (192.168.110.27) Wed Jun 18 17:59:16 2003
IPC$         nobody   nobody   76298   pc116-22 (192.168.116.22) Wed Jun 18 17:59:12 2003
IPC$         nobody   nobody   76308   pc117-07 (192.168.117.7) Wed Jun 18 17:59:13 2003

lsof smbpasswd 
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF   NODE NAME
smbd    75837   root   19rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    75909   root   25rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76073   root   19rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76083   root   19rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76233   root   19rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76284   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76285   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76286   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76287   root   19rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76288   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76289   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76290   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76291   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76292   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76293   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76294   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76296   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd
smbd    76298   root   22rr VREG 13,131076  1663526 174833 /usr/local/samba/private/smbpasswd

My question is why does smbd locks smbpasswd, and anyway to solve my problem?

Our environment:
Samba server: FreeBSD 4.8-STABLE
              CPU 2*p3-1.4G with 4G mem
length of smbpasswd: 15093

This is for pc-room of university computer center.
We have about 300 clients, windows XP installed, some were upgraded to SP1
while some are remained the original XP.

What I have tried:
1. compile with --with-sendfile-support and turn on use sendfile in smb.conf
2. compile with --with-spinlocks
3. tried profile acl yes/no in smb.conf
4. try to reduce the number of entries in smbpasswd,
   we reduced it from 35000+ to 15000 now (removing those seldom used accounts)

And as you can imagine, all in vain.

One thing that I am not quire sure, I remembered that in the days we use
samba 2.2.5 or even older versions, the problem seems not that severe,
although the system loading still increased, however, it will recover
in acceptable duration(such as several minutes). The problem became
obvious after we upgrade the samba to 2.2.7 and later.

We're running out of ideas here, and hope someone can give us a hand :-)

Sincerely,
Jason Chang

Computer Center
Taiwan University