Samba 2.0.7 Memory Bloat under Sparc Solaris 5.8

[acherry at pobox.com]

David Collier-Brown writes:
 > 
 > 	Sure enough, in the main application heap.
 > 	If it had been a Solaris problem, it would not
 > 	have shown up here.  So far so good...

We've been doing a large-scale test of Samba 2.0.7 environment, and
although we're not exhausting our resources yet, we appear to also be
having some memory issues -- since we weren't running 2.0.6 before, I
don't really have anything to compare with, but some of the Samba
processes on our server (E4500 running Solaris 2.6) seem to be getting
a large virtual footprint.  It's not causing us problems (yet), and we
don't have any printer shares:

ctcxssfs1p:/opt-fs1# memhogs
    USER   PID  PPID %CPU RSS  VSZ   NI COMMAND
    root 24610 10056  0.0 6176 19776 20 /opt/samba/bin/smbd
    root 19557 10056  0.0 6032 10888 20 /opt/samba/bin/smbd
    root  6220 10056  0.0 6032 10880 20 /opt/samba/bin/smbd
    root  4002 10056  0.0 6032 10880 20 /opt/samba/bin/smbd
   bq979 22702 10056  0.0 6024 10880 20 /opt/samba/bin/smbd
    root 15903 10056  0.0 6024 10888 20 /opt/samba/bin/smbd
    root 18934 10056  0.0 6016 10888 20 /opt/samba/bin/smbd
    root 10097 10056  0.0 6016 10928 20 /opt/samba/bin/smbd
    root 17060 10056  0.0 5992 10888 20 /opt/samba/bin/smbd
    root 15935 10056  0.0 5976 10888 20 /opt/samba/bin/smbd
    root  9686 10056  0.0 5976 10888 20 /opt/samba/bin/smbd
    root   495 10056  0.0 5976 10888 20 /opt/samba/bin/smbd
    root 24609 10056  0.0 5968 10880 20 /opt/samba/bin/smbd
   ay427  8670 10056  0.0 5952 10880 20 /opt/samba/bin/smbd
    root 16260 10056  0.0 5952 10888 20 /opt/samba/bin/smbd
   bo762 10379 10056  0.0 5936 10880 20 /opt/samba/bin/smbd
   mm329 22531 10056  0.0 5888 10880 20 /opt/samba/bin/smbd
    root 18071 10056  0.1 5880 10888 20 /opt/samba/bin/smbd
  etc....

Sample 'pmap' output (on the top process):

ctcxssfs1p:/opt-fs1# /usr/proc/bin/pmap -x 24610
24610:  /opt/samba/bin/smbd -D -a -s /opt-fs1/samba/lib/smb.conf -l /opt-fs1/s
Address   Kbytes Resident Shared Private Permissions       Mapped File
00010000     816     712     704       8 read/exec         smbd
000EA000     240     232     184      48 read/write/exec   smbd
00126000   12016    3200       -    3200 read/write/exec    [ heap ]
EF000000    5120     632     632       - read/write/exec/shared  [shmid=0x1]
EF530000       8       8       8       - read/exec         libdoor.so.1
EF540000       8       8       8       - read/write/exec   libdoor.so.1
EF560000      24      24      24       - read/exec         nss_nisplus.so.1
EF574000       8       8       8       - read/write/exec   nss_nisplus.so.1
EF580000     592     592     592       - read/exec         libc.so.1
EF622000      32      32       8      24 read/write/exec   libc.so.1
EF62A000       8       8       -       8 read/write/exec    [ anon ]
EF640000       8       8       -       8 read/write/exec    [ anon ]
EF650000      16      16      16       - read/exec         nss_files.so.1
EF662000       8       8       -       8 read/write/exec   nss_files.so.1
EF670000      16      16      16       - read/exec         libc_psr.so.1
EF680000     448     448     448       - read/exec         libnsl.so.1
EF6FE000      40      40      16      24 read/write/exec   libnsl.so.1
EF708000      24       8       8       - read/write/exec    [ anon ]
EF720000      16      16      16       - read/exec         libmp.so.2
EF732000       8       8       -       8 read/write/exec   libmp.so.2
EF740000      24      24      24       - read/exec         libpam.so.1
EF754000       8       8       -       8 read/write/exec   libpam.so.1
EF756000       8       -       -       - read/write/exec    [ anon ]
EF760000      16      16      16       - read/shared       dev:157,8 ino:217170
EF770000      32      32      32       - read/exec         libsocket.so.1
EF786000       8       8       -       8 read/write/exec   libsocket.so.1
EF788000       8       -       -       - read/write/exec    [ anon ]
EF790000       8       8       8       - read/exec         libsec.so.1
EF7A0000      16      16       8       8 read/write/exec   libsec.so.1
EF7B0000       8       8       8       - read/exec         libdl.so.1
EF7C0000       8       8       -       8 read/write/exec    [ anon ]
EF7D0000     112     112     112       - read/exec         ld.so.1
EF7FA000       8       8       -       8 read/write/exec   ld.so.1
EFFF2000      56      24       -      24 read/write/exec    [ stack ]
--------  ------  ------  ------  ------
total Kb   19776    6296    2896    3400


Any idea why the heap would get so large?  (Much of it isn't resident;
also, I'm doing this at 11 PM, so the system is largely idle).  Any
ideas for diagnosis?  Unlike the problem you are looking at, this
doesn't happen immediately upon startup, so I'm not sure how useful
truss and the like will be.  We seem to be averaging about 5.5MB
virtual mem per smbd (after subtracting out the 5MB shared segment),
and about 2.8MB per smbd in resident private memory. Most of that is
on the application heap.  The average is taken over a total of 227
smbd processes (during the day we usually have more than this).

As you can see from the ps listing at the top, many of them seem to
have near-identical memory usage.

-Andrew Cherry
 UNIX System Admin
 Cummins, Inc.