Samba 2.0.7 mem leaks, continued

[acherry at pobox.com]

Hello-

I've done a bit more investigation regarding our memory problems with
Samba 2.0.7 under Solaris 2.6.  Going through some process accounting
logs I've been keeping, it looks pretty clear that there is some
memory leakage going on. Some things to note:

Going through the logs, I see the size of ALL smbd processes gradually 
increasing, which implies that the parent process is the one that is
growing.  Moreover, the growth happened over the weekend, when not
many people were working -- however, I *was* tweaking some of the smb
configs and sending HUP signals to the parent smbd process to re-read
the smb.conf files.

Below is some output from the memory checking functions of dbx.  To do
this, I preloaded the appropriate library in the smbd startup script.
Since dbx seems to require that the process being checked end normally
(i.e. not be killed) in order to gather its data, I had to attach it
to the parent process, set it to follow children, and then open up a
session.  During this session I sent several HUPs to the child
process, as well as doing various other operations (running
applications, loading/saving files, etc).  I then exited the session
to get dbx to return its stats.  I've included the results below.  I'm
not all that familiar with Samba's internals yet, but it appears to
suggest a leak in the config file reloading code, particularly the
(new as of 2.0.7) hash-based stat_cache code.  This would be
consistent with the symptoms I'm seeing.

Hope this is helpful.  If I get a chance to look at things in more
detail I'll send another note -- I'm using a debug build of Samba to
do this testing, so I might be able to gather more info.

BTW, the version I'm using right now is built with Sun WorkShop 5.0
(since dbx needs that in order to do the leak checking)... if that
makes a difference.

-Andrew Cherry
 UNIX System Admin
 Cummins Engine Company

======================================

<rtc> Read from unallocated (rua):
Attempting to read 1 byte at address 0x213d40
    which is at start of heap block of size 8 bytes at 0x213d40
This block was allocated from:
	[1] tzcpy() at 0xef55175c 
	[2] getzname() at 0xef5516ac 
	[3] _ltzset_u() at 0xef551260 
	[4] _tzset() at 0xef5515d0 
	[5] strftime() at 0xef57d50c 
	[6] timestring() at line 513 in "time.c"
	[7] dbghdr() at line 548 in "debug.c"
	[8] sig_hup() at line 340 in "server.c"
Location of error:
  [1] strlen(0x213d40, 0x3, 0xef7a8000, 0x1, 0xefffce48, 0xef7a738c), at 0xef78513c
=>[2] tzcpy(0x213d40, 0xefffff42, 0x0, 0x7, 0xef5a3180, 0x10), at 0xef5516fc
  [3] getzname(0xefffff42, 0xef5a6594, 0x0, 0xefffff49, 0xefffcf08, 0x0), at 0xef5516ac
  [4] _ltzset_u(0xef5a6594, 0xef5a3180, 0x0, 0x1, 0xef5a3180, 0xef5a6570), at 0xef551260
  [5] _tzset(0xef5a65a0, 0x25ca0, 0xef7a8000, 0x1, 0xefffcfc8, 0x39b034a5), at 0xef5515d0
  [6] strftime(0x1a76e0, 0x1a76e0, 0x64, 0xef5a9880, 0xef5a9880, 0x136f10), at 0xef57d50c
  [7] timestring(hires = ???) (optimized), at 0xcb794 (line ~513) in "time.c"
  [8] dbghdr(level = ???, file = ???, func = ???, line = ???) (optimized), at 0xc45b4 (line ~548) in "debug.c"
  [9] sig_hup(sig = ???) (optimized), at 0x2c510 (line ~340) in "server.c"
  [10] sigacthandler(0x1, 0x0, 0xefffd280, 0x18, 0xffffffff, 0xef7a738c), at 0xef538a0c
  ---- called from signal handler with signal 1 (SIGHUP) ------
  [11] poll(0x5b, 0x2, 0xea60, 0x18, 0xffffffff, 0xef7a738c), at 0xef53743c
  [12] _poll(0xef7a8400, 0x2, 0xefffd598, 0xea60, 0x0, 0xefffd610), at 0xef786a50
  [13] _select(0xefffd620, 0xef5a60fc, 0xef5a60fc, 0xef5a6100, 0xef5a6100, 0x9), at 0xef54cc74
  [14] sys_select(maxfd = ???, fds = ???, tval = ???) (optimized), at 0xc9198 (line ~94) in "system.c"
  [15] receive_message_or_smb(buffer = ???, buffer_len = ???, timeout = ???, got_smb = ???) (optimized), at 0x652d4 (line ~135) in "process.c"
  [16] smbd_process() (optimized), at 0x667f8 (line ~991) in "process.c"
  [17] main(argc = ???, argv = ???) (optimized), at 0x2d1d0 (line ~502) in "server.c"

<rtc> Read from uninitialized (rui):
Attempting to read 4 bytes at address 0xeffff640
    which is 112 bytes above the current stack pointer
Variable is 'nread'
=>[1] reply_lockread(conn = ???, inbuf = ???, outbuf = ???, length = ???, dum_buffsiz = ???) (optimized), at 0x4fc3c (line ~2160) in "reply.c"
  [2] reply_lockread(conn = ???, inbuf = ???, outbuf = ???, length = ???, dum_buffsiz = ???) (optimized), at 0x4fc30 (line ~2160) in "reply.c"
  [3] switch_message(type = ???, inbuf = ???, outbuf = ???, size = ???, bufsize = ???) (optimized), at 0x65a10 (line ~416) in "process.c"
  [4] construct_reply(inbuf = ???, outbuf = ???, size = ???, bufsize = ???) (optimized), at 0x65aa0 (line ~555) in "process.c"
  [5] process_smb(inbuf = ???, outbuf = ???) (optimized), at 0x65c8c (line ~586) in "process.c"
  [6] smbd_process() (optimized), at 0x66870 (line ~991) in "process.c"
  [7] main(argc = ???, argv = ???) (optimized), at 0x2d1d0 (line ~502) in "server.c"

<rtc> Read from uninitialized (rui):
Attempting to read 4 bytes at address 0xeffff644
    which is 116 bytes above the current stack pointer
=>[1] reply_writeunlock(conn = ???, inbuf = ???, outbuf = ???, size = ???, dum_buffsize = ???) (optimized), at 0x50c68 (line ~2433) in "reply.c"
  [2] reply_writeunlock(conn = ???, inbuf = ???, outbuf = ???, size = ???, dum_buffsize = ???) (optimized), at 0x50c5c (line ~2433) in "reply.c"
  [3] switch_message(type = ???, inbuf = ???, outbuf = ???, size = ???, bufsize = ???) (optimized), at 0x65a10 (line ~416) in "process.c"
  [4] construct_reply(inbuf = ???, outbuf = ???, size = ???, bufsize = ???) (optimized), at 0x65aa0 (line ~555) in "process.c"
  [5] process_smb(inbuf = ???, outbuf = ???) (optimized), at 0x65c8c (line ~586) in "process.c"
  [6] smbd_process() (optimized), at 0x66870 (line ~991) in "process.c"
  [7] main(argc = ???, argv = ???) (optimized), at 0x2d1d0 (line ~502) in "server.c"


Actual leaks report    (actual leaks:       195  total size:   53568 bytes)

 Total  Num of  Leaked      Allocation call stack
 Size   Blocks  Block
                Address
======  ====== ==========  =======================================
 43764       7      -      hash_table_init < reset_stat_cache 
  4887      88      -      hash_insert < stat_cache_add 
  4696      83      -      stat_cache_add < unix_convert 
   150       5      -      stat_cache_add < unix_convert 
    71      12      -      strdup < set_namearray 
 

Possible leaks report  (possible leaks:       2  total size:  133118 bytes)

 Total  Num of  Leaked      Allocation call stack
 Size   Blocks  Block
                Address
======  ====== ==========  =======================================
 66559       1   0x3ccfa8  smbd_process < main 
 66559       1   0x3bcb90  smbd_process < main 
 

Blocks in use report   (blocks in use:     5656  total size:  828251 bytes)

 Total  % of Num of  Avg     Allocation call stack
 Size    All Blocks  Size
======= ==== ====== ======  =======================================
 304128  36%    297   1024  init_copymap < copy_service 
 131072  15%      1 131072  load_unicode_map < load_dos_unicode_map < codepage_initialise < main 
 114848  13%    296    388  add_a_service < do_section 
  81772   9%   4786     17  string_init < string_set 
  65539   7%      1  65539  cli_initialise < server_cryptkey < reply_nt1 < reply_negprot < switch_message < construct_reply < process_smb < smbd_process 
  65539   7%      1  65539  cli_initialise < server_cryptkey < reply_nt1 < reply_negprot < switch_message < construct_reply < process_smb < smbd_process 
  16384   1%      1  16384  clnt_vc_create < clnt_tli_create < nis_make_rpchandle_uaddr < nis_handle < __nis_get_server < __nis_remote_lookup < nis_list < __nis_list_localcb 
   8200  <1%      1   8200  read_vc < fill_input_buf < get_input_bytes < set_input_fragment < xdrrec_getbytes < xdrrec_getlong < xdr_replymsg < clnt_vc_call 
   8192  <1%      1   8192  talloc < lp_string < lp_configfile < reload_services < check_reload < timeout_processing < smbd_process < main 
   6252  <1%      1   6252  hash_table_init < reset_stat_cache < reload_services < check_reload < timeout_processing < smbd_process < main 
   4100  <1%      1   4100  xdrrec_create < clnt_vc_create < clnt_tli_create < nis_make_rpchandle_uaddr < nis_handle < __nis_get_server < __nis_remote_lookup < nis_list 
   4096  <1%      1   4096  clnt_vc_create < clnt_tli_create < nis_make_rpchandle_uaddr < nis_handle < __nis_get_server < __nis_remote_lookup < nis_list < __nis_list_localcb 
   3080  <1%      1   3080  calloc < __get_local_names1 < __get_local_names < nis_local_directory < _nss_nisplus_constr < nss_get_backend_u < nss_search < _getgroupsbymember 
   1344  <1%      1   1344  _nss_XbyY_buf_alloc < gethostbyname < sys_gethostbyname < Get_Hostbyname < open_socket_in < open_sockets < main 
   1188  <1%      1   1188  Realloc < add_a_service < lp_add_ipc < lp_load < reload_services < main 
   1072  <1%      1   1072  _nss_XbyY_buf_alloc < getpwnam < sys_getpwnam < do_reseed < generate_random_buffer < main 
   1024  <1%      1   1024  getmntent < getdevinfo < getcwd < sys_getwd < dos_getwd < dos_GetWd < oplock_break < request_oplock_break 
    956  <1%     18     53  hash_insert < stat_cache_add 
    912  <1%      1    912  _t_alloc_bufs < _t_create < _tx_open < clnt_tli_create < nis_make_rpchandle_uaddr < nis_handle < __nis_get_server < __nis_remote_lookup 
    912  <1%      1    912  _t_alloc_bufs < _t_create < _tx_open < clnt_tli_create < nis_make_rpchandle_uaddr < nis_handle < __nis_get_server < __nis_remote_lookup