smbd's running wild in Solaris x86 malloc...

[Lans H Carstensen]

Greetings!

We've got a neat little problem running Samba 1.9.18p8 on Solaris x86 2.5.1
and AFS on a dual-processor Dell PowerEdge 2200 w/ 128M RAM.  The machine has
11M of free real memory and 417M of free swap according to "top".

Every once in a while (there goes one now...) an "smbd" process will attempt
to consume all available processing time it can.  We recompiled with GNU
debugging and attached to the running process.  It appears that the behavior
is happening in the malloc that occurs in ipc.c at line 3295.  Information
that might be useful to someone:

1) Yes, that's right, malloc gets stuck.  The actual trace is:
#0  0x8005ea3f in t_splay ()
#1  0x8005e956 in t_delete ()
#2  0x8005e705 in realfree ()
#3  0x8005e2e2 in _malloc_unlocked ()
#4  0x8005e1b9 in malloc ()
#5  0x807773d in api_reply (cnum=12, vuid=0, outbuf=0x8122271 "", data=0x0, 
    params=0x80fea08 "h", tdscnt=0, tpscnt=26, mdrcnt=24900, mprcnt=8)
    at ipc.c:3295
#6  0x80778b9 in named_pipe (cnum=12, vuid=0, outbuf=0x8122271 "", 
    name=0x8047852 "LANMAN", setup=0x0, data=0x0, params=0x80fea08 "h", 
    suwcnt=0, tdscnt=0, tpscnt=26, msrcnt=0, mdrcnt=24900, mprcnt=8)
    at ipc.c:3340
#7  0x8077ddf in reply_trans (inbuf=0x8111e69 "", outbuf=0x8122271 "", 
    size=110, bufsize=61440) at ipc.c:3473
#8  0x808cfd1 in switch_message (type=0, inbuf=0x8111e69 "", 
    outbuf=0x8122271 "", size=110, bufsize=61440) at server.c:5086
#9  0x808d290 in construct_reply (inbuf=0x8111e69 "", outbuf=0x8122271 "", 
    size=110, bufsize=61440) at server.c:5251
#10 0x8089ace in process_smb (inbuf=0x8111e69 "", outbuf=0x8122271 "")
    at server.c:2835
#11 0x808d5ab in process () at server.c:5392
#12 0x808ddb9 in main (argc=2, argv=0x8047a44) at server.c:5713

2) api_command=104 (api_commands[i].name="NetServerEnum")
3) params+2="WrLehDO"
4) tdscnt=0
5) tpscnt=26
6) mdrcnt=24900 (24750 in another one I looked at)
7) mprcnt=8

So finally, !!!HELP!!! - does anyone have any idea why this might be
happening?  Can you link SAMBA against any different malloc libraries, like
GNU malloc?  Or does anyone else have any better ideas?  Thanks.

-- 
Lans Carstensen
Lans.Carstensen at rose-hulman.edu
Systems Integrator, Waters Computing Center
Rose-Hulman Institute of Technology