solaris 2.5.1 smbd 2.2.2 (& 2.2.0) SIGSEGV in initgroups when using nisplus

[Tyler, Ross E]

this IS NOT a problem on my solaris 2.7 machine with smbd 2.2.0.

it IS a problem on my solaris 2.5.1 machine (with latest kernel/nis/libc/libnsl patch 103640-37) on both smbd 2.2.2 and 2.2.0.

to reproduce the problem i simply try to list the shares of the local smbd (as the my 'samba' user).

    smbclient -L localhost -U samba
    Password: <sambapassword>
    session setup failed: code 0

log.smbd shows:

        INTERNAL ERROR: Signal 11 in pid 8519 (2.2.2)

connecting gdb to the fork'ed smbd process and break'ing at initgroups, i get the following stack traceback

#0  0xef5cdbf8 in initgroups () from /usr/lib/libc.so.1
#1  0x10bb78 in winbind_initgroups (user=0xefffef80 "samba", gid=125)
    at ../../src/source/nsswitch/wb_client.c:291
#2  0x613f4 in initialise_groups (user=0xefffef80 "samba", uid=125, gid=125)
    at ../../src/source/smbd/sec_ctx.c:234
#3  0x34a64 in register_vuid (uid=125, gid=125, unix_name=0xefffef80 "samba",
    requested_name=0x1b7578 "SAMBA", domain=0xefffea80 "C3I83.CV.FUL.US",
    guest=0) at ../../src/source/smbd/password.c:296
#4  0x4b1b0 in reply_sesssetup_and_X (conn=0xefffeb80, inbuf=0x214791 "",
    outbuf=0x224bd9 "", length=146, bufsize=65535)
    at ../../src/source/smbd/reply.c:1071
#5  0x68cd0 in switch_message (type=115, inbuf=0x214791 "",
    outbuf=0x224bd9 "", size=146, bufsize=65535)
    at ../../src/source/smbd/process.c:756
#6  0x68d5c in construct_reply (inbuf=0x214791 "", outbuf=0x224bd9 "",
    size=146, bufsize=65535) at ../../src/source/smbd/process.c:785
#7  0x69008 in process_smb (inbuf=0x214791 "", outbuf=0x224bd9 "")
    at ../../src/source/smbd/process.c:879
#8  0x69a3c in smbd_process () at ../../src/source/smbd/process.c:1270
#9  0x2e834 in main (argc=0, argv=0xeffffe2c)
    at ../../src/source/smbd/server.c:811

attempting to step, i get the following error

    Program received signal SIGSEGV, Segmentation fault.
    0xef7ddf14 in ?? ()

sometimes the stack seems corrupted after this:

#0  0xef5da290 in select () from /usr/lib/libc.so.1
#1  0xef6aab38 in __nis_cast_proc () from /usr/lib/libnsl.so.1
Cannot access memory at address 0xe6d

other times the stack seems OK:

#0  0xef7ddf14 in ?? ()
#1  0xef7d23f8 in ?? ()
#2  0xef616918 in _PROCEDURE_LINKAGE_TABLE_ () from /usr/lib/libc.so.1
#3  0xef5cdbf8 in initgroups () from /usr/lib/libc.so.1
:

the arguments to initgroups are

    initgroups("samba", 125)

samba is the name of a unix user (in nisplus passwd) and 125 is the gid of the samba group (in nisplus group).

note that a standalone program run as root with only a single call to initgroups with the same arguments succeeds!

note that smbd will succeed if i remove nisplus from the group line in /etc/nsswitch.conf:

    group: files nisplus

perhaps the parent's nisplus related sockets (file descriptors) are inherited by the child smbd and the child using them causes this
problem?????

perhaps this is a sun problem (since it does work on a later version of the OS).
if you believe it is, i would appreciate it if you took it up with them as i am not familiar with the inner workings of samba.

as i noted before, i am using the latest libnsl, nisplus and kernel for solaris 2.5

i really appreciate your support and your product.

keep up the good work!