3.0.11 with AIX 5.3, unable to claim a session

[yaberger at ca.ibm.com]

Hi,

I'm trying to build samba-3.0.11 on AIX 5.3 ML 1. The compilation is
working fine. I can start smbd server without any problem, but I'm unable
to connect a share to it (the authentication is working). Most of the time,
it's coredumping.
I'm not experiencing this problem on AIX 4.3.3 and AIX 5.2. I've tried with
those compiler: ibmcxx 3.6.6 PTF8 and IBM XL C for Enterprise for AIX 7.0
PTF 1

The problem seem to be with the TDB_CONTEXT tdb in smbd/session.c. It is
initialized when we start the server with the session_init() function and
it's useable (I've tried tdb->hash_fn(&key) right after the TDB_CONTEXT is
created).
When a first session is requested, it's unable to use the TDB_CONTEXT that
was previously created by the server. It's crashing when calling a method
of tdb (tbd->hash_fn(&key) in this case, when tdb_store() is called from
session_claim()).

Could this be because the memory declared for the tdb variable in the smbd
server could not be use by the child process?



Here is the backtrace of a core file (from dbx and from gdb)

==> dbx /usr/local/samba/sbin/smbd core
Type 'help' for help.
[using memory image in core]
reading symbolic information ...

Illegal instruction (illegal opcode) in . at 0x0 ($t1)
warning: Unable to access address 0x0 from core
(dbx) where
<.() at 0x0
tdb_store(0x2ff21848, 0x2ff21340, 0xd, 0x2ff21440, 0x610, 0x1), line 1471
in "tdb.c"
session_claim(??), line 143 in "session.c"
register_vuid(0x30189318, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2ff2211c), line
252 in "password.c"
reply_sesssetup_and_X(??, ??, ??, ??, ??), line 915 in "sesssetup.c"
switch_message(??, ??, ??, ??, ??), line 968 in "process.c"
construct_reply(??, ??, ??, ??), line 998 in "process.c"
process_smb(??, ??), line 1098 in "process.c"
unnamed block in smbd_process(), line 1558 in "process.c"
smbd_process(), line 1558 in "process.c"
main(argc = 0, argv = (nil)), line 951 in "server.c"




==> gdb /usr/local/samba/sbin/smbd core
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "powerpc-ibm-aix5.1.0.0"...
Core was generated by `smbd'.
Program terminated with signal 4, Illegal instruction.
#0  0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0x100606d0 in tdb_store (tdb=0x2ff21848, key={dptr = 0x2ff21340
"ID/40216/100", dsize = 13}, dbuf={dptr = 0x2ff21440 "", dsize = 1552},
flag=1) at tdb/tdb.c:1471
#2  0x101152ac in session_claim (vuser=0x0) at smbd/session.c:143
#3  0x100abc98 in register_vuid (server_info=0x30189318, session_key={data
= 0x0, length = 0, free = @0x0: 0x1}, response_blob={data = 0x0, length =
0, free = @0x0: 0x1}, smb_name=0x2ff2211c "yaberge2") at
smbd/password.c:252
#4  0x102126cc in reply_sesssetup_and_X (conn=0x0, inbuf=0x0,
outbuf=0x42360484 <Address 0x42360484 out of bounds>, length=806645800,
bufsize=0) at smbd/sesssetup.c:915
#5  0x100320dc in switch_message (type=0, inbuf=0x0, outbuf=0x0, size=0,
bufsize=0) at smbd/process.c:968
#6  0x10031b28 in construct_reply (inbuf=0x300104f0
"receive_message_or_smb", outbuf=0x3000ffbc "smbd/process.c",
size=804399040, bufsize=-559038737) at smbd/process.c:998
#7  0x100339e0 in process_smb (inbuf=0x1007252c "\200A", outbuf=0x0) at
smbd/process.c:1098
#8  0x1003327c in smbd_process () at smbd/process.c:1558
#9  0x10000d24 in main (argc=0, argv=0x0) at smbd/server.c:951


Yannick Bergeron
yaberger at ca.ibm.com