Samba 3.0 / 3.2 heap overflow on AIX?

[yaberger at ca.ibm.com]

Hi,

I'm trying to find a possible heap overflow which first seemed to be in 
AIX 5.3 with Samba (3.0 ou 3.2).
With the AIX support, we've been able to use some debugging utility, debug 
libc, etc. on AIX that allow the support to think the problem might be in 
Samba code

Actually, without any modification to Samba code and using a local (AIX) 
userid, Samba is working #1.
But when I add a few lines to allow AIX authentication on using the LAM 
(Loadable Authentication Module) and authenticating with a DCE userid 
belonging to more than 32 groups, it coredump.
The same patch/tests is working #1 without this problem on AIX 5.2 btw.
Before telling me that the problem might come from my patch or the DCE 
client or AIX, please take the time to read what's below

With this scenario, Samba was coredumping when I was doing the 
connection/authentication from a windows workstation with the DCE userid 
(belonging to more than DCE groups). By looking at the stack/traces of the 
core file, it pointed me that it was coredumping in initgroups(), which is 
an OS syscall. By replacing the AIX syscall with the rep_initgroups() 
you're providing to system without an initgroups() implementation, I'm 
able to authenticate with this user correctly (no coredump)

I've opened a PMR (Problem Management Request) to the AIX team because I 
thought the overflow was in initgroups() or a subroutine called by it 
which wasn't in debug mode. After some collaboration with them, we've 
ended with the following 2 tests that let us think the heap overflow might 
be in Samba and possibly in rbtree.c (explaination in the test below).
It might be possible that you guys are doing some intentional overflow 
while managing memory like a Java JRE would do and this would explain why 
it coredump that fast when

We're using 2 environment variable to track the overflow: MALLOCDEBUG and 
MALLOCTYPE. With MALLOCDEBUG, we can set the parameter "align", which is a 
value in byte (0,1,2,4,8, maybe more) of what is permitted to "overflow". 
By setting it value to 0, we expect Samba to coredump as soon as an 
overflow occur and not further in the code (like it was happening before 
setting the 2 env. var., ie coredumping in initgroups())

So here is the 2 tests I've made:
Download the latest 3_2_stable branch (I can also reproduce with 3.0.28a 
if you want). Do not apply my patch or any modification to the code
Compile it one time with the CFLAGS I'm usually using (test 1) and one 
time with default CFLAGS (test 2)
install, set the MALLOCDEBUG and MALLOCTYPE env. variable and start samba
it coredump, I gather the stack from the core

I'm trying to discuss about it with William Jojo (because I know he's used 
with the AIX platform from my past experience) but he's currently in his 
finals and I don't want to disturbe him too much.

So if any developper is understanding my problem/explanations/tests and 
would like to comment them, suggest something (test, modification, etc.), 
I would gratefully accept.
I still have my workaround in my pocket which is to for the use of 
rep_initgroups() but I think it would be great to find and correct the 
real problem if there is one


root at aix53tst ==> rsync -avzP --delete 
samba.org::ftp/unpacked/samba_3_2_stable .
# the update date/time for this branch is Apr 28 04:05

root at aix53tst ==> cd samba_3_2_stable/source
root at aix53tst ==> ./configure

# modifying the CFLAGS in the Makefile
# was: CFLAGS= -O -D_SAMBA_BUILD_=3 -D_LINUX_SOURCE_COMPAT -qmaxmem=32000
# now: CFLAGS= -g -O3 -qstrict -D_SAMBA_BUILD_=3 -D_LINUX_SOURCE_COMPAT 
-qmaxmem=-1 -bmaxdata:0x80000000

root at aix53tst ==> make
root at aix53tst ==> make install


# Ok here is the test first test

root at aix53tst ==> export MALLOCTYPE=debug
root at aix53tst ==> export 
MALLOCDEBUG=align:0,catch_overflow,postfree_checking
root at aix53tst ==> /usr/local/samba/sbin/smbd -D 
-s/usr/local/samba/lib/smb.conf
IOT/Abort trap(coredump)

root at aix53tst ==> dbx -d 100 /usr/local/samba/sbin/smbd
(dbx) where
raise.raise(sig = 6), line 78 in "raise.c"
abort.abort(), line 94 in "abort.c"
dump_core(), line 212 in "fault.c"
smb_panic(why = 
"===============================================================\n"), line 
1635 in "util.c"
fault_report(sig = -800942328), line 46 in "fault.c"
rb_insert_color at AF14_5(??, ??), line 152 in "rbtree.c"
db_rbt_store(rec = 0x30e57ff4, data = (...), flag = 0), line 183 in 
"dbwrap_rbt.c"
dbwrap_store_bystring(db = 0x30019720, key = 
"/\362$\320"$B$^P^B^\81^F/\357", data = (...), flags = 805410592), line 
193 in "dbwrap.c"
hash_a_service(name = "tempdisk", idx = 3), line 5801 in "loadparm.c"
add_a_service(pservice = 0x1006aa70, name = "0\312\242'0\312\247}"), line 
5749 in "loadparm.c"
loadparm.do_section(pszSectionName = warning: Unable to access address 
0x6b from core
(invalid char ptr (0x0000006b)), userdata = 0x30023ff8), line 7353 in 
"loadparm.c"
Section(buf = 0x2ff22640, InFile = 0x00000100, sfunc = 0x2ff22600, 
userdata = 0x2945164c), line 301 in "params.c"
Parse(buf = (nil), InFile = 0x00001dd8, sfunc = 0x2ff22680, pfunc = 
0x00000050, userdata = 0x100238c4), line 486 in "params.c"
pm_process(FileName = "\200A", sfunc = (nil), pfunc = 0x2ff226e0, userdata 
= 0x302233c8), line 577 in "params.c"
lp_load_ex(pszFname = warning: Unable to access address 0x1a4 from core
(invalid char ptr (0x000001a4)), global_only = @0x00000000, save_defaults 
= @0x2ff22ab4, add_ipc = @0x00000000, initialize_globals = @0x30bbafe2, 
allow_include_registry = @0x00000000), line 8702 in "loadparm.c"
lp_load(pszFname = "Tc^F>\201\201", global_only = @0x00000000, 
save_defaults = @0x30212048, add_ipc = @0x302233c8, initialize_globals = 
@0x00000000), line 8780 in "loadparm.c"
reload_services(test = @0x30000718), line 846 in "server.c"
main(argc = 3, argv = 0x300006e0), line 1233 in "server.c"

# Last subroutine before the fault_report/smb_panic was in rbtree.c !!!


# Ok, I'll recompile but this time without using any additional CFLAGS 
except "-g"
# CFLAGS= -g -O -D_SAMBA_BUILD_=3 -D_LINUX_SOURCE_COMPAT -qmaxmem=32000

root at aix53tst ==> export MALLOCTYPE=
root at aix53tst ==> export MALLOCDEBUG=

root at aix53tst ==> make clean
root at aix53tst ==> make
root at aix53tst ==> make install

root at aix53tst ==> export MALLOCTYPE=debug
root at aix53tst ==> export 
MALLOCDEBUG=align:0,catch_overflow,postfree_checking
root at aix53tst ==> /usr/local/samba/sbin/smbd -D 
-s/usr/local/samba/lib/smb.conf
IOT/Abort trap(coredump)

root at aix53tst ==> dbx -d 100 /usr/local/samba/sbin/smbd
(dbx) where
raise.raise(sig = 6), line 78 in "raise.c"
abort.abort(), line 94 in "abort.c"
dump_core(), line 212 in "fault.c"
smb_panic(why = 
"===============================================================\n"), line 
1635 in "util.c"
fault_report(sig = -800942328), line 46 in "fault.c"
rb_insert_color at AF14_5(??, ??), line 152 in "rbtree.c"
db_rbt_store(rec = 0x20e57ff4, data = (...), flag = 0), line 183 in 
"dbwrap_rbt.c"
dbwrap_store_bystring(db = 0x20019720, key = 
"/\362$\240"$B$^P^B^\8!^F/\357", data = (...), flags = 536975136), line 
193 in "dbwrap.c"
hash_a_service(name = "tempdisk", idx = 3), line 5801 in "loadparm.c"
add_a_service(pservice = 0x1006aa70, name = " \312\242' \312\247}"), line 
5749 in "loadparm.c"
loadparm.do_section(pszSectionName = warning: Unable to access address 
0x6b from core
(invalid char ptr (0x0000006b)), userdata = 0x20023ff8), line 7353 in 
"loadparm.c"
Section(buf = 0x2ff22610, InFile = 0x00000100, sfunc = 0x2ff225d0, 
userdata = 0x31a4e5e3), line 301 in "params.c"
Parse(buf = (nil), InFile = 0x00001dd8, sfunc = 0x2ff22650, pfunc = 
0x00000050, userdata = 0x100238c4), line 486 in "params.c"
pm_process(FileName = "\200A", sfunc = (nil), pfunc = 0x2ff226b0, userdata 
= 0x202233c0), line 577 in "params.c"
lp_load_ex(pszFname = warning: Unable to access address 0x1a4 from core
(invalid char ptr (0x000001a4)), global_only = @0x00000000, save_defaults 
= @0x2ff22a8c, add_ipc = @0x00000000, initialize_globals = @0x20bbafe2, 
allow_include_registry = @0x00000000), line 8702 in "loadparm.c"
lp_load(pszFname = "Tc^F>\201\201", global_only = @0x00000000, 
save_defaults = @0x20212040, add_ipc = @0x202233c0, initialize_globals = 
@0x00000000), line 8780 in "loadparm.c"
reload_services(test = @0x20000718), line 846 in "server.c"
main(argc = 3, argv = 0x200006e0), line 1233 in "server.c"

# Last subroutine before the fault_report/smb_panic was in rbtree.c 
again!!!

Yannick Bergeron
yaberger at ca.ibm.com