Samba 3.0 / 3.2 heap overflow on AIX?

[yaberger at ca.ibm.com]

Hi Volker

Thanks for your prompt answer

This is new information for me that rbtree.c is new in 3.2 but I've 
figured it was from the Linux kernel by reading the headers. Good 
information there
This would mean that the problem might be different between 3.0 and 3.2 
but the result is the same. And maybe the problem is not related to 
rbtree.c at all...... I'm having some hard time to understand how now 
correctly narrow the overflow :/

About 3.2 vs 3.0, as a reference, you can read this mail on 
samba-technical mailing list that explain the exact same problematic I had 
when I tried to run samba 3.0.11 and 3.0.25 on AIX 5.3 with DCE userids 
but reported by someone else: 
http://lists.samba.org/archive/samba-technical/2006-September/049234.html

I've compiled 3.0.28a vanilla this night and did the same test, ie setting 
the MALLOCDEBUG/MALLOCTYPE env variable and it has coredumped silently (no 
"IOT/Abort trap(coredump)" output on screen).
Here is the stack gathered from the core:

root at aix53tst ==> dbx /usr/local/samba/sbin/smbd
Type 'help' for help.
[using memory image in core]
reading symbolic information ...

IOT/Abort trap in raise.raise [/lib/libc.a] at line 78 in file ""
could not read 
"/gsa/ausgsa/projects/a/aix.4/aix53L/53L_SERVICE/src/bos/usr/ccs/lib/libc/raise.c"
(dbx) where
raise.raise(sig = 6), line 78 in "raise.c"
abort.abort(), line 94 in "abort.c"
dump_core(), line 192 in "fault.c"
smb_panic(why = "smb_xmalloc_array failed to allocate %lu * %lu bytes\n"), 
line 1649 in "util.c"
smb_xmalloc_array(size = 536927316, count = 0), line 2460 in "util.c"
smb_xmemdup(p = (nil), size = 0), line 2472 in "util.c"
data_blob(p = (nil), length = 0), line 50 in "data_blob.c"
make_new_server_info_guest(server_info = (nil)), line 1514 in 
"auth_util.c"
init_guest_info at AF50_25(), line 1582 in "auth_util.c"
main(argc = 3, argv = 0x200003b0), line 1058 in "server.c"




I might try to recompile 3.0.28a with my patch, start it without the 
MALLOCDEBUG/MALLOCTYPE, try the authentication from my windows workstation 
with the DCE userids and gather the stack associated with the core to 
confirm it's similar to the one from 3_2_stable (ie: coredump in 
initgroups)




Yannick Bergeron
yaberger at ca.ibm.com



Volker Lendecke <Volker.Lendecke at SerNet.DE> 
2008-05-01 17:21
Please respond to
Volker.Lendecke at SerNet.DE


To
yaberger at ca.ibm.com
cc
samba-technical at lists.samba.org
Subject
Re: Samba 3.0 / 3.2 heap overflow on AIX?






On Thu, May 01, 2008 at 03:50:57PM -0400, yaberger at ca.ibm.com wrote:
> I'm trying to find a possible heap overflow which first seemed to be in 
> AIX 5.3 with Samba (3.0 ou 3.2).
> With the AIX support, we've been able to use some debugging utility, 
debug 
> libc, etc. on AIX that allow the support to think the problem might be 
in 
> Samba code

rbtree.c is new in 3.2, it's from the Linux kernel. It might
well be that the AIX compiler does not like it, it uses some
pointer tricks. I would guess that the AIX compiler in that
strict mode would not allow C code to set a pointer to a
non-aligned value at all, and that this raises the
exception. rbtree.c however is quite careful to strip those
bits again before the pointer is actually dereferenced.

We might have to modify the code for these special compiler
settings if they are of general, wider use in production
systems.

But you said that you also get it with 3.0 which does not
have this new code. Can you try to do the same analysis
there?

Thanks,

Volker
[attachment "atti9zza.dat" deleted by Yannick Y Bergeron/Bromont/IBM]