Samba 3.0 / 3.2 heap overflow on AIX?

[Volker Lendecke]

On Thu, May 01, 2008 at 03:50:57PM -0400, yaberger at ca.ibm.com wrote:
> I'm trying to find a possible heap overflow which first seemed to be in 
> AIX 5.3 with Samba (3.0 ou 3.2).
> With the AIX support, we've been able to use some debugging utility, debug 
> libc, etc. on AIX that allow the support to think the problem might be in 
> Samba code

rbtree.c is new in 3.2, it's from the Linux kernel. It might
well be that the AIX compiler does not like it, it uses some
pointer tricks. I would guess that the AIX compiler in that
strict mode would not allow C code to set a pointer to a
non-aligned value at all, and that this raises the
exception. rbtree.c however is quite careful to strip those
bits again before the pointer is actually dereferenced.

We might have to modify the code for these special compiler
settings if they are of general, wider use in production
systems.

But you said that you also get it with 3.0 which does not
have this new code. Can you try to do the same analysis
there?

Thanks,

Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.samba.org/archive/samba-technical/attachments/20080501/a08b9255/attachment.bin