[Samba] Kerberos-related configure script problem on Solaris with 3.2.0

[David Eisner]

I'm building samba-3.2.0 on a Solaris 10 x64 box using the Sun Studio
cc compiler.

I'm running into a problem.  I have a workaround.  Let me know if I
should post a bug report.

A. The problem:

 $ ./configure --prefix=/usr/local/samba-3.2.0  --with-ads --with-pam
--with-acl-support --with-shared-modules=idmap_ad
--enable-socket-wrapper --enable-nss-wrapper
--with-krb5=/usr/local/heimdal/heimdal-1.1
...
config.status: creating pkgconfig/smbsharemodes.pc
config.status: creating include/config.h
config.status: executing rm-stdint.h commands
config.status: executing mk-stdbool.h commands
$ make
...
Compiling libsmb/clikrb5.c
"/usr/include/netdb.h", line 475: warning: macro redefined: MAXHOSTNAMELEN
"./librpc/gen_ndr/srvsvc.h", line 232: warning: enumerator value
overflows INT_MAX (2147483647)
"./librpc/gen_ndr/srvsvc.h", line 235: warning: enumerator value
overflows INT_MAX (2147483647)
"./librpc/gen_ndr/srvsvc.h", line 238: warning: enumerator value
overflows INT_MAX (2147483647)
"./librpc/gen_ndr/srvsvc.h", line 242: warning: enumerator value
overflows INT_MAX (2147483647)
"./librpc/gen_ndr/wkssvc.h", line 10: warning: useless declaration
"./librpc/gen_ndr/samr.h", line 633: warning: useless declaration
"./librpc/gen_ndr/nbt.h", line 496: warning: useless declaration
"./librpc/gen_ndr/netlogon.h", line 204: warning: useless declaration
"./librpc/gen_ndr/netlogon.h", line 206: warning: useless declaration
"./librpc/gen_ndr/libnet_join.h", line 10: warning: useless declaration
"libsmb/clikrb5.c", line 242: warning: implicit function declaration:
error_message
"libsmb/clikrb5.c", line 1050: warning: statement not reached
"libsmb/clikrb5.c", line 1531: #error: UNKNOWN_KRB5_ENCTYPE_TO_STRING_FUNCTION
cc: acomp failed for libsmb/clikrb5.c
The following command failed:
cc -I. -I/home/david/build/samba-3.2.0-temp/samba-3.2.0/source
-I/usr/local/heimdal/heimdal-1.1/include -O -D_SAMBA_BUILD_=3
-I/home/david/build/samba-3.2.0-temp/samba-3.2.0/source/iniparser/src
-Iinclude -I./include  -I. -I. -I./lib/replace -I./lib/talloc
-I./lib/tdb/include -I./libaddns -I./librpc -DHAVE_CONFIG_H
-I/usr/local/heimdal/heimdal-1.1/include -D_LARGEFILE_SOURCE
-D_REENTRANT -D_FILE_OFFSET_BITS=64 -Iinclude -I./include -I. -I.
-I./lib/replace -I./lib/talloc -I./lib/tdb/include -I./libaddns
-I./librpc -I./popt -DLDAP_DEPRECATED -DSUNOS5   -I/include
-I/home/david/build/samba-3.2.0-temp/samba-3.2.0/source/lib
-D_SAMBA_BUILD_=3 -KPIC -c libsmb/clikrb5.c -o libsmb/clikrb5.o
*** Error code 2
...
make: Fatal error: Command failed for target `libsmb/clikrb5.o'


B. The immediate cause of the problem:

What's causing the #error: UNKNOWN_KRB5_ENCTYPE_TO_STRING_FUNCTION is
that with Heimdal, configure should be setting
HAVE_KRB5_ENCTYPE_TO_STRING_WITH_KRB5_CONTEXT_ARG in config.h, but it's not:

  $ grep HAVE_KRB5_ENCTYPE_TO_STRING include/config.h
  #define HAVE_KRB5_ENCTYPE_TO_STRING 1
  /* #undef HAVE_KRB5_ENCTYPE_TO_STRING_WITH_KRB5_CONTEXT_ARG */
  /* #undef HAVE_KRB5_ENCTYPE_TO_STRING_WITH_SIZE_T_ARG */


C. The root cause of the problem:

The configure script test for the right form of
krb5_enctype_to_string() is failing because <stdlib.h> isn't #included
in conftest.c, and as a result free is not found:

configure:62117: cc -o conftest -errwarn=%all
-I/usr/local/heimdal/heimdal-1.1/include   -D_LARGEFILE_SOURCE
-D_REENTRANT -D_FILE_OFFSET_BITS=64 -Iinclude -I./include  -I. -I.
-I./lib/replace -I./lib/talloc -I./lib/tdb/include -I./libaddns
-I./librpc -I./popt -I/usr/include -DLDAP_DEPRECATED
-L/usr/local/heimdal/heimdal-1.1/lib  -L/usr/local/lib
-R/usr/local/lib -lthread -L./bin -L/usr/lib conftest.c
-L/usr/local/heimdal/heimdal-1.1/lib
-L/usr/local/heimdal/heimdal-1.1/lib -lgssapi -lheimntlm -lkrb5
-lhx509 -lcom_err -lhcrypto -lasn1 -lroken -ldoor -lresolv -lnsl
-lsocket -mt -lresolv   -lnsl -lsocket  >&5
"conftest.c", line 538: warning: implicit function declaration: free
cc: acomp failed for conftest.c
configure:62120: $? = 2
configure: program exited with status 2
configure: failed program was:
...
| /* end confdefs.h.  */
|
|               #include <krb5.h>
|               int main(void) {
|                       krb5_context context = NULL;
|                       char *str = NULL;
|                       krb5_enctype_to_string(context, 1, &str);
|                       if (str) free (str);
|                       return 0;
|               }


D. The Workaround:

Put "#include <stdlib.h>" after "#include <krb5.h>".    The configure
script will correctly detect and set
HAVE_KRB5_ENCTYPE_TO_STRING_WITH_KRB5_CONTEXT_ARG, and the build will
succeed.

-David


-- 
David Eisner http://cradle.brokenglass.com