waf/autoconf and the SYSTEM krb5 support matrix for Samba 4.0

[Andrew Bartlett]

On Thu, 2012-05-31 at 16:20 +0200, Michael Adam wrote:
> Hi Andrew,

> Thanks for that summary. This is very helpful!
> 
> Let's see what the implications are (for the pure s3 file server
> build, something that is certainly used out there and will be for
> a while):
> 
> The basis for the problems that we discuss is this:
> 
> >>>>> There are systems out there that neither have a recent <<<<<
> >>>>> (enough) version of kerberos nor python available.     <<<<<
> 
> For all other systems, we can somehow work around with the above
> matrix (without the need to fiddle with the system-installed
> libs and so on), given one is willing to accept some compromise:
> 
>   * On a system with too old kerberos but appropriate python
>     available (e.g. SLES11), we can use the waf build to produce
>     a samba build using shipped heimdal which should work.
> 
>     The same applies to FreeBSD 9 in principle, but it has been
>     reported that the waf build had problems (build failures)
>     there that did not show up on recent linux platforms, so
>     the hurdle might be higher on non-Linux systems.

I remain confident that we can keep FreeBSD as a fully supported
platform for Samba, using waf.  I and others have put a lot of effort
into the recently raised issues - I have a FreeBSD 8.2 VM and I do
regularly use it to diagnose FreeBSD specific issues. 

There are two FreeBSD-specific issues I'm aware of:
 - Samba4 as a DC will not provision until the FreeBSD xattr wrappers
are put into libreplace. 
 - Developers must use --abi-check-disable because the gdb on FreeBSD
does not escape as many characters as the reference output from Linux.

>     This assumes that one is ok with using the waf build!...
> 
>     BTW: We should make sure that the configure runs have the
>     same results under waf and autoconf. I know that Andrew and
>     Andreas (and others) have put some effort into this in the
>     past, but maybe we should make this scriptable, so that
>     we have a (top lvl make) test facility to compare the results
>     (say config.h) of waf and autoconf configure. This is
>     actually imho a prerequisite for establishing the waf build
>     as a replacement for autoconf for the pure fileserver
>     installation.

Here we certainly have some tools we can apply.  

 - buildtools/compare_config_h3.sh: I've added automatic invocation of
this script to the autoconf build when building smbtorture4.  This is
the output so far:

#define HAVE_BUILTIN_HAVE_ISBLANK 1
#define HAVE_DECL_RL_EVENT_HOOK 0
#define HAVE_LDAP 1
#define HAVE_LDAP_ADD_RESULT_ENTRY 1
#define HAVE_LDAP_INIT 1
#define HAVE_LDAP_INITIALIZE 1
#define HAVE_LDAP_SASL_WRAPPING 1
#define HAVE_LDAP_SET_REBIND_PROC 1
#define HAVE_LIBLDAP 1
#define HAVE_LINUX_XFS_QUOTAS 1
#define HAVE_MREMAP 1
#define HAVE_NO_AIO 1
#define HAVE_QUOTACTL_LINUX 1
#define HAVE_RES_NINIT 1

 - I did an analysis of the different config.h files late last year, due
to the same concerns:
https://lists.samba.org/archive/samba-technical/2011-November/080568.html and this lists a number of tests that should be investigated.  Additional to that I have noticed we need to bring across the ACL header tests. 

 - Additionally, despite the appearance, the two sets of configure
checks actually do have a common heritage.  The waf tests are derived
from the autoconf tests, so the primary divergence (as you see above) is
in changes made since waf was introduced.  (So doing a git log on
configure.in would give a good set of candidates to spot-check).

>   * On a system with recent krb5, one can use the autoconf build
>     (if one does not want to or can not use the waf build).
> 
> So how can we make it possible or more easy for those other
> systems without recent python and krb5, to get to the goal
> of a full samba(3) build?
> 
> Possible internal solution:
> 
>   * Add support for building and using shipped heimdal from
>     the autoconf build.
>     This would be an internal change that would work on any
>     system without the need to change system installed
>     libs/programs.
> 
>     It would make it easier for people who for some reason
>     can not or do not want to use the waf build to still
>     be able to build a recent file server.

If we don't need to make it internal, we could simply have the 'krb5
libs required' message we agreed on earlier refer users to waf or a
recent version of Heimdal (or MIT) and have them install it in a custom
prefix.  Both have krb5 systems have autoconf based build systems. 

> If the waf build is an option but python support is lacking:
> 
>   * Can we make it very easy to build an appropriate python?
> 
>     I assume that the python code is too big to ship along
>     samba, but can we provide mechanisms (scripts/makefiles)
>     that allow for a very easy building of python, e.g. using
>     a python tarball url on the web or a local path where
>     the builder has dropped an appropriate tarball and then
>     build a local version of python only to be used for the
>     waf build?
> 
>     I think something like this was already done by
>     Tridge(?) and others for the build farm. Can we integrate
>     such a mechanism as an optional step in our build process?

We actually already have this, the install_with_python.sh script.  Would
it be helpful if we detected python as being unavailable in ./configure
and mentioned this script?

> So the fundamental questions/proposals are these:
> 
> 1) Can we force alignment of the configure results
>    between autoconf and waf build (s3-part) or at least
>    provide a mechanism to chech these?

With the whitelist for known/expected differences, this check will soon
be part of the autoconf build when recursing into waf.  We can then grab
these results from the build farm returns.

> 2) Can we add internal heimdal support to the autoconf build?
> 
> 3) Can we add building (if not shipping) of appropriate pyhton
>    as an option to the top level build?

This has always been my preferred option, and one that I'm happy to make easier.

Thank you very much for your time and thoughts on this.  It really is
helpful to tease out these broader issues into specific ideas we can
discuss and specific proposals we can work on. 

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org