heimdal headers ordering (gssapi/gssapi.h, crypto-headers.h)

[Stefan Metzmacher]

Hi,

are your using 'waf build' directly? This that I saw the same
problem in the past. Please use 'make' and 'make install',
the difference is that it passes PYTHONHASHSEED=1, without that
the order is random.

metze

Am 28.03.22 um 09:14 schrieb Michael Tokarev via samba-technical:
> In the other email, we wrote:
> 
>  >> First, the includes.  For example, while compiling
>  >> lib/krb5_wrap/gss_samba.c ,
>  >> the include-path includes -Ithird_party/heimdal/lib
>  >> -Ithird_party/heimdal/lib/gssapi .
>  >>
>  >> gss_samba.c #includes gss_samba.h which includes
>  >> lib/replace/system/gssapi.h,
>  >> which - based on HAVE_GSSAPI_GSSAPI_H, includes
>  >> <gssapi/gssapi.h>.  The first
>  >> include path which has gssapi/gssapi.h is third_party/heimdal/lib,
>  >> so we include
>  >> third_party/heimdal/lib/gssapi/gssapi.h. But this is a simple
>  >> dispatcher file, it merely includes <gssapi/gssapi.h>.
>  >> Which, as we know already, is
>  >> third_party/heimdal/lib/gssapi/gssapi.h. So we end up
>  >>  without all the gssapi definitions altogether.
>  >> The correct file to include for <gssapi/gssapi.h>
>  >> is third_party/heimdal/lib/gssapi/gssapi/gssapi.h (note the double,
>  >> or even
>  >> triple, gssapi in there) - so the _second_ -I path should be used
>  >> from the
>  >> above.  This is quite messy and not really reliable.
>  >
>  > Again, like symbols, header include paths are dependent on the 'deps='
>  > of the subsystems.  So a subsystem is missing a dependency.  We have
>  > had reports (to Heimdal, frustratingly) about this, but someone needs
>  > to chase it down.
>  >
>  > See https://bugzilla.samba.org/show_bug.cgi?id=15033
> 
> Too bad I didn't know about this when chasing this issue here :)
> But it is good anyway to know there's someone else who hit it too ;)
> LOL :)
> 
> Okay.
> 
> The thing here is with include path ordering. And with timing issues,
> it seems.
> 
> I have logs from two consequtive builds, one of which failed (bad),
> and another succeeded (good) without any changes in between (the
> same source, the same packages installed from debian since I haven't
> run apt update, - everything is the same).
> 
> We should list heimdal_build/include *before* heimdal/include in
> all cases. Yet the order is random, non-deterministic:
> 
> --- good0    2022-03-28 09:59:07.502426113 +0300
> +++ bad0    2022-03-28 09:59:01.606521064 +0300
> @@ -2,0 +3 @@
> +-Ithird_party/heimdal/lib/krb5
> @@ -3,0 +5 @@
> +-Ithird_party/heimdal/include
> @@ -4,0 +7,2 @@
> +-Ithird_party/heimdal/lib/roken
> +-Ithird_party/heimdal_build/include
> @@ -7 +10,0 @@
> --Ithird_party/heimdal/lib/krb5
> @@ -9 +11,0 @@
> --Ithird_party/heimdal_build/include
> @@ -12,4 +14,2 @@
> --Ithird_party/heimdal/lib/roken
> --Ithird_party/heimdal/include
> --Ithird_party/heimdal/lib/hcrypto
> --Ithird_party/heimdal/lib
> +-Ithird_party/heimdal/lib/hcrypto/libtommath
> +-Ithird_party/heimdal/lib/wind
> @@ -20 +19,0 @@
> --Ithird_party/heimdal/lib/hcrypto/libtommath
> @@ -22 +21,2 @@
> --Ithird_party/heimdal/lib/hdb
> +-Ithird_party/heimdal/lib/hcrypto
> +-Ithird_party/heimdal/lib
> @@ -24 +24 @@
> --Ithird_party/heimdal/lib/wind
> +-Ithird_party/heimdal/lib/hdb
> 
> There, I took one command line (compiling source4/kdc/wdc-samba4.c),
> split it into separate lines, saved into two files, filtered
> everything related to heimdal, and diffed.  In the bad case,
> heimdal/include is moved before heimdal_build/include.
> And this break stuff for the reasons already mentioned above.
> 
> So we do have obvious ordering instability.
> 
> Besides this, we have one more very annoying issue. Namely, it
> looks like "next" waf steps always include all previous steps,
> complete or in parts.  In particular, `waf install' *always*
> rebuilds about 6k files even if nothing at all had changed
> in between. While `waf build' builds only about 4k files.
> 
> Why it is relevant in this context.
> 
> In debian, waf configure step is done with -j1, to avoid
> parallel/ordering issues.  However, waf build and waf install
> are done with default parallelism. But since `waf install'
> re-does some of `waf configure' steps, but this time with
> parallelism, this may lead to different results with the
> ordering as we see with heimdal includes.
> 
> So it looks like we have 2 or 3 separate issues here:
> the always-rebuild and install-builds-more-than-build,
> and include ordering depending on parallelism.
> 
> I'll try to figure out what's going on, but these things
> definitely require good understanding of waf internals
> and machinery.
> 
> The two build logs are available, just in case, at
> http://www.corpit.ru/mjt/tmp/samba-heimdal-include-ordering.tar.gz
> 
> Thanks,
> 
> /mjt
>