waf, PYTHONHASHSEED & -I order on other architectures

[Thomas Nagy]

On Mon, 09 May 2022 08:53:12 +1200, Andrew Bartlett via samba-technical <samba-technical at lists.samba.org> wrote:

> On Sat, 2022-04-09 at 09:41 +0300, Michael Tokarev via samba-technical
> wrote:
> > Hi!
> > 
> > I'm having a build failure of samba on sparc64.  It fails due to
> > finding
> > wrong include for <gssapi/gssapi.h>, as it has already seen before
> > due to
> > PYTHONHASHSEED not being set, having python hashes in random order so
> > -I
> > includes were unpredictable.
> > 
> > The good -I order is this:
> > 
> >   -Ithird_party/heimdal/lib/gssapi
> >   -Ithird_party/heimdal/lib
> > 
> > the bad is:
> > 
> >   -Ithird_party/heimdal/lib
> >   -Ithird_party/heimdal/lib/gssapi
> > 
> > (I picked up only the -I options for dirs where <gssapi/gssapi.h>
> > exists).
> > 
> > This is stable on sparc64, all builds of samba-4.16 are failing due
> > to
> > this very issue.
> > 
> > How to work around this?
> 
> I agree that at this point there is a bug, but nobody has had the time
> to work out the missing dependencies or get into the guts of waf to get
> a more proper solution. 

About the guts of waf:

When changing the PYTHONHASHSEED value in the Makefile and adding traces (--zones=deps):
"""
WAF_BINARY=$(PYTHON) ./buildtools/bin/waf --zones=deps
WAF=PYTHONHASHSEED=20 WAF_MAKE=1 $(WAF_BINARY)
"""

A full rebuild is observed here. The library link order changes as reflected by the traces below:

with PYTHONHASHSEED=10: deps computed dependencies for target ldb_ildap: uselib=[] uselib_local=['talloc', 'ldb', 'tevent', 'samdb', 'cli-ldap', 'samba-security', 'samba-credentials'] add_objects=['ldb_ildap.objlist']

with PYTHONHASHSEED=20: deps computed dependencies for target ldb_ildap: uselib=[] uselib_local=['tevent', 'samba-security', 'cli-ldap', 'samba-credentials', 'talloc', 'ldb', 'samdb'] add_objects=['ldb_ildap.objlist']

They take their origin in the file buildtools/wafsamba/samba_deps.py before reaching the Waf parts. That file contains a significant amount of non-trivial set() operations.

Even if a hash seed is provided, the link order may change with a new Python version or on a different operating system.

Thomas