WAF 2.x upgrade for 4.9

Fri Jul 6 10:00:42 UTC 2018

On Fri, 2018-07-06 at 12:08 +0300, Alexander Bokovoy wrote:
> On pe, 06 heinä 2018, Andrew Bartlett wrote:
> > 
> > Sure, but I'm still lost as to how this fits with a 4.9 release.  Can
> > you explain that bit some more?  It seems I'm missing something here,
> > because otherwise you wouldn't be at such urgency to have it in 4.9. 
> > Is there some other external requirement?
> 
> I need to get to a point where a full build can be done without Python 2
> as a PYTHON interpreter. Not being able to reduce a patchset to do so
> down is a huge liability. Fedora 29 final freeze is around a month after
> we are going to release final 4.9.0.

Good.  This is an end goal we all share. 

To be clear, because perhaps this is what I've been missing:  are you
hoping Noel et all will finish the Python3 patches during the RC period
and we will (you you will in the packages) backport them all before
Fedora 29 freezes?

> > You say you want to get the dependent libs sorted.  Could we just make
> > extra releases of those after 4.9 freezes?
> 
> WAF versions cannot be mixed. If we have third_party/waf updated, all
> the dependent things in buildtools/ will need to correspond. Given
> amount of changes, lib/*/wscript files need to change as well, not just
> in Samba tree.

Sure, for a given branch (which has one set of
samba/tdb/ldb/talloc/tevent in it) that is true. 

But you can build Samba 4.9 against the talloc from master, and once
they are in independent releases and tarballs the waf versions can
diverge.  So you can build those with python3 and waf 2.x once that is
mature in master, even for the Samba 4.9 release you make and build
with python2 and waf 1.x.

I wouldn't suggest doing it with LDB (really, don't do that), but the
value is in the demonstration, going though the full build process etc.

That can happen in master with a new talloc (eg 2.2.0, to allow
patching in 4.9 if needed) and tdb (eg 1.4.0) package tagged and
released, in just ten days time, for example.

> > Can tarballs and packages be made for testing of 4.10pre1 and testing
> > done of that?  Given the distribution build can't be done with python3
> > (because python2 packages need to be produced, and the python3 packages
> > don't work), how does the new waf version change things?
> 
> Running Python 3-based waf build is no different from running Python
> 2-based waf build -- we do produce both py2 and py3 packages right now,
> so after incompatibilities are fixed in waf build files, we just should
> get away with reversing what PYTHON and --extra-python defines are. 

It isn't that simple.  For example make test strictly assumes that
py3_compatible means run with extra_python.  In talloc and tdb both
only look for PYTHON_SO_ABI_FLAG in the extra_python case.

While these are 'only' bugs, it works against your case that something
truly special happens if these patches are merged.

> This
> doesn't mean I would't need Python 2 to run Samba AD in Fedora 29 but it
> allows to get closer to getting rid of Python 2 in a base build root.

I'm confused, how does it change what is in the build root if you still
need it to produce the packages?

Anyway, it doesn't even start to work (on your gitlab branch):
[abartlet at jesse samba]$ PYTHON=/usr/bin/python3 ./configure && make -j
  File "./buildtools/bin/waf", line 136
    return [os.path.abspath(os.path.join(os.path.dirname(__file__), x))
for x in paths]

               ^
TabError: inconsistent use of tabs and spaces in indentation

> I realise this may look minor to you. We are working constantly on
> improving python upgrade from release to release, F28 was a skip from
> Samba side because we weren't able to complete waf migration in time. If
> I wouldn't be able to get this improvement with F29, this means
> earliest I could do so is Spring 2019 and that is becoming too close
> (two Fedora releases doesn't mean we have a year, rather like 7 months)
> to Python 2 orphaning and removal (slated for Fedora 32).

Perhaps I can ask for something more specific.  Can you list the Samba
build commands (I've explained above how to handle this for tdb and
talloc) that will work with your branch but won't in master, that are
useful for a distribution package?

All I can think of is (after the above is fixed):

PYTHON=/usr/bin/python3 ./configure --without-python

and perhaps 

PYTHON=/usr/bin/python3 ./configure --without-ad-dc

But given Fedora ships Python bindings and the AD DC, how does rushing
this into 4.9 help?  What am I missing?

I'm as excited as you to get this work in, and every step does help.  I
was all gung-ho to get this in till Gary spoke up with his years of
experience to say 'please don't do that now'.  So what I'm trying to
tease out from you is the answer to that question: 

"why the urgency to get the WAF upgrade into 4.9"?

Alexander,

What I'm saying is you have sadly left this run just a bit too late. 
There are still loose ends and there isn't the compelling outcome that
would justify the disruption in the most stressful few days of the team
calendar. 

We all try and get just one more thing in, and now you know why I get
grumpy when major features get held up 'with plenty of time before the
release', because we all see how that time slips away.  But the last
six days are crazy, I really don't think this should land this during
them.

Sorry,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba