WAF 2.x upgrade for 4.9

[Alexander Bokovoy]

On pe, 06 heinä 2018, Andrew Bartlett wrote:
> On Fri, 2018-07-06 at 12:08 +0300, Alexander Bokovoy wrote:
> > On pe, 06 heinä 2018, Andrew Bartlett wrote:
> > > 
> > > Sure, but I'm still lost as to how this fits with a 4.9 release.  Can
> > > you explain that bit some more?  It seems I'm missing something here,
> > > because otherwise you wouldn't be at such urgency to have it in 4.9. 
> > > Is there some other external requirement?
> > 
> > I need to get to a point where a full build can be done without Python 2
> > as a PYTHON interpreter. Not being able to reduce a patchset to do so
> > down is a huge liability. Fedora 29 final freeze is around a month after
> > we are going to release final 4.9.0.
> 
> Good.  This is an end goal we all share. 
> 
> To be clear, because perhaps this is what I've been missing:  are you
> hoping Noel et all will finish the Python3 patches during the RC period
> and we will (you you will in the packages) backport them all before
> Fedora 29 freezes?
We both want to get Python 3 native build a reality faster than later.
Noel is on vacation right now but he was running with my changes in past
so reconciling our work is a bit easier.

> > > You say you want to get the dependent libs sorted.  Could we just make
> > > extra releases of those after 4.9 freezes?
> > 
> > WAF versions cannot be mixed. If we have third_party/waf updated, all
> > the dependent things in buildtools/ will need to correspond. Given
> > amount of changes, lib/*/wscript files need to change as well, not just
> > in Samba tree.
> 
> Sure, for a given branch (which has one set of
> samba/tdb/ldb/talloc/tevent in it) that is true. 
> 
> But you can build Samba 4.9 against the talloc from master, and once
> they are in independent releases and tarballs the waf versions can
> diverge.  So you can build those with python3 and waf 2.x once that is
> mature in master, even for the Samba 4.9 release you make and build
> with python2 and waf 1.x.
> 
> I wouldn't suggest doing it with LDB (really, don't do that), but the
> value is in the demonstration, going though the full build process etc.
So doing that with ldb is much harder and just confirms my point (as
well as yours).

> That can happen in master with a new talloc (eg 2.2.0, to allow
> patching in 4.9 if needed) and tdb (eg 1.4.0) package tagged and
> released, in just ten days time, for example.
Looking forward to that.

> > > Can tarballs and packages be made for testing of 4.10pre1 and testing
> > > done of that?  Given the distribution build can't be done with python3
> > > (because python2 packages need to be produced, and the python3 packages
> > > don't work), how does the new waf version change things?
> > 
> > Running Python 3-based waf build is no different from running Python
> > 2-based waf build -- we do produce both py2 and py3 packages right now,
> > so after incompatibilities are fixed in waf build files, we just should
> > get away with reversing what PYTHON and --extra-python defines are. 
> 
> It isn't that simple.  For example make test strictly assumes that
> py3_compatible means run with extra_python.  In talloc and tdb both
> only look for PYTHON_SO_ABI_FLAG in the extra_python case.
> 
> While these are 'only' bugs, it works against your case that something
> truly special happens if these patches are merged.
Can you give me more of those details you know about? We have no
recording of those anywhere, I think.

> > This
> > doesn't mean I would't need Python 2 to run Samba AD in Fedora 29 but it
> > allows to get closer to getting rid of Python 2 in a base build root.
> 
> I'm confused, how does it change what is in the build root if you still
> need it to produce the packages?
> 
> Anyway, it doesn't even start to work (on your gitlab branch):
> [abartlet at jesse samba]$ PYTHON=/usr/bin/python3 ./configure && make -j
>   File "./buildtools/bin/waf", line 136
>     return [os.path.abspath(os.path.join(os.path.dirname(__file__), x))
> for x in paths]
>                                                                        
>                ^
> TabError: inconsistent use of tabs and spaces in indentation
This is a next step to work on. However, it will be easier without 80+
patches on top to maintain separately, given that the same files will be
touched again and again.

> > I realise this may look minor to you. We are working constantly on
> > improving python upgrade from release to release, F28 was a skip from
> > Samba side because we weren't able to complete waf migration in time. If
> > I wouldn't be able to get this improvement with F29, this means
> > earliest I could do so is Spring 2019 and that is becoming too close
> > (two Fedora releases doesn't mean we have a year, rather like 7 months)
> > to Python 2 orphaning and removal (slated for Fedora 32).
> 
> Perhaps I can ask for something more specific.  Can you list the Samba
> build commands (I've explained above how to handle this for tdb and
> talloc) that will work with your branch but won't in master, that are
> useful for a distribution package?
> 
> All I can think of is (after the above is fixed):
> 
> PYTHON=/usr/bin/python3 ./configure --without-python
> 
> and perhaps 
> 
> PYTHON=/usr/bin/python3 ./configure --without-ad-dc
> 
> But given Fedora ships Python bindings and the AD DC, how does rushing
> this into 4.9 help?  What am I missing?
> 
> I'm as excited as you to get this work in, and every step does help.  I
> was all gung-ho to get this in till Gary spoke up with his years of
> experience to say 'please don't do that now'.  So what I'm trying to
> tease out from you is the answer to that question: 
> 
> "why the urgency to get the WAF upgrade into 4.9"?
I feel we are going in circles.

> Alexander,
> 
> What I'm saying is you have sadly left this run just a bit too late. 
> There are still loose ends and there isn't the compelling outcome that
> would justify the disruption in the most stressful few days of the team
> calendar. 
> 
> We all try and get just one more thing in, and now you know why I get
> grumpy when major features get held up 'with plenty of time before the
> release', because we all see how that time slips away.  But the last
> six days are crazy, I really don't think this should land this during
> them.
You are correct in that it might be a problem. Perhaps, I instead should
take a vacation and forget about doing this work until the end of 4.9
release time frame.

Sorry for my frustration.
-- 
/ Alexander Bokovoy