WAF 2.x upgrade for 4.9

[Alexander Bokovoy]

On pe, 06 heinä 2018, Andrew Bartlett wrote:
> On Fri, 2018-07-06 at 09:13 +0300, Alexander Bokovoy wrote:
> > On pe, 06 heinä 2018, Andrew Bartlett wrote:
> > > On Fri, 2018-07-06 at 07:40 +0300, Alexander Bokovoy via samba-
> > > technical wrote:
> > > > On pe, 06 heinä 2018, Gary Lockyer via samba-technical wrote:
> > > > > Can I ask why the urgency to get the WAF upgrade into 4.9.  It's going
> > > > > to b much safer if this gets landed after the 4.9 cut-off.  My concern
> > > > > is that changing the build system this close to a release is going to
> > > > > cause all sorts of unnecessary pain.  Whereas if we wait until after the
> > > > > 4.9 release we'll have 6 months to sort out any issues.
> > > > 
> > > > Python 2 will be dropped by Python upstream in 2020. Postponing a
> > > > release to 2019 means we couldn't sort out distribution-wide Python 3
> > > > migration for Samba and Samba AD targets until first half of 2019. This
> > > > puts Fedora Project (in my case) into a dangerous state.
> > > > 
> > > > Waf upgrade is a first step in cleaning up Python 2 use and replacing it
> > > > with Python 3-compatible code. Unfortunately, a reality is that it takes
> > > > more than just porting Samba to py3 to do that. We need to make sure
> > > > distribution as a whole deals well with that -- for example, while
> > > > FreeIPA is using Python 3 already in Fedora 28 (current release) with
> > > > select Samba Python modules and there we can be sure things work, the
> > > > same couldn't be said for Samba AD path. Right now I couldn't even build
> > > > Samba AD with Python 3 to test whether things would work at all.
> > > 
> > > I don't see how the distribution-wide stuff can come before the Python3
> > > porting of the code is done.  Given that, why is the build-system step
> > > urgent?  
> > 
> > It is extremely hard to maintain large patchsets which touch every
> > aspect of Samba build off-tree.
> 
> Sure, which is why I urged this be merged soon.  There are still a few
> things to sort out (not many thankfully).  Could it be done on Friday,
> just after the freeze and branch?
> 
> > > 
> > > > Note I'm talking about use outside of git master as we need to have
> > > > supporting libraries packaged with upgraded waf. We have ~2 months until
> > > > 4.9 release, I think it is sufficient time to sort out waf issues in
> > > > prereleases given the analysis Andrew did to the resulting generated
> > > > config files.
> > > 
> > > The concern is more that in the next 6 days, many developers will be
> > > trying to land patches into master for the 4.9 freeze.  
> > > 
> > > So we don't have 2 months, we have 6 busy days.  I've seen how grumpy
> > > folks get when their autobuilds stop working, and I wouldn't advise
> > > being on the pointy end of that wrath. 
> > 
> > I don't mind taking responsibility for bugs where it is due.
> 
> I've been at the pointy end of 'you broke the autobuild', I wouldn't
> recommend it. 
> 
> > Are you saying I'm now blocked from even merging this to master?
> 
> I think a merge after Samba 4.9rc1 on Friday would be the best
> timeframe. 
> 
> > > This isn't the absolute last step before a distribution package builds
> > > and operates with python3 is it?
> > 
> > Finding integration bugs is much harder when there is no base for
> > integration at all.
> 
> Sure, but I'm still lost as to how this fits with a 4.9 release.  Can
> you explain that bit some more?  It seems I'm missing something here,
> because otherwise you wouldn't be at such urgency to have it in 4.9. 
> Is there some other external requirement?
I need to get to a point where a full build can be done without Python 2
as a PYTHON interpreter. Not being able to reduce a patchset to do so
down is a huge liability. Fedora 29 final freeze is around a month after
we are going to release final 4.9.0.

> You say you want to get the dependent libs sorted.  Could we just make
> extra releases of those after 4.9 freezes?
WAF versions cannot be mixed. If we have third_party/waf updated, all
the dependent things in buildtools/ will need to correspond. Given
amount of changes, lib/*/wscript files need to change as well, not just
in Samba tree.

> Can tarballs and packages be made for testing of 4.10pre1 and testing
> done of that?  Given the distribution build can't be done with python3
> (because python2 packages need to be produced, and the python3 packages
> don't work), how does the new waf version change things?
Running Python 3-based waf build is no different from running Python
2-based waf build -- we do produce both py2 and py3 packages right now,
so after incompatibilities are fixed in waf build files, we just should
get away with reversing what PYTHON and --extra-python defines are. This
doesn't mean I would't need Python 2 to run Samba AD in Fedora 29 but it
allows to get closer to getting rid of Python 2 in a base build root.

I realise this may look minor to you. We are working constantly on
improving python upgrade from release to release, F28 was a skip from
Samba side because we weren't able to complete waf migration in time. If
I wouldn't be able to get this improvement with F29, this means
earliest I could do so is Spring 2019 and that is becoming too close
(two Fedora releases doesn't mean we have a year, rather like 7 months)
to Python 2 orphaning and removal (slated for Fedora 32).

-- 
/ Alexander Bokovoy