WAF 2.x upgrade status

[Alexander Bokovoy]

On to, 28 kesä 2018, Alexander Bokovoy via samba-technical wrote:
> On to, 28 kesä 2018, Noel Power via samba-technical wrote:
> > Hi Alexander
> > 
> > 
> > On 27/06/18 19:29, Andrew Bartlett via samba-technical wrote:
> > > On Wed, 2018-06-27 at 18:14 +0300, Alexander Bokovoy via samba-
> > > technical wrote:
> > >> Hi,
> > >>
> > >> Since February I am working on and off on migrating to a newer WAF
> > >> version. We want to use WAF 2.x as it is compatible with Python 3.
> > >> Over years WAF also evolved and most of infrastructure Samba was
> > >> building upon since WAF 1.5 times got deprecated.
> > >>
> > >> Below is a progress I have to date.
> > >>
> > >> Following gitlab branch contains the current state:
> > >> https://gitlab.com/samba-team/devel/samba/commits/abbra-py3-master-waf
> > >>
> > >> I migrated from WAF 1.8 to 1.9 to 2.0 to 2.0.4 and finally to 2.0.8.
> > >> Autobuild still fails but the code passes configure, builds, and in some
> > >> targets even 'make test'.
> > >>
> > >> Thanks to Thomas Nagy help, I've got past the issue I was banging my
> > >> head against for long: building Heimdal compilers (asn1_compile and
> > >> compile_et).
> > >>
> > >> What's not working:  anything that require newly built python bindings
> > >> in 'make test'. For example, tests fail somehow to find ldb bindings
> > >> that were built just now. I'll look at this tomorrow.
> > does it work on a second make test issue ? I have seen sometimes in the
> > normal build make test does not pick up ldb until the second make test.
> > Unfortunately I don't know what triggers this.
> There seem to be some issue with PYTHONPATH, not sure why. A cannot have
> a second run over the same code in a CI. ;)
> 
> Locally I tried to play with the variables unsuccessfully so far.
Got this fixed by reverting 5bdbe1ac62b334a2890bf3e5a9ca6e47bb371048
partially. There are, of course, more subtle issues, see below.

> 
> > >>
> > >> https://gitlab.com/samba-team/devel/samba/pipelines/24684552 is a latest
> > >> run -- we pass roughly 1/3 of all build jobs.
> > > Thank you so much Alexander for your patience and effort in this area!
> > That's super news, sounds really promising. Unfortunately I am on
> > vacation for 2 weeks from Wed next week (and tomorrow also) so not sure
> > if I will get a chance to try this out before I return. 
> No problem. Andreas wants me to get this done before first 4.9 RC
> anyway, so I need to get full autobuild working during first week of
> July.
I'm now at the point where only two build jobs are failing out of 15:

 - ctdb tests aren't finding cmdline_test binary. Existing code in 'waf
   autotest' in ctdb/wscript attempts to use in-source test script
   instead of a generated one that uses proper path

 - it is not possible to run full autobuild on a system with ldb without
   lmdb. This is reproducible on Gitlab runners as well.

The rest should be fixed in my branch on gitlab:
https://gitlab.com/samba-team/devel/samba/commits/abbra-py3-master-waf

A recent pipeline which demonstrates this branch is
https://gitlab.com/samba-team/devel/samba/pipelines/25021621

The pipeline above has two more failures which are fixed already in the
branch above but as a new pipeline
https://gitlab.com/samba-team/devel/samba/pipelines/25026002 was failing
to start some of jobs at the time of this email was written due to
gitlab/runners issues (error 500 serverside), so I had to restart few
jobs (ctdb), thus the status is split over two pipelines right now.

-- 
/ Alexander Bokovoy