WAF 2.x upgrade status

[Amitay Isaacs]

On Tue, Jul 3, 2018 at 9:10 PM, Alexander Bokovoy via samba-technical
<samba-technical at lists.samba.org> wrote:
> On to, 28 kesä 2018, Alexander Bokovoy via samba-technical wrote:
>> On to, 28 kesä 2018, Noel Power via samba-technical wrote:
>> > Hi Alexander
>> >
>> >
>> > On 27/06/18 19:29, Andrew Bartlett via samba-technical wrote:
>> > > On Wed, 2018-06-27 at 18:14 +0300, Alexander Bokovoy via samba-
>> > > technical wrote:
>> > >> Hi,
>> > >>
>> > >> Since February I am working on and off on migrating to a newer WAF
>> > >> version. We want to use WAF 2.x as it is compatible with Python 3.
>> > >> Over years WAF also evolved and most of infrastructure Samba was
>> > >> building upon since WAF 1.5 times got deprecated.
>> > >>
>> > >> Below is a progress I have to date.
>> > >>
>> > >> Following gitlab branch contains the current state:
>> > >> https://gitlab.com/samba-team/devel/samba/commits/abbra-py3-master-waf
>> > >>
>> > >> I migrated from WAF 1.8 to 1.9 to 2.0 to 2.0.4 and finally to 2.0.8.
>> > >> Autobuild still fails but the code passes configure, builds, and in some
>> > >> targets even 'make test'.
>> > >>
>> > >> Thanks to Thomas Nagy help, I've got past the issue I was banging my
>> > >> head against for long: building Heimdal compilers (asn1_compile and
>> > >> compile_et).
>> > >>
>> > >> What's not working:  anything that require newly built python bindings
>> > >> in 'make test'. For example, tests fail somehow to find ldb bindings
>> > >> that were built just now. I'll look at this tomorrow.
>> > does it work on a second make test issue ? I have seen sometimes in the
>> > normal build make test does not pick up ldb until the second make test.
>> > Unfortunately I don't know what triggers this.
>> There seem to be some issue with PYTHONPATH, not sure why. A cannot have
>> a second run over the same code in a CI. ;)
>>
>> Locally I tried to play with the variables unsuccessfully so far.
> Got this fixed by reverting 5bdbe1ac62b334a2890bf3e5a9ca6e47bb371048
> partially. There are, of course, more subtle issues, see below.
>
>>
>> > >>
>> > >> https://gitlab.com/samba-team/devel/samba/pipelines/24684552 is a latest
>> > >> run -- we pass roughly 1/3 of all build jobs.
>> > > Thank you so much Alexander for your patience and effort in this area!
>> > That's super news, sounds really promising. Unfortunately I am on
>> > vacation for 2 weeks from Wed next week (and tomorrow also) so not sure
>> > if I will get a chance to try this out before I return.
>> No problem. Andreas wants me to get this done before first 4.9 RC
>> anyway, so I need to get full autobuild working during first week of
>> July.
> I'm now at the point where only two build jobs are failing out of 15:
>
>  - ctdb tests aren't finding cmdline_test binary. Existing code in 'waf
>    autotest' in ctdb/wscript attempts to use in-source test script
>    instead of a generated one that uses proper path

The tests are supposed to run from the source tree.  So it must use
the in-tree test script.  Can't think of why that should break with
the change in waf.  Unless waf 2.0 uses paths for scripts relative to
bin/default instead of the source dir.

Will take a look at it tomorrow.

>  - it is not possible to run full autobuild on a system with ldb without
>    lmdb. This is reproducible on Gitlab runners as well.
>
> The rest should be fixed in my branch on gitlab:
> https://gitlab.com/samba-team/devel/samba/commits/abbra-py3-master-waf
>
> A recent pipeline which demonstrates this branch is
> https://gitlab.com/samba-team/devel/samba/pipelines/25021621
>
> The pipeline above has two more failures which are fixed already in the
> branch above but as a new pipeline
> https://gitlab.com/samba-team/devel/samba/pipelines/25026002 was failing
> to start some of jobs at the time of this email was written due to
> gitlab/runners issues (error 500 serverside), so I had to restart few
> jobs (ctdb), thus the status is split over two pipelines right now.
>
> --
> / Alexander Bokovoy
>

Amitay.