WAF 2.x upgrade status

Thu Jul 5 08:21:31 UTC 2018

On Thu, Jul 5, 2018 at 6:16 PM, Alexander Bokovoy <ab at samba.org> wrote:
> On to, 05 heinä 2018, Amitay Isaacs wrote:
>> On Wed, Jul 4, 2018 at 10:14 PM, Alexander Bokovoy <ab at samba.org> wrote:
>> > On ke, 04 heinä 2018, Alexander Bokovoy via samba-technical wrote:
>> >> On ke, 04 heinä 2018, Amitay Isaacs wrote:
>> >> > On Tue, Jul 3, 2018 at 9:10 PM, Alexander Bokovoy via samba-technical
>> >> > <samba-technical at lists.samba.org> wrote:
>> >> > > On to, 28 kesä 2018, Alexander Bokovoy via samba-technical wrote:
>> >> > >> On to, 28 kesä 2018, Noel Power via samba-technical wrote:
>> >> > >> > Hi Alexander
>> >> > >> >
>> >> > >> >
>> >> > >> > On 27/06/18 19:29, Andrew Bartlett via samba-technical wrote:
>> >> > >> > > On Wed, 2018-06-27 at 18:14 +0300, Alexander Bokovoy via samba-
>> >> > >> > > technical wrote:
>> >> > >> > >> Hi,
>> >> > >> > >>
>> >> > >> > >> Since February I am working on and off on migrating to a newer WAF
>> >> > >> > >> version. We want to use WAF 2.x as it is compatible with Python 3.
>> >> > >> > >> Over years WAF also evolved and most of infrastructure Samba was
>> >> > >> > >> building upon since WAF 1.5 times got deprecated.
>> >> > >> > >>
>> >> > >> > >> Below is a progress I have to date.
>> >> > >> > >>
>> >> > >> > >> Following gitlab branch contains the current state:
>> >> > >> > >> https://gitlab.com/samba-team/devel/samba/commits/abbra-py3-master-waf
>> >> > >> > >>
>> >> > >> > >> I migrated from WAF 1.8 to 1.9 to 2.0 to 2.0.4 and finally to 2.0.8.
>> >> > >> > >> Autobuild still fails but the code passes configure, builds, and in some
>> >> > >> > >> targets even 'make test'.
>> >> > >> > >>
>> >> > >> > >> Thanks to Thomas Nagy help, I've got past the issue I was banging my
>> >> > >> > >> head against for long: building Heimdal compilers (asn1_compile and
>> >> > >> > >> compile_et).
>> >> > >> > >>
>> >> > >> > >> What's not working:  anything that require newly built python bindings
>> >> > >> > >> in 'make test'. For example, tests fail somehow to find ldb bindings
>> >> > >> > >> that were built just now. I'll look at this tomorrow.
>> >> > >> > does it work on a second make test issue ? I have seen sometimes in the
>> >> > >> > normal build make test does not pick up ldb until the second make test.
>> >> > >> > Unfortunately I don't know what triggers this.
>> >> > >> There seem to be some issue with PYTHONPATH, not sure why. A cannot have
>> >> > >> a second run over the same code in a CI. ;)
>> >> > >>
>> >> > >> Locally I tried to play with the variables unsuccessfully so far.
>> >> > > Got this fixed by reverting 5bdbe1ac62b334a2890bf3e5a9ca6e47bb371048
>> >> > > partially. There are, of course, more subtle issues, see below.
>> >> > >
>> >> > >>
>> >> > >> > >>
>> >> > >> > >> https://gitlab.com/samba-team/devel/samba/pipelines/24684552 is a latest
>> >> > >> > >> run -- we pass roughly 1/3 of all build jobs.
>> >> > >> > > Thank you so much Alexander for your patience and effort in this area!
>> >> > >> > That's super news, sounds really promising. Unfortunately I am on
>> >> > >> > vacation for 2 weeks from Wed next week (and tomorrow also) so not sure
>> >> > >> > if I will get a chance to try this out before I return.
>> >> > >> No problem. Andreas wants me to get this done before first 4.9 RC
>> >> > >> anyway, so I need to get full autobuild working during first week of
>> >> > >> July.
>> >> > > I'm now at the point where only two build jobs are failing out of 15:
>> >> > >
>> >> > >  - ctdb tests aren't finding cmdline_test binary. Existing code in 'waf
>> >> > >    autotest' in ctdb/wscript attempts to use in-source test script
>> >> > >    instead of a generated one that uses proper path
>> >> >
>> >> > The tests are supposed to run from the source tree.  So it must use
>> >> > the in-tree test script.  Can't think of why that should break with
>> >> > the change in waf.  Unless waf 2.0 uses paths for scripts relative to
>> >> > bin/default instead of the source dir.
>> >> >
>> >> > Will take a look at it tomorrow.
>> >> I fixed this by properly promoting a build environment.
>> >> So far I have all but one or two tests passing. Those two tests are
>> >> different for Gitlab runner and my local autobuild run though. See
>> >> my response to Andrew in this thread.
>> > Here is my current pipeline: https://gitlab.com/samba-team/devel/samba/pipelines/25108846
>> > https://gitlab.com/samba-team/devel/samba/-/jobs/79336115 shows a
>> > failure in ctdb test:
>> >
>> > --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--
>> > Running test /tmp/samba-testbase/b17/prefix/ctdb/share/ctdb/tests/onnode/0001.sh (11:32:28)
>> > --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--
>> > 0001.sh onnode all hostname - all nodes OK
>> > --------------------------------------------------
>> > Output (Exit status: 127):
>> > --------------------------------------------------
>> > /tmp/samba-testbase/b17/prefix/ctdb/share/ctdb/tests/onnode/0001.sh: 62: /tmp/samba-testbase/b17/prefix/ctdb/share/ctdb/tests/onnode/0001.sh: onnode: not found
>> > --------------------------------------------------
>> > Required output (Exit status: 0):
>> > --------------------------------------------------
>> >
>> >>> NODE: 192.168.1.101 <<
>> > -n 192.168.1.101 hostname
>> >
>> >>> NODE: 192.168.1.102 <<
>> > -n 192.168.1.102 hostname
>> >
>> >>> NODE: 192.168.1.103 <<
>> > -n 192.168.1.103 hostname
>> >
>> >>> NODE: 192.168.1.104 <<
>> > -n 192.168.1.104 hostname
>> > --------------------------------------------------
>> > CTDB_BASE="/tmp/samba-testbase/b17/tmp/tmp.8dqr44XjX3/unit_onnode/etc-ctdb"
>> > ctdb client is /tmp/samba-testbase/b17/prefix/ctdb/share/ctdb/tests/onnode/stubs/ctdb
>> > --------------------------------------------------
>> >
>> > FAILED
>> > ==========================================================================
>> > TEST FAILED: /tmp/samba-testbase/b17/prefix/ctdb/share/ctdb/tests/onnode/0001.sh (status 1) (duration: 0s)
>> > ==========================================================================
>> >
>> > 'onnode' is not found because there is no onnode in PATH. It is in
>> > $topdir/ctdb/tests/onnode or $topdir/ctdb/tools/onnode or in
>> > $BINDIR/onnode after install. Same with ctdb_run_tests -- when I use
>> > installed version instead of tests/run_tests.sh, my tests run.
>> >
>> > And we do install before running tests in ctdb/wscript.
>> >
>> > I'm looking at some ways to mitigate this by setting ONNODE before
>> > running the tests but that doesn't help with the other failure.
>>
>> Attached patch fixes the tests.
>>
>> You can drop the patch that tries to use the installed run_tests.sh
>> (ctdb/wscript: use generated ctdb_run_tests).
> Thanks! Do you think we should keep bin in ctdb/?
>

I'm not sure what the correct thing to do is.  Did you change the bin
directory explicitly? Or did that change happen automatically with
waf-2.0?

If it's not too much trouble, then it's easier to do sub-directory
builds (e.g. do tdb or talloc build for testing) and discard them
without having to touch the top-level build.

Amitay.