A plan to get to py3

[Noel Power]

Hi Andrew


On 09/08/18 01:46, Andrew Bartlett wrote:
[...]
> So, here is a plan:
>
> Please make a waf that works with 
>
> PYTHON=python3 ./configure --enable-developer && make -j
>
> Then change selftest/selftesthelpers.py:planpythontestsuite() to only
> run python3 tests if py3_compatible is set and we are built with
> python3, and only python2 tests if built primarily with python2.
Do I understand correctly this is something you are proposing for when
Alexanders waf 2.0 is merged ? or right now.
I ask because a configure like the above won't work, it will force waf
to run under python3 (which of course wont work)

Even with Alexanders changes merged there there is still an unknown
quantity of work to do to get make test even to work, I managed to get
some noenv tests to work but for sure any of the other test environment
stuff wont work. One of the problems here is that lots of python scripts
(e.g. ones used in blackbox tests) are run directly. By directly I mean
that %(path)/something.py is called without a preceding python
executable (like python3 %(path)/something.py) These can be from perl or
shell. These scripts end up been run by the default python (python2).
This causes problems (mixing the versions) and in a pure python3 build
python2 executed scripts looking for samba python modules won't find
them. And it's not just blackbox scripts, the same is true for other
scripts called as part of make test including samba_tool which is used
by the test environment, there is a whole house of cards here.

So while maybe this is something we want to have after Alexanders work
hits master it still is some way off I think.

Maybe a variation on this theme might work, iiuc correctly you wish to
duplicate some CI jobs doing py2 and py3 testing simultaneously in a
single job so now there will be 2 jobs, one doing python2 only
configure/build/test and the other python3 configure/build/test.

But we really need something now... to be able to get appropriate CI
coverage to get the code in place that is needed for when waf2.0 gets
used in a pure python3 way.

In the interim (until Alexanders waf is merged and the necessary work to
get make test (with ad_dc environments etc.) working we could do
something like what you suggest (again assuming I got it correctly) e.g.
we could adjust our existing python2 waf to split out jobs as follows

* use same configure as we do now

  ./configure --enable-developer --extra-python=/usr/bin/python3 blah
blah blah

* build_samba
    py2 configure/ py2 build / p2test (e.g. run all the tests under py2
completely ignore EXTRA_PYTHON)

* build_samba_py3
   py2 configure/ py2 build/  py3test (e.g. run only those test that are
py3_compatible with python3, don't run them under python2, same
suggestion nearly as yours for what to do with planpythontestsuite but
with a twist, not based on python version but some other yet to be
decided flag/env variable)

maybe instead of using PYTHON=python3 we could use USE_PYTHON=python3
(and later change that to PYTHON and hopefully if coded correctly most
of what is done above could be rejigged easily to be more in line with
your orig suggestion or some variation)

How does that sound ? I know you included lots of detail (snipped) which
I don't really understand yet, nor do I really know how to achieve the
above but I would hope to be able to cobble something together :-)

Firstly though, I'm going to hardcode a pair of branches that achieve
something similar to what I suggest e.g. a build of just PY2 alone
(despite my changes) and another build that will only run those
py3compatible tests (and not the p2 ones) The hope would be that both
branches will pass CI, if they don't think I'm afraid were back at
square one (at least for my branch)

Noel