A plan to get to py3

[Noel Power]

Hi Andrew


On 15/08/18 05:07, Andrew Bartlett wrote:
> On Fri, 2018-08-10 at 11:03 +1200, Andrew Bartlett via samba-technical
> wrote:
>> On Thu, 2018-08-09 at 12:12 +0100, Noel Power wrote:
>>> Hi Andrew
>>> In the interim (until Alexanders waf is merged and the necessary work to
>>> get make test (with ad_dc environments etc.) working we could do
>>> something like what you suggest (again assuming I got it correctly) e.g.
>>> we could adjust our existing python2 waf to split out jobs as follows
>>>
>>> * use same configure as we do now
>>>
>>>   ./configure --enable-developer --extra-python=/usr/bin/python3 blah
>>> blah blah
>>>
>>> * build_samba
>>>     py2 configure/ py2 build / p2test (e.g. run all the tests under py2
>>> completely ignore EXTRA_PYTHON)
>>>
>>> * build_samba_py3
>>>    py2 configure/ py2 build/  py3test (e.g. run only those test that are
>>> py3_compatible with python3, don't run them under python2, same
>>> suggestion nearly as yours for what to do with planpythontestsuite but
>>> with a twist, not based on python version but some other yet to be
>>> decided flag/env variable)
>>>
>>> maybe instead of using PYTHON=python3 we could use USE_PYTHON=python3
>>> (and later change that to PYTHON and hopefully if coded correctly most
>>> of what is done above could be rejigged easily to be more in line with
>>> your orig suggestion or some variation)
>>>
>>> How does that sound ? I know you included lots of detail (snipped) which
>>> I don't really understand yet, nor do I really know how to achieve the
>>> above but I would hope to be able to cobble something together :-)
>> This sounds like a plan, and a practical first step on the way to
>> building with python3 (then we can then strip the USE_PYTHON bit back
>> out). 
> G'Day Noel,
>
> Do you need a hand implementing that?
I did try at least some experimentation, firstly I hard coded some
changes into some copies of my problematic branch e.g. disabling all the
py3 tests I enabled and running the CI, no joy, build_samba was still
failing even after some retries. So running just python2 tests alone did
not seem to help.

I also tried something along the lines of what we discussed, it didn't
help as expected but still I think in the long run it could help as more
python3 code is enabled so I wanted to get a feel for what was involved
(see https://gitlab.com/samba-team/devel/samba/pipelines/27560482)

I don't like my implementation of this, the amount of duplication of
task, dirs etc in autobuild. I think maybe a better way is to push the
burden more into wscript and extend the USE_PYTHON3 variable somehow so

a) if USE_PYTHON3 is defined then
     * if set (e.g. 1 or something similarly indicative) it will only
run py3 tests
     * if not set (e.g. 0 ) then it will only run py2 tests
b) If USE_PYTHON3 is not defined then *both* will run as before

   -  something like TEST_PYTHON=python or TEST_PYTHON=python3 is maybe
a better choice for the env variable and usage instead of 'USE_PYTHON3'

I suggest this for a couple of reasons
   1. less mess in autobuild where the current approach needs all this
duplicated stuff. ok, maybe you could come up with some lambda type
thing in the various maps to avoid that (personally I am not a fan of
lamba in such usage, it hurts my head when I look at it... and I
couldn't figure out how to do it quickly when hacking this)
   2. you can easily just pick a job from .gitlab-ci* and quickly split
it (or leave existing ones alone if necessary)

like

-build_samba:
+build_samba-py2:
   <<: *private_template
   script:
     # this one takes about 4 hours to finish
-    - python script/autobuild.py samba            --verbose --nocleanup
--keeplogs --tail --testbase /tmp/samba-testbase
+    - USE_PYTHON3=0 python script/autobuild.py samba           
--verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
+build_samba-py3:
+  <<: *private_template
+  script:
+    # this one takes about 4 hours to finish
+    - USE_PYTHON3=1 python script/autobuild.py samba           
--verbose --noc


Anyway given the split of autobuild above is not going to help with my
wip-branch  (and again I can't figure out how/why) I decided to
concentrate for the moment on getting a patchset out for review (with
the intention to return to the above). So I have been pulling out
patches into a new branch (while trying to monitor when CI seems goes
south or become less able to complete)

https://gitlab.com/samba-team/devel/samba/commits/npower-python3-misc%233
https://gitlab.com/samba-team/devel/samba/pipelines/27967739

I seem to have problems with build_samba and am in the process of
rewinding the commits to hopefully hit a point where CI succeeds again.
I'll look into the CI python job split changes again next week.

Noel