Python3 progress

[Noel Power]

Hi Andrew
On 26/11/2018 04:54, Andrew Bartlett wrote:
> G'Day Noel,
>
> I notice some branches with improvements for the python3 effort passing
> CI.  Well done!
>
> One thing I think we do need is to have the PYTHON set during provision
> stick.  That is, if you run:
>
> PYTHON=python3 ./configure
>
> That should be enough, we shouldn't have to set it for a subsequent
> make or make test.
That is something certainly we can look at in the next steps
>
> The 'make test' code should pull it from waf in the same way perl is,
> and to get waf itself run under the correct python, perhaps we could
> revert to the autoconf style Makefile.in and have configure substitute
> Makefile.in -> Makefile.
yep that's possible I guess
>
> That would allow us to move to python3 by default simply by changing
> the ./configure wrapper.

My current plan is to get everything to work with the current wrappers 
e.g. the transition the  default setting of PYTHON:=python => 
PYTHON:=python3 (in configure & Makefile) This is kindof necessary 
otherwise autobuild gets kindof full of PYTHON=python3 (and you need to 
remember to do this *everywhere* otherwise you will have mismatches of 
the build resulting in at least some parts still building under python2 
or worse some strange errors with some of the more complicated build 
jobs (e.g. samba-libs)..... already been there :-)

This means that by default we will move to python3 on master and to run 
the python2 build you would need to do the reverse of what we do at the 
moment e.g. (PYTHON=python ./configure && PYTHON=python make)

My intention was to get to this point and provide patches with and 
without the final step of switching Makefile, configure, gitlab-* & 
autobuild to build/run/test under python3 as default. Additionally to 
have CIs with 2 branches, 1 with patches up to the py3 switch (showing 
the current python2 build is working) and the second branch & CI with 
the additional patches that provide the final switch to python3 to show 
the pure python3 is working. This would allow us to be confident about 
the remaining porting patches and be able to merge those and also give 
us the option to merge the remainder to 'flick the switch' or discuss 
that part further.

Also worth noting that I have not being issuing any new merge requests 
recently because we really are close to this point above now, 
additionally adding the separate py3 temporary CI tasks for the 
remaining samba_ad_dc , build_samba etc. will just add too much time 
onto the CI. To properly test these patches (and run the tests) we need 
to run from python3 so at least in my private builds at the moment, it's 
kindof all or nothing, I have to get pure py3 fully working for all 
configure/build/test (and it's close)

>
> Finally, we should use the hook in the build system currently employed
> to change os.path during the install to also change:
>
> #!/usr/bin/env python
> to eg
> #!/usr/bin/env python3
> (or the full path if a full path was set)

Yes, for any script that is a build product we should make sure that the 
shebang is correct, I presume since you mention it we aren't doing this, 
thanks for mentioning 'cause I didn't even think about it.

>
> What do you think?

Well I still think I need to get to the point where everything runs 
under python3 + python2 (all build artefacts & tests) Then... we need to 
have a discussion as to how we proceed. If we want to retain python2 
(which I suppose we do for a period of time at least)  then we need to 
think about how we test that, it's clear to me at least from the last 
weeks that the python3 tests we currently run are only a partial tests, 
in fact it is a bit worse than that because the python3 tests (in our 
python2 build) are a kindof hybrid where some parts of the tests can be 
running python2 and others python3 :-) (clearly not ideal, but better 
than nothing)

Right now I would lean to (when possible)

   1) transition to python3 as default, make the current tests reverse 
sense, e.g. we run what are currently the py3 tests (as python2)

   2) We should additionally add additional duplicate autobuild tasks 
(and CI jobs) to build talloc, tevent, ldb, ctdb, samba-libs, samba-xc 
etc as pure python2 (to make sure we don't break the basic py2 build 
support... these shouldn't add too much extra time, they are just 
build/install))

   3) We maybe should add extra (but optional) autobuild tasks for those 
remaining  tasks that don't have purepython2 duplicates e.g. 
build_samba, build_ad_dc (etc) and run these on master maybe once or 
twice a week ?

This would give up a good mix of fairly good py2 coverage for normal CI 
and the opportunity for full py2 CI coverage occasionally (on a 
frequency that makes sense) But... yes this will for sure cause issues 
if/when py2 specific problems occur that are detected after the fact :/ 
they will be harder to track down. Anyway it's just a thought


Noel