Python3 progress

[Andrew Bartlett]

On Mon, 2018-11-26 at 10:47 +0000, Noel Power wrote:
> Hi Andrew
> On 26/11/2018 04:54, Andrew Bartlett wrote:
> > G'Day Noel,
> > 
> > I notice some branches with improvements for the python3 effort passing
> > CI.  Well done!
> > 
> > One thing I think we do need is to have the PYTHON set during provision
> > stick.  That is, if you run:
> > 
> > PYTHON=python3 ./configure
> > 
> > That should be enough, we shouldn't have to set it for a subsequent
> > make or make test.
> 
> That is something certainly we can look at in the next steps
> > 
> > The 'make test' code should pull it from waf in the same way perl is,
> > and to get waf itself run under the correct python, perhaps we could
> > revert to the autoconf style Makefile.in and have configure substitute
> > Makefile.in -> Makefile.
> 
> yep that's possible I guess
> > 
> > That would allow us to move to python3 by default simply by changing
> > the ./configure wrapper.
> 
> My current plan is to get everything to work with the current wrappers 
> e.g. the transition the  default setting of PYTHON:=python => 
> PYTHON:=python3 (in configure & Makefile) This is kindof necessary 
> otherwise autobuild gets kindof full of PYTHON=python3 (and you need to 
> remember to do this *everywhere* otherwise you will have mismatches of 
> the build resulting in at least some parts still building under python2 
> or worse some strange errors with some of the more complicated build 
> jobs (e.g. samba-libs)..... already been there :-)

OK.

> This means that by default we will move to python3 on master and to run 
> the python2 build you would need to do the reverse of what we do at the 
> moment e.g. (PYTHON=python ./configure && PYTHON=python make)

Good.  Depending on how we go we could even make python3 the default
for Samba 4.10.

> My intention was to get to this point and provide patches with and 
> without the final step of switching Makefile, configure, gitlab-* & 
> autobuild to build/run/test under python3 as default. Additionally to 
> have CIs with 2 branches, 1 with patches up to the py3 switch (showing 
> the current python2 build is working) and the second branch & CI with 
> the additional patches that provide the final switch to python3 to show 
> the pure python3 is working. This would allow us to be confident about 
> the remaining porting patches and be able to merge those and also give 
> us the option to merge the remainder to 'flick the switch' or discuss 
> that part further.

OK.

> Also worth noting that I have not being issuing any new merge requests 
> recently because we really are close to this point above now, 
> additionally adding the separate py3 temporary CI tasks for the 
> remaining samba_ad_dc , build_samba etc. will just add too much time 
> onto the CI. To properly test these patches (and run the tests) we need 
> to run from python3 so at least in my private builds at the moment, it's 
> kindof all or nothing, I have to get pure py3 fully working for all 
> configure/build/test (and it's close)

I would suggest that review resources, not autobuild time is the
limited factor.  I would prefer to see more CPU time spent now so that
we can be patches reviewed in before Christmas, and so allow Samba 4.10
to ship with the experimental ability to run with Python3.  

If you could resume submitting merge requests we can focus on getting
patches into master and ensure they don't bitrot. 

It is easiest to review small sets of patches than 100 patch monster
branches, so do try and drip-feed that patches as they become ready,
even if they cause more jobs to be created.

If the autobuild time becomes unmanageable for the rest of the team,
even knowing it will only last until Janurary, we can address that.


> > Finally, we should use the hook in the build system currently employed
> > to change os.path during the install to also change:
> > 
> > #!/usr/bin/env python
> > to eg
> > #!/usr/bin/env python3
> > (or the full path if a full path was set)
> 
> Yes, for any script that is a build product we should make sure that the 
> shebang is correct, I presume since you mention it we aren't doing this, 
> thanks for mentioning 'cause I didn't even think about it.
> 
> > 
> > What do you think?
> 
> Well I still think I need to get to the point where everything runs 
> under python3 + python2 (all build artefacts & tests) Then... we need to 
> have a discussion as to how we proceed. If we want to retain python2 
> (which I suppose we do for a period of time at least)  then we need to 
> think about how we test that, it's clear to me at least from the last 
> weeks that the python3 tests we currently run are only a partial tests, 
> in fact it is a bit worse than that because the python3 tests (in our 
> python2 build) are a kindof hybrid where some parts of the tests can be 
> running python2 and others python3 :-) (clearly not ideal, but better 
> than nothing)
> 
> Right now I would lean to (when possible)
> 
>    1) transition to python3 as default, make the current tests reverse 
> sense, e.g. we run what are currently the py3 tests (as python2)
> 
>    2) We should additionally add additional duplicate autobuild tasks 
> (and CI jobs) to build talloc, tevent, ldb, ctdb, samba-libs, samba-xc 
> etc as pure python2 (to make sure we don't break the basic py2 build 
> support... these shouldn't add too much extra time, they are just 
> build/install))
> 
>    3) We maybe should add extra (but optional) autobuild tasks for those 
> remaining  tasks that don't have purepython2 duplicates e.g. 
> build_samba, build_ad_dc (etc) and run these on master maybe once or 
> twice a week ?

I would have everything supported be 'in'.  Anything we can't afford to
run in autobuild should not be in what we support, and we need to be
clear about that.

Given the timing, doing one last release (Samba 4.10) with both Python2
and Python3 will then give us the best opportunity to then drop python2
cold right after the release, because for 4.10 folks can build either
and we can address the real-world issues seen either by advising folks
to rebuild with python2 while we resolve things properly in master.

> > 
> This would give up a good mix of fairly good py2 coverage for normal CI 
> and the opportunity for full py2 CI coverage occasionally (on a 
> frequency that makes sense) But... yes this will for sure cause issues 
> if/when py2 specific problems occur that are detected after the fact :/ 
> they will be harder to track down. Anyway it's just a thought

Indeed.  I would avoid that.

Thanks,

Andrew Bartlett


-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba