Pure python3 for the AD DC for Samba 4.10 (5.0)

[Noel Power]

Hi


On 03/08/18 01:16, Andrew Bartlett wrote:
> On Thu, 2018-08-02 at 22:45 +0200, Stefan Metzmacher wrote:
>> Am 02.08.2018 um 22:21 schrieb Andrew Bartlett via samba-technical:
>>> On Thu, 2018-08-02 at 21:02 +0100, Noel Power wrote:
>>>> 4)
>>>>
>>>>   PYTHON=/usr/bin/python3 python3 ./buildtools/bin/waf [FAILS]
>>>>
>>>> fails with
>>>>
>>>> "ERROR: grouping library target
>>>> pytalloc-util.cpython-36m-x86_64-linux-gnu not declared in
>>>> samba_python.cpython-36m-x86_64-linux-gnu"
>>>>
>>>> I haven't tried to get to the bottom of this yet, waf scares me :0
>>>>
>>>> But I think this is a configuration (as in wbuild) problem because a
>>>> python3 build (using detected python lib version 2.7 works fine)
>>> Noel,
>>>
>>> Can you strip out all the extra python stuff and try again?  We seem to
>>> be on the way to a rough consensus of py3 only for 4.10 and that would
>>> greatly simplify everything.
>> I'm not for python3 only. I think setups with the primary python as 2.7
>> and extra python being python3 should keep working.
>> At least pure python2.7 or python3 should work.
> For fileserver builds --without-python, sure, long-term python 2.7
> support is reasonable to request.   
>
> I have no issue with the waf build still operating on python 2.7. 
>
> For the AD DC, we can't afford the maintaince burden to try and keep it
> all working with both, we need to cut over and just write Samba's
> 'python' in a single language, not the dumbed down combination that
> happens to be mostly compatible with the two distinct languages
> 'python2' and 'python3'.  
>
> This follows the strongest advise I got from python experts when we
> started this endeavour: to migate, then cut ties with python2 as soon
> as possible.  
>
> Even if we make every 'python' test operate with py3_compatible=True,
> we won't fully test python3, because binaries like smbtorture (which
> invoke python) won't be built against python3, and the provision
> scripts run in the environment setup won't be run as python3.  Other
> tests are destructive (the demote tests) so can't be run twice. 
>
> Therefore, the --extra-python code in our build system, while helpful
> during development, is actually incredible painful for packagers and
> hindrance to Samba long-term.  It, as has been seen here, just gets in
> the way of a pure python3 build.  
>
> As only real test would be a full ./configure with pure python3, and
> while possible I suggest adding another whole duplicate to autobuild
> would be absurd, we should just move to the AD DC build being with
> python3 only.  
While I can agree with alot of points here, however I don't think we
need to make hard decisions about if/when to cut off py2 just yet.
Lets keep our options open. I think the reality is that anyway in the
first instance we will have to have py2 and py3 working together in a
compatible way at the very least for a reasonable run in time while
getting py3 to work.
> We should push hard and finish this!
Oh I'm all for pushing to get this in as quickly as possible, my own
patches are getting stale really quickly with our fast moving master, I
am stuggling even to keep my py2-py3-WIP branch up to date (even just in
terms of rebasing). In fact I have failed to get a successful CI build
since my initial attempt to rebase to latest master post vacation.

So even now my currently failing 'rebase' branch  is quite out of date.
Not certain whether this is a resource issue or something to do with
environments getting polluted because of the extra py3 tests running.
The following tests have failed (and appear to be flapping) but
eventually after a number of retries passed.

samba.tests.samba_tool.visualize_drs
samba.tests.join
samba4.drs.getncchanges
samba4.drs.replica_sync

my last attempt got as far as
[951(7400)/959 at 4h56m31s] samba.tests.samba_tool.demote(promoted_dc)
then failed (most of the errors with the above tests seem to have lots
of messages around connection failures and time)outs

With that job now taking ~310 minutes the turn around time is really
torturous, something odd is going on there, I am running out of ideas
about getting to the bottom of it. Of course running any of the
problematic tests in question manually pass no problem on my own machine.

Noel