[PATCHES] Build pytalloc for two Python versions at once, port to py3

[Thomas Nagy]

On Thu, Mar 12, 2015 at 6:53 PM, Petr Viktorin wrote:
> On 03/12/2015 03:41 PM, Jelmer Vernooij wrote:
>>
>> On Thu, Mar 12, 2015 at 03:13:26PM +0100, Petr Viktorin wrote:
>>>
>>> On 03/12/2015 02:09 PM, Jelmer Vernooij wrote:
>>>>
>>>> On Thu, Mar 12, 2015 at 12:08:48PM +0100, Petr Viktorin wrote:
>>>>>
>>>>> On 03/05/2015 10:08 PM, Andrew Bartlett wrote:
>>>>>>
>>>>>> On Wed, 2015-03-04 at 15:53 +0100, Jelmer Vernooij wrote:
>>>>>>>
>>>>>>> On Wed, Mar 04, 2015 at 11:18:08AM +1300, Andrew Bartlett wrote:
>>>>>>>>
>>>>>>>> On Tue, 2015-03-03 at 22:09 +0100, Jelmer Vernooij wrote:
>>>>>>>>>
>>>>>>>>> On Mon, Mar 02, 2015 at 12:09:25PM +0100, Petr Viktorin wrote:
>>>>>>
>>>>>> Petr,
>>>>>>
>>>>>> The reason I'm being so awkward is that I don't like any of the
>>>>>> options.
>>>>>>   - waiting forever means we:
>>>>>>    - push aside your valuable contributions
>>>>>>    - may never make the move
>>>>>
>>>>>
>>>>> I don't think the second point is true; you'd be forced to make the
>>>>> move
>>>>> eventually, most likely when Python 2 support ends in 5 years. It's
>>>>> good not
>>>>> to rush things, but waiting forever is hardly an option -- it would
>>>>> just
>>>>> make us rush things later.
>>>>
>>>>
>>>> If we move later, it saves us from a couple of years of juggling support
>>>> for two different Python versions. In fact, if we move late enough, we
>>>> might
>>>> be able to just drop Python2 support altogether and just port to Python3
>>>> rather
>>>> than supporting both versions.
>>>
>>>
>>> If there were no dependencies, there might be a chance of that working.
>>> However, if you want to coordinate with FreeIPA or distros, it becomes
>>> somewhat harder.
>>> Also, allowing more time for the port would improve chances to do thigs
>>> right, rather than rushing them all at once to avoid "juggling".
>>
>> We won't have to rush if we wait another year or two.
>>
>>> I liked Andrew's suggestion to support only Python 2, but start putting
>>> unsupported py3 bits in "unofficially", so dependent projects can start
>>> preparing their own py3 support before the big switch.
>>
>> Right, see
>> https://lists.samba.org/archive/samba-technical/2015-March/105949.html for
>> my thoughts on adding Python 3 support.
>
>
> Sorry for the misattribution, it was your suggestion.
>
>> The onus for adding support of Python3 is on those who care about it, and
>> should
>> not inconvenience the rest of us.
>
>
> Alright, then we agree about this bit at least.
> What in the patches proposed here would inconvenience you? Note that with
> these patches I *am* looking specifically at the stand-alone libraries.
>
>>>>>>   - waiting for things to improve:
>>>>>>    - you suggest is foolish, there is no indication that the python
>>>>>> community will
>>>>>>    - and we don't know that someone else will be keen to do the work
>>>>>> in the future
>>>>>
>>>>>
>>>>> Also, there's a non-trivial chance that my work on Samba could drive
>>>>> these
>>>>> improvements.
>>>>> It's been suggested a CPython list that I write a porting guide for C
>>>>> code.
>>>>> I'm quite tempted to share what I write for Samba.
>>>>
>>>>
>>>> Thanks! Upstream improvements to make it easier to support both versions
>>>> would be great.
>>>
>>> Just note I wrote *porting guide*, not upstream code.
>>
>> Ah :-(
>>
>>>> I think defines to make it easier to compile 2.6 C code with Python3
>>>> belong
>>>> in an upstream header (perhaps hidden behind and #ifdef _COMPAT_PYTHON26
>>>> or
>>>> with a deprecation warning), rather than in every Python projects'
>>>> source
>>>> tree.
>>>
>>> I don't think that is going to happen in Python itself. I can however
>>> make a
>>> library in the spirit of six/future for C extensions. And I probably
>>> could
>>> get it to be mentioned in Python docs as a best practice (along with
>>> Cython/cffi).
>>
>> In that case I wouldn't be a fan of it. I hate six, it makes python
>> programming
>> more complex, while the whole point of python3 was to clean up some things
>> about the language.
>
>
> From the code complexity point of view, I don't see the difference between
> the ifdefs being in upstream Python, or in an external library.
>
>>>>>>    - risks restricting developers to the cumbersome common subset -
>>>>>> yet another language, essentially
>>>>>
>>>>>
>>>>> That mostly applies to Python source. In the C extensions, alternatives
>>>>> to
>>>>> the removed things have usually been ported to Python 2.6, so not even
>>>>> an
>>>>> #ifdef is necesary. The main problem is deciding between bytes and
>>>>> unicode.
>>>>
>>>>
>>>> There are already quite a number of ifdefs in the current set of
>>>> patches, and
>>>> this is a fairly simple module..
>>>
>>>
>>> The ifdefs are all* in one header file, which works for tdb & ldb as
>>> well.
>>> The header would be much smaller if it was just for pytalloc.
>>
>> Same difference - it uses magic defines that are ifdeffed in a single
>> place.
>
>
> Is it the magic that is bothersome?
>
> Most of these they alias one function to another or define a constant.
> The others:
> - MODULE_INIT_FUNC has one specific use, so I believe it's safe, but I can
> replace it by 4 lines of boilerplate (up from python2's two).
> - PY_RICHCMP is relatively magical, yes. Unfortunately this is the way
> Python devs decided comparison is done (it cleans stuff up at the Python
> level, at the expense of C extensions). Also it's not in an ifdef; richcmp
> works the same on both versions.
> - Py_RETURN_NOTIMPLEMENTED mirrors Py_RETURN_{NONE,TRUE,FALSE}.
> - PyModule_Create could be a function, at the cost of a link-time
> dependency; I don't think it's worth it.
>
> I believe the resulting code is quite readable, do you disagree here?

There are new changes in the Waf files that deviate even further from
waf 1.5. Unless you want to maintain that Waf 1.5 fork forever, I
strongly recommend to focus on waf 1.8 instead. Waf 1.8 has better
support for building targets for other python versions, already
supports python3 and provides better configuration default for Redhat
systems.

I have prepared a proof of concept that builds the branch "master"
using Waf 1.8; I believe that about half of the work is done. The
differences are very too large to be attached as raw patches, so I
have added them as one archive instead:
http://ftp.waf.io/pub/developer/tnagy/samba_patches.tgz.gpg
(extract with:  gpg -d -o samba_patches.tgz samba_patches.tgz.gpg)

I could provide some help to complete the port; for this we would need
at least a branch running on the buildfarm and some plans to avoid
premature bitrot.

Thomas