waf and building python modules for multiple python versions

[Michael Tokarev]

09.04.2022 10:52, Andrew Bartlett via samba-technical wrote:

>> Here with waf, I don't see how it is possible. Well, I do
>> see one possibility: to copy source into yet another separate
>> subdir (as many subdirs as there are additional python
>> versions for the build) and run configure & build & install
>> in every of them in turn, specifying the right PYTHON= in
>> the environment.  This *should* work.

This way appears to be too fragile, since it requires the
dependencies to be done with extra care.  The thing is that
it is not enough to *link* with a different python libs and
use version-specific soname and library name. This also require
recompiling of the .c file with the right python includes.
While *some* of things will continue to work when compiled with
one python includes but linked with another python libs, it
definitely is not supported.

I'm not sure all .c files python modules are *compiled* into
.o files with python-version-specific names, if at all.

It is actually more than this.

I just built talloc with default python 3.10, removed
bin/default/libpytalloc-util.cpython-310-x86-64-linux-gnu.so, and
re-built it thysly:

  PYTHON=python3.9 make

and it re-linked the same libpytalloc-util.cpython-310-x86-64-linux-gnu.so.
So apparently it recorded somewhere the python binary which was
used during configure.

> That is the only plan I can think of.  But which python version would
> Samba's python-using libs link to?  For as long as you ship smbtorture
> and expect it to work you would have to do multiple versions of that as
> well...

It is not about samba proper. These will have to be recompiled as
a whole anyway.  This is about libraries (talloc, tdb) which are
used by other packages. When your default python is 3.9 but 3.10
is also available, you build, say, python3-talloc package to contain
*two* versions of the pytalloc module (containing
libpytalloc-util.cpython-39-x86-64-linux-gnu.so and
libpytalloc-util.cpython-310-x86-64-linux-gnu.so ).  Other packages
are built with whatever python version they choose, this is not
relevant by now (either of 3.9 or 3.10).  Next, you switch default
python from 3.9 to 3.10. If a package used 3.9 module in talloc,
it will just switch to 3.10 one which is *already provided* by
python3-talloc package, there's no need to rebuild python3-talloc
so that packages using it will work after python version switch.

But if python3-talloc is built with 3.9 python *only* (as it is
the case now), in order to switch python version, we'll have to
recompile python3-talloc to provide 3.10 version. But this means
that all programs which used 3.9 version-provided talloc will have
to be recompiled *too*, since after python3-talloc recompile it
wont provide 3.9 variant anymore.

This is why it is way easier for transitions if the cpython-providing
packages provide two versions of python libs. It becomes *dramatically*
less messy for transitions.

I'm not talking about samba internal libs (used by the main samba
package, smbclient and smbtorture) - since these all are built
from the same source anyway and have no other packages (outside
of samba source) using them. It all is rebuilt in one go and all
packages are updated at once.  I'm talking about utility libs
which are used by other packages too, which are not built from
samba sources.

Thanks,

/mjt