Samba and Python2
Tim Beale
timbeale at catalyst.net.nz
Wed Feb 13 03:31:44 UTC 2019
Hi,
Just following up on this. How about something like the following? It
tries to merge what was discussed with what was already in the WHATSNEW.
python3 support
---------------
This is the first release of Samba which has full support for Python 3.
Samba 4.10 still has support for Python 2, however, Python 3 will be used by
default, i.e. 'configure' & 'make' will execute using python3.
To build Samba with python2 you *must* set the 'PYTHON' environment variable
for both the 'configure' and 'make' steps, i.e.
'PYTHON=python2 ./configure'
'PYTHON=python2 make'
This will override the python3 default.
Alternatively, it is possible to produce Samba Python bindings for both
Python 2 and Python 3. To do so, specify '--extra-python=/usr/bin/python2'
as part of the 'configure' command. Note that python3 will still be used as
the default in this case.
Note that Samba 4.10 supports Python 3.4 onwards.
Future Python support
---------------------
Samba 4.10 will be the last release that comes with support for Python 2.
Unfortunately, the Samba Team doesn't have the resources to support both
Python 2 and Python 3 long-term.
Samba 4.11 will not have any support for Python 2. This means if you use
Python 2 bindings it is time to migrate to Python 3 now.
Also note that Samba 4.11 will only support Python 3.6 onwards.
-------------------------------------------------------------------
If you've got suggestions/changes, let me know. Attached is a patch for
the v4.10 WHATSNEW that uses the above text.
I also started looking at removing the py2 jobs in master. Patch also
attached.
CI pass here: https://gitlab.com/samba-team/devel/samba/pipelines/47268749
(I also ran each patch through CI separately, to prove I hadn't broken
bisects or anything).
https://gitlab.com/catalyst-samba/samba/pipelines/47262120
https://gitlab.com/catalyst-samba/samba/pipelines/47263442
https://gitlab.com/catalyst-samba/samba/pipelines/47271424
https://gitlab.com/catalyst-samba/samba/pipelines/47268570
https://gitlab.com/catalyst-samba/samba/pipelines/47268733
Cheers,
Tim
On 5/02/19 11:20 PM, Douglas Bagnall via samba-technical wrote:
> On 5/02/19 10:18 PM, Noel Power via samba-technical wrote:
>> Hi Andreas,
>>
>> imho no harm in spelling things out more so +1
> Yes. +1.
>
> However, we should also make a statement about *which* Python 3,
> which I don't think we have decided on.
>
> In my opinion we should say 3.6+, based on the following arguments:
>
> * Python 2.6 has shown us it is difficult to test old python versions.
>
> * The are features in 3.6 like async/await and ordered dictionaries that could be really useful.
>
> * We are leaving 2.7 because it goes EOL next year, but 3.5 also goes EOL next year (3.4, next month).
>
> That is my view as a non-distro person.
>
> My recollection from the last time we discussed this is the distros have
> a range of somewhat contradictory needs, having as they do large stakes in
> very specific past and future moments. I don't think there was a consensus,
> but if there was, could someone remind me of it?
>
> (I notice that Debian stable, which this laptop runs, only has 3.5).
>
> cheers,
> Douglas
>
-------------- next part --------------
From f7df4f6b7183d48ea3dfef0bc8c9e75aefaf4cae Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:12:18 +1300
Subject: [PATCH 1/6] autobuild: Drop py2 autobuild jobs
Samba v4.11 will no longer support python2, so let's drop the autobuild
jobs. This will save some gitlab/sn-devel time and monery, as it's less
work for CI to do.
Note that this highlights some previous inconsistencies:
- samba-none-env-py2 was being built for gitlab but not sn-devel.
- samba-nt4-py2 was being built for sn-devel but not gitlab
I've left samba-buildpy2-only for now, which will be addressed in a
subsequent patch.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
.gitlab-ci-private.yml | 13 -------------
.gitlab-ci.yml | 21 ---------------------
script/autobuild.py | 6 ------
3 files changed, 40 deletions(-)
diff --git a/.gitlab-ci-private.yml b/.gitlab-ci-private.yml
index ea22eaa..bd2d7f5 100644
--- a/.gitlab-ci-private.yml
+++ b/.gitlab-ci-private.yml
@@ -14,12 +14,6 @@ build_samba:
# this one takes about 4 hours to finish
- script/autobuild.py samba --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-build_samba_py2:
- <<: *private_template
- script:
- # this one takes about 4 hours to finish
- - script/autobuild.py samba-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
build_samba_fileserver:
<<: *private_template
script:
@@ -32,13 +26,6 @@ build_samba_ad_dc:
# this one takes about 1 hours to finish
- script/autobuild.py samba-ad-dc --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
-build_samba_ad_dc_py2:
- <<: *private_template
- script:
- # this one takes about 1 hours to finish
- - script/autobuild.py samba-ad-dc-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
build_nt4:
<<: *private_template
script:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0729599..4ecdbe7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -35,12 +35,6 @@ build_samba_none_env:
# this one takes about 1 hours to finish
- script/autobuild.py samba-none-env --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-build_samba_none_env_py2:
- <<: *shared_template
- script:
- # this one takes about 1 hours to finish
- - script/autobuild.py samba-none-env-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
build_samba_nopython:
<<: *shared_template
script:
@@ -72,22 +66,11 @@ build_samba_ad_dc_backup:
script:
- script/autobuild.py samba-ad-dc-backup --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-build_samba_ad_dc_2_py2:
- <<: *shared_template
- script:
- # this one takes about 1 hours to finish
- - script/autobuild.py samba-ad-dc-2-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
build_samba_libs:
<<: *shared_template
script:
- script/autobuild.py samba-libs --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-build_samba_libs_py2:
- <<: *shared_template
- script:
- - script/autobuild.py samba-libs-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
build_samba_static:
<<: *shared_template
script:
@@ -124,7 +107,3 @@ build_samba_ad_dc_ntvfs:
# this one takes about 100 mins to finish
- script/autobuild.py samba-ad-dc-ntvfs --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-build_samba_ad_dc_ntvfs_py2:
- <<: *shared_template
- script:
- - script/autobuild.py samba-ad-dc-ntvfs-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
diff --git a/script/autobuild.py b/script/autobuild.py
index b1bd9d1..6356736 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -36,23 +36,17 @@ cleanup_list = []
builddirs = {
"ctdb": "ctdb",
"samba": ".",
- "samba-py2": ".",
"samba-nt4": ".",
- "samba-nt4-py2": ".",
"samba-fileserver": ".",
"samba-xc": ".",
"samba-o3": ".",
"samba-ctdb": ".",
"samba-libs": ".",
- "samba-libs-py2": ".",
"samba-static": ".",
"samba-none-env": ".",
"samba-ad-dc": ".",
- "samba-ad-dc-py2": ".",
"samba-ad-dc-ntvfs": ".",
- "samba-ad-dc-ntvfs-py2": ".",
"samba-ad-dc-2": ".",
- "samba-ad-dc-2-py2": ".",
"samba-ad-dc-backup": ".",
"samba-systemkrb5": ".",
"samba-nopython": ".",
--
2.7.4
From 75ba67b81686438b1e92188d825f16df4d520438 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:41:34 +1300
Subject: [PATCH 2/6] autobuild: Update variable name to make more sense
When we switched from python2 being the default to python3, we didn't
update this variable name. It's now handling the python2 case, but it's
a boolean flag named 'py3', which is rather confusing.
We're going to delete this code anyway, but renaming the variable first
will mean the next patch will make more sense.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 6356736..875aea4 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -457,9 +457,9 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
class builder(object):
'''handle build of one directory'''
- def __init__(self, name, sequence, cp=True, py3=False):
+ def __init__(self, name, sequence, cp=True, py2=False):
self.name = name
- self.py3 = py3
+ self.py2 = py2
if name in builddirs:
self.dir = builddirs[name]
else:
@@ -498,7 +498,7 @@ class builder(object):
(self.stage, self.cmd, self.output_mime_type) = self.sequence[self.next]
self.cmd = self.cmd.replace("${PYTHON_PREFIX}", get_python_lib(plat_specific=1, standard_lib=0, prefix=self.prefix))
self.cmd = self.cmd.replace("${PREFIX}", "--prefix=%s" % self.prefix)
- if self.py3:
+ if self.py2:
self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python)
# The trailing space is important
self.cmd = self.cmd.replace("${PY3_ONLY}", "python2 ")
@@ -542,7 +542,7 @@ class buildlist(object):
b = builder(n,
tasks[n[:-4]],
cp=n is not "pidl",
- py3=True)
+ py2=True)
else:
b = builder(n, tasks[n], cp=n is not "pidl")
self.tlist.append(b)
--
2.7.4
From 60c3aad9542f284f756d82f284232984cce29573 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:46:21 +1300
Subject: [PATCH 3/6] autobuild: drop special case for '-py2' jobs
These autobuild jobs no longer exist, so we can remove this extra code
now.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 20 ++++----------------
1 file changed, 4 insertions(+), 16 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 875aea4..5ea1e43 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -457,9 +457,8 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
class builder(object):
'''handle build of one directory'''
- def __init__(self, name, sequence, cp=True, py2=False):
+ def __init__(self, name, sequence, cp=True):
self.name = name
- self.py2 = py2
if name in builddirs:
self.dir = builddirs[name]
else:
@@ -498,13 +497,8 @@ class builder(object):
(self.stage, self.cmd, self.output_mime_type) = self.sequence[self.next]
self.cmd = self.cmd.replace("${PYTHON_PREFIX}", get_python_lib(plat_specific=1, standard_lib=0, prefix=self.prefix))
self.cmd = self.cmd.replace("${PREFIX}", "--prefix=%s" % self.prefix)
- if self.py2:
- self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python)
- # The trailing space is important
- self.cmd = self.cmd.replace("${PY3_ONLY}", "python2 ")
- else:
- self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "")
- self.cmd = self.cmd.replace("${PY3_ONLY}", "")
+ self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "")
+ self.cmd = self.cmd.replace("${PY3_ONLY}", "")
self.cmd = self.cmd.replace("${PREFIX_DIR}", "%s" % self.prefix)
self.cmd = self.cmd.replace("${TESTS}", options.restrict_tests)
# if self.output_mime_type == "text/x-subunit":
@@ -538,13 +532,7 @@ class buildlist(object):
os.environ['AUTOBUILD_RANDOM_SLEEP_OVERRIDE'] = '1'
for n in tasknames:
- if n not in tasks and n.endswith("-py2"):
- b = builder(n,
- tasks[n[:-4]],
- cp=n is not "pidl",
- py2=True)
- else:
- b = builder(n, tasks[n], cp=n is not "pidl")
+ b = builder(n, tasks[n], cp=n is not "pidl")
self.tlist.append(b)
if options.retry:
rebase_remote = "rebaseon"
--
2.7.4
From 031febd7652097a1bc8685f73e3723f2de1c4fd3 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:33:29 +1300
Subject: [PATCH 4/6] autobuild: Replace samba-buildpy2-only with
samba-nopython-py2
For Samba 4.11, the minimum python2 functionality we will support (for
now, at least - we may change our minds) is for the --disable-python
target, i.e. if you're excluding all the python functionality from
samba, then WAF should still support being built with python2.
The use case here is old unix platforms that want to use smbd, but don't
have python3 support.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
.gitlab-ci.yml | 10 +++++-----
script/autobuild.py | 6 +++---
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 4ecdbe7..3553881 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -40,6 +40,11 @@ build_samba_nopython:
script:
- script/autobuild.py samba-nopython --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_nopython-py2:
+ <<: *shared_template
+ script:
+ - script/autobuild.py samba-nopython-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
build_samba_systemkrb5:
<<: *shared_template
script:
@@ -96,11 +101,6 @@ build_others:
- script/autobuild.py tdb --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
- script/autobuild.py tevent --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-build_samba_buildpy2_only:
- <<: *shared_template
- script:
- - python script/autobuild.py samba-buildpy2-only --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
-
build_samba_ad_dc_ntvfs:
<<: *shared_template
script:
diff --git a/script/autobuild.py b/script/autobuild.py
index 5ea1e43..cfa77d9 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -50,7 +50,7 @@ builddirs = {
"samba-ad-dc-backup": ".",
"samba-systemkrb5": ".",
"samba-nopython": ".",
- "samba-buildpy2-only": ".",
+ "samba-nopython-py2": ".",
"ldb": "lib/ldb",
"tdb": "lib/tdb",
"talloc": "lib/talloc",
@@ -419,8 +419,8 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- "samba-buildpy2-only": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "PYTHON='python' ./configure.developer --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
+ "samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "PYTHON='python' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
("make", "PYTHON='python' make -j", "text/plain"),
("install", "PYTHON='python' make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
--
2.7.4
From 6a00adf2e6a354a12a0cb2be3535aa1120acc271 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 13:55:09 +1300
Subject: [PATCH 5/6] autobuild: Make python2 usage explicit
This currently assumes 'python' means python2. It makes more sense if we
make the python2 part explicit.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index cfa77d9..712e88c 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -420,11 +420,11 @@ tasks = {
("clean", "make clean", "text/plain")],
"samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "PYTHON='python' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
- ("make", "PYTHON='python' make -j", "text/plain"),
- ("install", "PYTHON='python' make install", "text/plain"),
+ ("configure", "PYTHON='python2' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
+ ("make", "PYTHON='python2' make -j", "text/plain"),
+ ("install", "PYTHON='python2' make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
- ("clean", "PYTHON='python' make clean", "text/plain")],
+ ("clean", "PYTHON='python2' make clean", "text/plain")],
# these are useful for debugging autobuild
--
2.7.4
From fd916a25d44926dba6c5cb5e91ae2e10acf18b60 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 13:58:08 +1300
Subject: [PATCH 6/6] autobuild: Move samba-nopython-py2 target
Move samba-nopython-py2 so that it's next to samba-nopython.
Note that samba-nopython does extra work because it's checking
that we can build and run all the core components without python.
For samba-nopython-py2, we just want to check we can do the WAF/
make part using python2.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 712e88c..d225d2a 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -353,6 +353,13 @@ tasks = {
("libs-clean", "make clean", "text/plain")
],
+ # also sanity-check we can do the WAF make part of the above using python2
+ "samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "PYTHON='python2' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
+ ("make", "PYTHON='python2' make -j", "text/plain"),
+ ("install", "PYTHON='python2' make install", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
+ ("clean", "PYTHON='python2' make clean", "text/plain")],
"ldb": [
@@ -419,13 +426,6 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- "samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "PYTHON='python2' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
- ("make", "PYTHON='python2' make -j", "text/plain"),
- ("install", "PYTHON='python2' make install", "text/plain"),
- ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
- ("clean", "PYTHON='python2' make clean", "text/plain")],
-
# these are useful for debugging autobuild
'pass': [("pass", 'echo passing && /bin/true', "text/plain")],
--
2.7.4
-------------- next part --------------
From 63b7c6e31ff18901f0f86bce487c886d5e6ad852 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 11:28:29 +1300
Subject: [PATCH] WHATSNEW: Add note that python2 support will be dropped on
v4.11
Add a warning to Samba users that v4.10 will be the last Samba release
with python2 support.
I've reworked the existing text describing the different python2 build
options for 4.10. Hopefully this makes it slightly clearer.
Finally, I've removed the note about --disable-python WAF support for
4.11. While we don't have immediate plans to drop this, including this
info was confusing. It was distracting from the main message aimed at
the majority of Samba users that python2 support is disappearing
in v4.11.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13785
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
WHATSNEW.txt | 41 ++++++++++++++++++++++++++---------------
1 file changed, 26 insertions(+), 15 deletions(-)
diff --git a/WHATSNEW.txt b/WHATSNEW.txt
index 8bbb1fd..e34b801 100644
--- a/WHATSNEW.txt
+++ b/WHATSNEW.txt
@@ -87,23 +87,34 @@ exceeded Samba drops new connections immediately.
python3 support
---------------
-The version of python which is now the default for samba is python3.
-'configure' & 'make' will execute using python3. It is possible to still
-specify an additional python version with '--extra-python'
-e.g. '--extra-python=/usr/bin/python2'. It should be noted that support for
-this option will be deprecated in a future release.
-
-What if I need to build with python2? To build with python2 you *must* set
-the 'PYTHON' environment variable to override the python3 default for both
-'configure' and 'make' steps.
-
- 'PYTHON=python2 ./configure.developer'
-&
+This is the first release of Samba which has full support for Python 3.
+Samba 4.10 still has support for Python 2, however, Python 3 will be used by
+default, i.e. 'configure' & 'make' will execute using python3.
+
+To build Samba with python2 you *must* set the 'PYTHON' environment variable
+for both the 'configure' and 'make' steps, i.e.
+ 'PYTHON=python2 ./configure'
'PYTHON=python2 make'
+This will override the python3 default.
+
+Alternatively, it is possible to produce Samba Python bindings for both
+Python 2 and Python 3. To do so, specify '--extra-python=/usr/bin/python2'
+as part of the 'configure' command. Note that python3 will still be used as
+the default in this case.
+
+Note that Samba 4.10 supports Python 3.4 onwards.
+
+Future Python support
+---------------------
+
+Samba 4.10 will be the last release that comes with support for Python 2.
+Unfortunately, the Samba Team doesn't have the resources to support both
+Python 2 and Python 3 long-term.
+
+Samba 4.11 will not have any support for Python 2. This means if you use
+Python 2 bindings it is time to migrate to Python 3 now.
-Note: Support for python2 (with the exception of a build configured with
- 'PYTHON=python2 ./configure --disable-python' and built with
- 'PYTHON=python2 make') will be deprecated in the next release.
+Also note that Samba 4.11 will only support Python 3.6 onwards.
JSON logging
------------
--
2.7.4
More information about the samba-technical
mailing list