Samba and Python2
Tim Beale
timbeale at catalyst.net.nz
Thu Feb 14 04:13:29 UTC 2019
I've attached an updated patch-set to drop the py2 jobs from CI
(.gitlab-ci.yml just got substantially reworked in master).
New CI link: https://gitlab.com/samba-team/devel/samba/pipelines/47451100
On 13/02/19 4:31 PM, Tim Beale via samba-technical wrote:
> Hi,
>
> Just following up on this. How about something like the following? It
> tries to merge what was discussed with what was already in the WHATSNEW.
>
> python3 support
> ---------------
> This is the first release of Samba which has full support for Python 3.
> Samba 4.10 still has support for Python 2, however, Python 3 will be used by
> default, i.e. 'configure' & 'make' will execute using python3.
>
> To build Samba with python2 you *must* set the 'PYTHON' environment variable
> for both the 'configure' and 'make' steps, i.e.
> 'PYTHON=python2 ./configure'
> 'PYTHON=python2 make'
> This will override the python3 default.
>
> Alternatively, it is possible to produce Samba Python bindings for both
> Python 2 and Python 3. To do so, specify '--extra-python=/usr/bin/python2'
> as part of the 'configure' command. Note that python3 will still be used as
> the default in this case.
>
> Note that Samba 4.10 supports Python 3.4 onwards.
>
> Future Python support
> ---------------------
> Samba 4.10 will be the last release that comes with support for Python 2.
> Unfortunately, the Samba Team doesn't have the resources to support both
> Python 2 and Python 3 long-term.
>
> Samba 4.11 will not have any support for Python 2. This means if you use
> Python 2 bindings it is time to migrate to Python 3 now.
>
> Also note that Samba 4.11 will only support Python 3.6 onwards.
> -------------------------------------------------------------------
>
> If you've got suggestions/changes, let me know. Attached is a patch for
> the v4.10 WHATSNEW that uses the above text.
>
> I also started looking at removing the py2 jobs in master. Patch also
> attached.
> CI pass here: https://gitlab.com/samba-team/devel/samba/pipelines/47268749
>
> (I also ran each patch through CI separately, to prove I hadn't broken
> bisects or anything).
> https://gitlab.com/catalyst-samba/samba/pipelines/47262120
> https://gitlab.com/catalyst-samba/samba/pipelines/47263442
> https://gitlab.com/catalyst-samba/samba/pipelines/47271424
> https://gitlab.com/catalyst-samba/samba/pipelines/47268570
> https://gitlab.com/catalyst-samba/samba/pipelines/47268733
>
> Cheers,
> Tim
>
> On 5/02/19 11:20 PM, Douglas Bagnall via samba-technical wrote:
>> On 5/02/19 10:18 PM, Noel Power via samba-technical wrote:
>>> Hi Andreas,
>>>
>>> imho no harm in spelling things out more so +1
>> Yes. +1.
>>
>> However, we should also make a statement about *which* Python 3,
>> which I don't think we have decided on.
>>
>> In my opinion we should say 3.6+, based on the following arguments:
>>
>> * Python 2.6 has shown us it is difficult to test old python versions.
>>
>> * The are features in 3.6 like async/await and ordered dictionaries that could be really useful.
>>
>> * We are leaving 2.7 because it goes EOL next year, but 3.5 also goes EOL next year (3.4, next month).
>>
>> That is my view as a non-distro person.
>>
>> My recollection from the last time we discussed this is the distros have
>> a range of somewhat contradictory needs, having as they do large stakes in
>> very specific past and future moments. I don't think there was a consensus,
>> but if there was, could someone remind me of it?
>>
>> (I notice that Debian stable, which this laptop runs, only has 3.5).
>>
>> cheers,
>> Douglas
>>
-------------- next part --------------
From b983f42498c82a8698653748ee0f0c8031cd783f Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:12:18 +1300
Subject: [PATCH 1/6] autobuild: Drop py2 autobuild jobs
Samba v4.11 will no longer support python2, so let's drop the autobuild
jobs. This will save some gitlab/sn-devel time and money, as it's less
work for CI to do.
Note that this highlights some previous inconsistencies:
- samba-none-env-py2 was being built for gitlab but not sn-devel.
- samba-nt4-py2 was being built for sn-devel but not gitlab
I've left samba-buildpy2-only for now, which will be addressed in a
subsequent patch.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
.gitlab-ci-private.yml | 8 --------
.gitlab-ci.yml | 13 -------------
script/autobuild.py | 6 ------
3 files changed, 27 deletions(-)
diff --git a/.gitlab-ci-private.yml b/.gitlab-ci-private.yml
index 4a6e503..4e83c19 100644
--- a/.gitlab-ci-private.yml
+++ b/.gitlab-ci-private.yml
@@ -18,10 +18,6 @@ samba:
extends: .private_template
# this one takes about 4 hours to finish
-samba-py2:
- extends: .private_template
- # this one takes about 4 hours to finish
-
samba-fileserver:
extends: .private_template
# this one takes about 1 hours to finish
@@ -30,9 +26,5 @@ samba-ad-dc:
extends: .private_template
# this one takes about 1 hours to finish
-samba-ad-dc-py2:
- extends: .private_template
- # this one takes about 1 hours to finish
-
samba-nt4:
extends: .private_template
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 53b2acf..f0787f8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -61,10 +61,6 @@ samba-none-env:
extends: .shared_template
# this one takes about 1 hours to finish
-samba-none-env-py2:
- extends: .shared_template
- # this one takes about 1 hours to finish
-
samba-nopython:
extends: .shared_template
@@ -84,16 +80,9 @@ samba-ad-dc-2:
samba-ad-dc-backup:
extends: .shared_template
-samba-ad-dc-2-py2:
- extends: .shared_template
- # this one takes about 1 hours to finish
-
samba-libs:
extends: .shared_template
-samba-libs-py2:
- extends: .shared_template
-
samba-static:
extends: .shared_template
@@ -110,5 +99,3 @@ samba-ad-dc-ntvfs:
extends: .shared_template
# this one takes about 100 mins to finish
-samba-ad-dc-ntvfs-py2:
- extends: .shared_template
diff --git a/script/autobuild.py b/script/autobuild.py
index b43adcf..30bc643 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -36,23 +36,17 @@ cleanup_list = []
builddirs = {
"ctdb": "ctdb",
"samba": ".",
- "samba-py2": ".",
"samba-nt4": ".",
- "samba-nt4-py2": ".",
"samba-fileserver": ".",
"samba-xc": ".",
"samba-o3": ".",
"samba-ctdb": ".",
"samba-libs": ".",
- "samba-libs-py2": ".",
"samba-static": ".",
"samba-none-env": ".",
"samba-ad-dc": ".",
- "samba-ad-dc-py2": ".",
"samba-ad-dc-ntvfs": ".",
- "samba-ad-dc-ntvfs-py2": ".",
"samba-ad-dc-2": ".",
- "samba-ad-dc-2-py2": ".",
"samba-ad-dc-backup": ".",
"samba-systemkrb5": ".",
"samba-nopython": ".",
--
2.7.4
From 497dbe80f1ea020a0728533bc7cd876469f02bb2 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:41:34 +1300
Subject: [PATCH 2/6] autobuild: Update variable name to make more sense
When we switched from python2 being the default to python3, we didn't
update this variable name. It's now handling the python2 case, but it's
a boolean flag named 'py3', which is rather confusing.
We're going to delete this code anyway, but renaming the variable first
will mean the next patch will make more sense.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 30bc643..72050e0 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -457,9 +457,9 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
class builder(object):
'''handle build of one directory'''
- def __init__(self, name, sequence, cp=True, py3=False):
+ def __init__(self, name, sequence, cp=True, py2=False):
self.name = name
- self.py3 = py3
+ self.py2 = py2
if name in builddirs:
self.dir = builddirs[name]
else:
@@ -498,7 +498,7 @@ class builder(object):
(self.stage, self.cmd, self.output_mime_type) = self.sequence[self.next]
self.cmd = self.cmd.replace("${PYTHON_PREFIX}", get_python_lib(plat_specific=1, standard_lib=0, prefix=self.prefix))
self.cmd = self.cmd.replace("${PREFIX}", "--prefix=%s" % self.prefix)
- if self.py3:
+ if self.py2:
self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python)
# The trailing space is important
self.cmd = self.cmd.replace("${PY3_ONLY}", "python2 ")
@@ -542,7 +542,7 @@ class buildlist(object):
b = builder(n,
tasks[n[:-4]],
cp=n is not "pidl",
- py3=True)
+ py2=True)
else:
b = builder(n, tasks[n], cp=n is not "pidl")
self.tlist.append(b)
--
2.7.4
From 8704fdd04b07d030cef3ddae018967c39f489b04 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:46:21 +1300
Subject: [PATCH 3/6] autobuild: drop special case for '-py2' jobs
These autobuild jobs no longer exist, so we can remove this extra code
now.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 20 ++++----------------
1 file changed, 4 insertions(+), 16 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 72050e0..3daab96 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -457,9 +457,8 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
class builder(object):
'''handle build of one directory'''
- def __init__(self, name, sequence, cp=True, py2=False):
+ def __init__(self, name, sequence, cp=True):
self.name = name
- self.py2 = py2
if name in builddirs:
self.dir = builddirs[name]
else:
@@ -498,13 +497,8 @@ class builder(object):
(self.stage, self.cmd, self.output_mime_type) = self.sequence[self.next]
self.cmd = self.cmd.replace("${PYTHON_PREFIX}", get_python_lib(plat_specific=1, standard_lib=0, prefix=self.prefix))
self.cmd = self.cmd.replace("${PREFIX}", "--prefix=%s" % self.prefix)
- if self.py2:
- self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python)
- # The trailing space is important
- self.cmd = self.cmd.replace("${PY3_ONLY}", "python2 ")
- else:
- self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "")
- self.cmd = self.cmd.replace("${PY3_ONLY}", "")
+ self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "")
+ self.cmd = self.cmd.replace("${PY3_ONLY}", "")
self.cmd = self.cmd.replace("${PREFIX_DIR}", "%s" % self.prefix)
self.cmd = self.cmd.replace("${TESTS}", options.restrict_tests)
# if self.output_mime_type == "text/x-subunit":
@@ -538,13 +532,7 @@ class buildlist(object):
os.environ['AUTOBUILD_RANDOM_SLEEP_OVERRIDE'] = '1'
for n in tasknames:
- if n not in tasks and n.endswith("-py2"):
- b = builder(n,
- tasks[n[:-4]],
- cp=n is not "pidl",
- py2=True)
- else:
- b = builder(n, tasks[n], cp=n is not "pidl")
+ b = builder(n, tasks[n], cp=n is not "pidl")
self.tlist.append(b)
if options.retry:
rebase_remote = "rebaseon"
--
2.7.4
From 2c8a883b019f4b6f5ec68c0f387a2b2063bb3990 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 12:33:29 +1300
Subject: [PATCH 4/6] autobuild: Replace samba-buildpy2-only with
samba-nopython-py2
For Samba 4.11, the minimum python2 functionality we will support (for
now, at least - we may change our minds) is for the --disable-python
target, i.e. if you're excluding all the python functionality from
samba, then WAF should still support being built with python2.
The use case here is old unix platforms that want to use smbd, but don't
have python3 support.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
.gitlab-ci.yml | 6 +++---
script/autobuild.py | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index f0787f8..5747dae 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -64,6 +64,9 @@ samba-none-env:
samba-nopython:
extends: .shared_template
+samba-nopython-py2:
+ extends: .shared_template
+
samba-systemkrb5:
extends: .shared_template
@@ -92,9 +95,6 @@ ctdb:
samba-ctdb:
extends: .shared_template
-samba-buildpy2-only:
- extends: .shared_template
-
samba-ad-dc-ntvfs:
extends: .shared_template
# this one takes about 100 mins to finish
diff --git a/script/autobuild.py b/script/autobuild.py
index 3daab96..9094cce 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -50,7 +50,7 @@ builddirs = {
"samba-ad-dc-backup": ".",
"samba-systemkrb5": ".",
"samba-nopython": ".",
- "samba-buildpy2-only": ".",
+ "samba-nopython-py2": ".",
"ldb": "lib/ldb",
"tdb": "lib/tdb",
"talloc": "lib/talloc",
@@ -419,8 +419,8 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- "samba-buildpy2-only": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "PYTHON='python' ./configure.developer --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
+ "samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "PYTHON='python' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
("make", "PYTHON='python' make -j", "text/plain"),
("install", "PYTHON='python' make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
--
2.7.4
From 31028c73267ad40ee4b351ab80e4c37418e4b959 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 13:55:09 +1300
Subject: [PATCH 5/6] autobuild: Make python2 usage explicit
This currently assumes 'python' means python2. It makes more sense if we
make the python2 part explicit.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 9094cce..b63b92f 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -420,11 +420,11 @@ tasks = {
("clean", "make clean", "text/plain")],
"samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "PYTHON='python' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
- ("make", "PYTHON='python' make -j", "text/plain"),
- ("install", "PYTHON='python' make install", "text/plain"),
+ ("configure", "PYTHON='python2' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
+ ("make", "PYTHON='python2' make -j", "text/plain"),
+ ("install", "PYTHON='python2' make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
- ("clean", "PYTHON='python' make clean", "text/plain")],
+ ("clean", "PYTHON='python2' make clean", "text/plain")],
# these are useful for debugging autobuild
--
2.7.4
From 8c8eb744e9bb36ee42d4780a271ea0bcd427fdca Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 13 Feb 2019 13:58:08 +1300
Subject: [PATCH 6/6] autobuild: Move samba-nopython-py2 target
Move samba-nopython-py2 so that it's next to samba-nopython.
Note that samba-nopython does extra work because it's checking
that we can build and run all the core components without python.
For samba-nopython-py2, we just want to check we can do the WAF/
make part using python2.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
script/autobuild.py | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index b63b92f..cbb83c8 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -353,6 +353,13 @@ tasks = {
("libs-clean", "make clean", "text/plain")
],
+ # also sanity-check we can do the WAF make part of the above using python2
+ "samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "PYTHON='python2' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
+ ("make", "PYTHON='python2' make -j", "text/plain"),
+ ("install", "PYTHON='python2' make install", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
+ ("clean", "PYTHON='python2' make clean", "text/plain")],
"ldb": [
@@ -419,13 +426,6 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- "samba-nopython-py2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "PYTHON='python2' ./configure --disable-python --without-ad-dc --with-selftest-prefix=./bin/ab " + samba_configure_params, "text/plain"),
- ("make", "PYTHON='python2' make -j", "text/plain"),
- ("install", "PYTHON='python2' make install", "text/plain"),
- ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
- ("clean", "PYTHON='python2' make clean", "text/plain")],
-
# these are useful for debugging autobuild
'pass': [("pass", 'echo passing && /bin/true', "text/plain")],
--
2.7.4
More information about the samba-technical
mailing list