[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Wed Aug 29 06:21:02 UTC 2018
The branch, master has been updated
via 825d67f autobuild: Avoid subshell for tail -f invocation
via 47c14ef autobuild: use close_fds=True to avoid *.stderr and *.stdout inheriting into every process
via acb8cf1 travis-ci: Add py3 jobs to match new jobs in autobuild.py
via 6b11643 autobuild: Move backup/restore environments into to samba-ad-dc-2 autobuild job
via f9e494a autobuild: Move ad_dc_no_nss into to samba-ad-dc-2 autobuild job
via 38784f2 autobuild: Reduce duplication of task list in autobuild
via ce53dd9 gitlab-ci: Run the new python3 autobuild tasks
via 0da15fa autobuild: Implement a split python2 and python2 build pattern
from ffa1c04 examples: Add winexe re-implemented on current Samba libs
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 825d67fd35a809e779b916a82e63df8c8de01772
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Aug 27 21:13:29 2018 +1200
autobuild: Avoid subshell for tail -f invocation
This should mean one less process in the process tree, and less places to hold
FDs open.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13591
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed Aug 29 08:20:55 CEST 2018 on sn-devel-144
commit 47c14ef64e0037ad3e5ac9d8ddf9801501dcdcab
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Aug 27 21:00:58 2018 +1200
autobuild: use close_fds=True to avoid *.stderr and *.stdout inheriting into every process
This closes fds other than 0, 1, 2.
This ensures only the correct *.stderr and *.stdout is attached, via
the stdout/stderr parameter to Popen(), but not every other FD
currently open in python at the time Popen is called.
For the tail invocation and other calls to Popen(), because fds 0, 1,
2 are still attached, these function as before.
Per https://docs.python.org/2.6/library/subprocess.html:
"If close_fds is true, all file descriptors except 0, 1 and
2 will be closed before the child process is executed. (Unix only)."
And regarding the passed in parameters:
"stdin, stdout and stderr specify the executed programs’ standard
input,
standard output and standard error file handles, respectively. "
...
"With None (the default), no redirection will occur;
the child’s file handles will be inherited from the parent. "
(The unwanted inherited files would be on a random high FD, where the
program wouldn't know what to do with them, but counting towards the
process FD limit).
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13591
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
commit acb8cf1eaf7232a994aafc83fc12d723b71d981b
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Aug 27 13:34:35 2018 +1200
travis-ci: Add py3 jobs to match new jobs in autobuild.py
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
commit 6b11643617dcdf75740be9218f83254d8826c14b
Author: Andrew Bartlett <abartlet at samba.org>
Date: Sat Aug 25 09:40:12 2018 +0200
autobuild: Move backup/restore environments into to samba-ad-dc-2 autobuild job
This tries to to split up the tasks more evenly and may help with the python3
work by isolating them from the long samba job.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
commit f9e494a6f2999641f386f658351c539d1ad9edae
Author: Andrew Bartlett <abartlet at samba.org>
Date: Sat Aug 25 09:03:07 2018 +0200
autobuild: Move ad_dc_no_nss into to samba-ad-dc-2 autobuild job
This tries to to split up the tasks more evenly and may help with the python3 tests
against this environment if started from a more isolated job.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
commit 38784f2b371af2471ec08cf38bbcaea004efcce3
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Aug 16 13:54:14 2018 +1200
autobuild: Reduce duplication of task list in autobuild
The defaulttasks or builddirs are often updated out of sync, which causes confusion until
it is resolved.
We simply choose "." as the builddir for the tasks that
are not in the default set.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
commit ce53dd9cf41f195ba784f40f74910891895bf695
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Aug 16 13:53:20 2018 +1200
gitlab-ci: Run the new python3 autobuild tasks
These additional tasks should be less complex than the full build and help get us to
a pure python3 build eventually
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
commit 0da15fa764c61bece2c99e1816ba0e782f2753de
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Aug 16 13:50:31 2018 +1200
autobuild: Implement a split python2 and python2 build pattern
The default tasks will run the tests without --extra-python specified and
the new -py3 tasks will run the python3 tests only.
This will reduce the complexity of the build combinations
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
-----------------------------------------------------------------------
Summary of changes:
.gitlab-ci-private.yml | 18 +++++++++
.gitlab-ci.yml | 17 ++++++++
.travis.yml | 5 +++
script/autobuild.py | 107 ++++++++++++++++++++++++++++++-------------------
4 files changed, 105 insertions(+), 42 deletions(-)
Changeset truncated at 500 lines:
diff --git a/.gitlab-ci-private.yml b/.gitlab-ci-private.yml
index bf045ca..bbbeae2 100644
--- a/.gitlab-ci-private.yml
+++ b/.gitlab-ci-private.yml
@@ -14,12 +14,24 @@ build_samba:
# this one takes about 4 hours to finish
- python script/autobuild.py samba --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_py3:
+ <<: *private_template
+ script:
+ # this one takes about 4 hours to finish
+ - python script/autobuild.py samba-py3 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
build_samba_nt4:
<<: *private_template
script:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-nt4 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_nt4_py3:
+ <<: *private_template
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-nt4-py3 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
build_samba_fileserver:
<<: *private_template
script:
@@ -32,3 +44,9 @@ build_samba_ad_dc:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-ad-dc --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_ad_dc_py3:
+ <<: *private_template
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-ad-dc-py3 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 42a425e..9aa9fb2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -29,6 +29,12 @@ build_samba_none_env:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-none-env --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_none_env_py3:
+ <<: *shared_template
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-none-env-py3 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
build_samba_nopython:
<<: *shared_template
script:
@@ -55,11 +61,22 @@ build_samba_ad_dc_2:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-ad-dc-2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_ad_dc_2_py3:
+ <<: *shared_template
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-ad-dc-2-py3 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
build_samba_libs:
<<: *shared_template
script:
- python script/autobuild.py samba-libs --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_libs_py3:
+ <<: *shared_template
+ script:
+ - python script/autobuild.py samba-libs-py3 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
build_samba_static:
<<: *shared_template
script:
diff --git a/.travis.yml b/.travis.yml
index e58b4a8..3fcd741 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -10,15 +10,20 @@ env:
- TASK=samba-xc
- TASK=samba-ctdb
- TASK=samba-libs
+ - TASK=samba-libs-py3
- TASK=samba-static
- TASK=samba-o3
- TASK=samba-none-env
+ - TASK=samba-none-env-py3
- TASK=samba-nopython
- TASK=samba-systemkrb5
- TASK=samba-nt4
+ - TASK=samba-nt4-py3
- TASK=samba-fileserver
- TASK=samba-ad-dc
+ - TASK=samba-ad-dc-py3
- TASK=samba-ad-dc-2
+ - TASK=samba-ad-dc-2-py3
- TASK=ldb
- TASK=tdb
- TASK=talloc
diff --git a/script/autobuild.py b/script/autobuild.py
index fcb4e4a..fec6409 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -29,17 +29,22 @@ cleanup_list = []
builddirs = {
"ctdb": "ctdb",
"samba": ".",
+ "samba-py3": ".",
"samba-nt4": ".",
+ "samba-nt4-py3": ".",
"samba-fileserver": ".",
"samba-xc": ".",
"samba-o3": ".",
"samba-ctdb": ".",
"samba-libs": ".",
+ "samba-libs-py3": ".",
"samba-static": ".",
- "samba-test-only": ".",
"samba-none-env": ".",
+ "samba-none-env-py3": ".",
"samba-ad-dc": ".",
+ "samba-ad-dc-py3": ".",
"samba-ad-dc-2": ".",
+ "samba-ad-dc-2-py3": ".",
"samba-systemkrb5": ".",
"samba-nopython": ".",
"ldb": "lib/ldb",
@@ -47,32 +52,10 @@ builddirs = {
"talloc": "lib/talloc",
"replace": "lib/replace",
"tevent": "lib/tevent",
- "pidl": "pidl",
- "pass": ".",
- "fail": ".",
- "retry": "."
+ "pidl": "pidl"
}
-defaulttasks = ["ctdb",
- "samba",
- "samba-nt4",
- "samba-fileserver",
- "samba-xc",
- "samba-o3",
- "samba-ctdb",
- "samba-libs",
- "samba-static",
- "samba-none-env",
- "samba-ad-dc",
- "samba-ad-dc-2",
- "samba-systemkrb5",
- "samba-nopython",
- "ldb",
- "tdb",
- "talloc",
- "replace",
- "tevent",
- "pidl"]
+defaulttasks = builddirs.keys()
if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
defaulttasks.remove("samba-o3")
@@ -106,10 +89,12 @@ tasks = {
"samba": [("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", "make test FAIL_IMMEDIATELY=1 "
- "TESTS='--exclude-env=none "
+ "TESTS='${PY3_ONLY}"
+ "--exclude-env=none "
"--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
"--exclude-env=ad_dc "
+ "--exclude-env=ad_dc_no_nss "
"--exclude-env=fl2003dc "
"--exclude-env=fl2008r2dc "
"--exclude-env=ad_member "
@@ -118,7 +103,13 @@ tasks = {
"--exclude-env=chgdcpass "
"--exclude-env=vampire_2000_dc "
"--exclude-env=fl2000dc "
- "--exclude-env=fileserver'",
+ "--exclude-env=fileserver "
+ "--exclude-env=backupfromdc "
+ "--exclude-env=restoredc "
+ "--exclude-env=renamedc "
+ "--exclude-env=offlinebackupdc "
+ "--exclude-env=labdc "
+ "'",
"text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
@@ -128,7 +119,9 @@ tasks = {
"samba-nt4": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --without-ads --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=nt4_dc --include-env=nt4_member'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=nt4_dc --include-env=nt4_member'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
@@ -137,14 +130,17 @@ tasks = {
"samba-fileserver": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --without-ad-dc --without-ldap --without-ads --without-json-audit --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=fileserver'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=fileserver'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
# We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-ad-dc": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='"
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
"--include-env=ad_dc "
"--include-env=fl2003dc "
"--include-env=fl2008r2dc "
@@ -157,7 +153,19 @@ tasks = {
"samba-ad-dc-2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=chgdcpass --include-env=vampire_2000_dc --include-env=fl2000dc'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=chgdcpass "
+ "--include-env=vampire_2000_dc "
+ "--include-env=fl2000dc "
+ "--include-env=ad_dc_no_nss "
+ "--include-env=backupfromdc "
+ "--include-env=restoredc "
+ "--include-env=renamedc "
+ "--include-env=offlinebackupdc "
+ "--include-env=labdc "
+ "'",
+ "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
"samba-test-only": [("configure", "./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
@@ -177,7 +185,9 @@ tasks = {
"samba-o3": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "ADDITIONAL_CFLAGS='-O3' ./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make quicktest FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_dc'", "text/plain"),
+ ("test", "make quicktest FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=ad_dc'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
@@ -237,7 +247,8 @@ tasks = {
("make", "make -j", "text/plain"),
("test", "make test "
"FAIL_IMMEDIATELY=1 "
- "TESTS='--include-env=none'",
+ "TESTS='${PY3_ONLY}"
+ "--include-env=none'",
"text/plain")],
"samba-static": [
@@ -266,7 +277,9 @@ tasks = {
("make", "make -j", "text/plain"),
# we currently cannot run a full make test, a limited list of tests could be run
# via "make test TESTS=sometests"
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=ktest'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=ktest'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")
@@ -394,7 +407,7 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
if show:
do_print("Running: '%s' in '%s'" % (cmd, dir))
if output:
- return Popen([cmd], shell=True, stdout=PIPE, cwd=dir).communicate()[0]
+ return Popen([cmd], shell=True, stdout=PIPE, cwd=dir, close_fds=True).communicate()[0]
elif checkfail:
return check_call(cmd, shell=True, cwd=dir)
else:
@@ -407,7 +420,10 @@ class builder(object):
def __init__(self, name, sequence, cp=True, py3=False):
self.name = name
self.py3 = py3
- self.dir = builddirs[name]
+ if name in builddirs:
+ self.dir = builddirs[name]
+ else:
+ self.dir = "."
self.tag = self.name.replace('/', '_')
self.sequence = sequence
@@ -442,7 +458,13 @@ class builder(object):
(self.stage, self.cmd, self.output_mime_type) = self.sequence[self.next]
self.cmd = self.cmd.replace("${PYTHON_PREFIX}", get_python_lib(standard_lib=1, prefix=self.prefix))
self.cmd = self.cmd.replace("${PREFIX}", "--prefix=%s" % self.prefix)
- self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python)
+ if self.py3:
+ self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "%s" % extra_python)
+ # The trailing space is important
+ self.cmd = self.cmd.replace("${PY3_ONLY}", "python3 ")
+ else:
+ self.cmd = self.cmd.replace("${EXTRA_PYTHON}", "")
+ self.cmd = self.cmd.replace("${PY3_ONLY}", "")
self.cmd = self.cmd.replace("${PREFIX_DIR}", "%s" % self.prefix)
self.cmd = self.cmd.replace("${TESTS}", options.restrict_tests)
# if self.output_mime_type == "text/x-subunit":
@@ -451,6 +473,7 @@ class builder(object):
cwd = os.getcwd()
os.chdir("%s/%s" % (self.sdir, self.dir))
self.proc = Popen(self.cmd, shell=True,
+ close_fds=True,
stdout=self.stdout, stderr=self.stderr, stdin=self.stdin)
os.chdir(cwd)
self.next += 1
@@ -590,11 +613,11 @@ class buildlist(object):
os.unlink(b.stderr_path)
def start_tail(self):
- cwd = os.getcwd()
- cmd = "tail -f *.stdout *.stderr"
- os.chdir(gitroot)
- self.tail_proc = Popen(cmd, shell=True)
- os.chdir(cwd)
+ cmd = ["tail", "-f"]
+ for b in self.tlist:
+ cmd.append(b.stdout_path)
+ cmd.append(b.stderr_path)
+ self.tail_proc = Popen(cmd, close_fds=True)
def cleanup():
--
Samba Shared Repository
More information about the samba-cvs
mailing list