[PATCH] Autobuild in about 3h30m
Andrew Bartlett
abartlet at samba.org
Thu Mar 22 04:29:59 UTC 2018
As you would have followed, I've been playing with getting our
autobuild to be more parallel, based on some great work Jamie did over
the summer to remove the inter-dependencies.
I've tested an earlier version of this on sn-devel (twice) and you can
see the CI status below:
https://gitlab.com/samba-team/samba/merge_requests/3
This is the same code as:
https://github.com/samba-team/samba/pull/148
Please carefully review as I've changed the environments some tests run
against to balance the test load.
Thanks,
Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team https://samba.org
Samba Development and Support, Catalyst IT
https://catalyst.net.nz/services/samba
-------------- next part --------------
From c999ad6c29e71c494411ead9e58d4e997dd87895 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Tue, 20 Mar 2018 14:15:47 +1300
Subject: [PATCH 01/38] gitlab-ci: Create swap space to work around the 2G
image
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 2ae9eb4032d..c02c02e681c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,7 +1,12 @@
# see https://docs.gitlab.com/ce/ci/yaml/README.html for all available options
before_script:
- - echo "Build starting ..."
+ - echo "Build starting (preparing swap)..."
+ - if [ $(df -m / --output=avail | tail -n1) -gt 10240 ]; then
+ sudo dd if=/dev/zero of=/samba-swap bs=1M count=6144;
+ sudo mkswap /samba-swap;
+ sudo swapon /samba-swap;
+ fi
build_samba:
stage: build
--
2.11.0
From ebfc71bf66e6bb84fbd32599e04db6edac1adaeb Mon Sep 17 00:00:00 2001
From: Joe Guo <joeg at catalyst.net.nz>
Date: Tue, 28 Nov 2017 10:20:54 +1300
Subject: [PATCH 02/38] gitlab-ci: set docker image and change tag
1. Specify docker image to use in gitlab-ci.yml
2. Change tag autobuild to docker
So we can use gitlab.com shared runners.
Signed-off-by: Joe Guo <joeg at catalyst.net.nz>
---
.gitlab-ci.yml | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c02c02e681c..ed6382241a7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,7 @@
# see https://docs.gitlab.com/ce/ci/yaml/README.html for all available options
+image: registry.gitlab.com/catalyst-samba/samba:latest
+
before_script:
- echo "Build starting (preparing swap)..."
- if [ $(df -m / --output=avail | tail -n1) -gt 10240 ]; then
@@ -11,7 +13,7 @@ before_script:
build_samba:
stage: build
tags:
- - autobuild
+ - docker
script:
# this one takes about 4 hours to finish
- python script/autobuild.py samba --verbose --tail --testbase /tmp/samba-testbase
@@ -19,7 +21,7 @@ build_samba:
build_samba_others:
stage: build
tags:
- - autobuild
+ - docker
script:
- python script/autobuild.py samba-nopython --verbose --tail --testbase /tmp/samba-testbase
- python script/autobuild.py samba-systemkrb5 --verbose --tail --testbase /tmp/samba-testbase
@@ -31,7 +33,7 @@ build_samba_others:
build_ctdb:
stage: build
tags:
- - autobuild
+ - docker
script:
- python script/autobuild.py samba-ctdb --verbose --tail --testbase /tmp/samba-testbase
- python script/autobuild.py ctdb --verbose --tail --testbase /tmp/samba-testbase
@@ -39,7 +41,7 @@ build_ctdb:
build_others:
stage: build
tags:
- - autobuild
+ - docker
script:
- python script/autobuild.py ldb --verbose --tail --testbase /tmp/samba-testbase
- python script/autobuild.py pidl --verbose --tail --testbase /tmp/samba-testbase
--
2.11.0
From d56412380f9f96b6d78958a8cb88715bae081d87 Mon Sep 17 00:00:00 2001
From: Joe Guo <joeg at catalyst.net.nz>
Date: Tue, 12 Dec 2017 14:26:07 +1300
Subject: [PATCH 03/38] gitlab-ci: add git variable to speed up clone
Set GIT_STATEGY to fetch, and GIT_DEPTH to 3.
This will speed up cloning for repos.
Signed-off-by: Joe Guo <joeg at catalyst.net.nz>
---
.gitlab-ci.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ed6382241a7..7d677260128 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,6 +2,10 @@
image: registry.gitlab.com/catalyst-samba/samba:latest
+variables:
+ GIT_STRATEGY: fetch
+ GIT_DEPTH: "3"
+
before_script:
- echo "Build starting (preparing swap)..."
- if [ $(df -m / --output=avail | tail -n1) -gt 10240 ]; then
--
2.11.0
From bad115445a7b44798e0ea2445ec273bf1a55081e Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 21 Jul 2017 19:54:36 +1200
Subject: [PATCH 04/38] autobuild: Move defaulttasks to one-per-line
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index ebe49bb75cb..fe1a96104fe 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -44,7 +44,21 @@ builddirs = {
"retry" : "."
}
-defaulttasks = [ "ctdb", "samba", "samba-xc", "samba-o3", "samba-ctdb", "samba-libs", "samba-static", "samba-systemkrb5", "samba-nopython", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ]
+defaulttasks = [ "ctdb",
+ "samba",
+ "samba-xc",
+ "samba-o3",
+ "samba-ctdb",
+ "samba-libs",
+ "samba-static",
+ "samba-systemkrb5",
+ "samba-nopython",
+ "ldb",
+ "tdb",
+ "talloc",
+ "replace",
+ "tevent",
+ "pidl" ]
if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
defaulttasks.remove("samba-o3")
--
2.11.0
From 555f7e756761323673f8a67b6832ff373726811b Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 30 Jun 2017 11:15:40 +1200
Subject: [PATCH 05/38] autobuild: Run nt4_dc and nt4_member tests in parallel
These do not interact with the main AD DC environments, so can run in parallel
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index fe1a96104fe..3ccbd255353 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -25,6 +25,7 @@ cleanup_list = []
builddirs = {
"ctdb" : "ctdb",
"samba" : ".",
+ "samba-nt4" : ".",
"samba-xc" : ".",
"samba-o3" : ".",
"samba-ctdb" : ".",
@@ -46,6 +47,7 @@ builddirs = {
defaulttasks = [ "ctdb",
"samba",
+ "samba-nt4",
"samba-xc",
"samba-o3",
"samba-ctdb",
@@ -87,14 +89,22 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
- # We have 'test' before 'install' because, 'test' should work without 'install'
+ # We have 'test' before 'install' because, 'test' should work without 'install (runs ad_dc_ntvfs and all the other envs)'
"samba" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--exclude-env=nt4_dc --exclude-env=nt4_member'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
+ # We split out this so the isolated nt4_dc tests do not wait for ad_dc or ad_dc_ntvfs tests (which are long)
+ "samba-nt4" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=nt4_dc --include-env=nt4_member'", "text/plain"),
+ ("install", "make install", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
+ ("clean", "make clean", "text/plain") ],
+
"samba-test-only" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", 'make test FAIL_IMMEDIATELY=1 TESTS="${TESTS}"',"text/plain") ],
--
2.11.0
From 87fcd30447e5e8e8ae41a7e46ec76f6b296453a3 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 30 Jun 2017 11:44:58 +1200
Subject: [PATCH 06/38] travis-ci: Run new samba-nt4 environment
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.travis.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.travis.yml b/.travis.yml
index 645658b9e7d..db0e00f66df 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,6 +14,7 @@ env:
- TASK=samba-o3
- TASK=samba-nopython
- TASK=samba-systemkrb5
+ - TASK=samba-nt4
- TASK=ldb
- TASK=tdb
- TASK=talloc
--
2.11.0
From 1fc9d2b804d5b5d151520fd25eb48f27fcfce732 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Tue, 5 Dec 2017 11:34:08 +1300
Subject: [PATCH 07/38] gitlab-ci: Add samba-nt4 environment to the CI
This parallel build is de-coupled from the main samba build
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7d677260128..6935c69255f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -22,6 +22,14 @@ build_samba:
# this one takes about 4 hours to finish
- python script/autobuild.py samba --verbose --tail --testbase /tmp/samba-testbase
+build_samba_nt4:
+ stage: build
+ tags:
+ - autobuild
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-nt4 --verbose --tail --testbase /tmp/samba-testbase
+
build_samba_others:
stage: build
tags:
--
2.11.0
From 8b6f980d5ca3d9f72df2e9d9f712fdd401b8a11b Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sat, 3 Mar 2018 19:51:29 +1300
Subject: [PATCH 08/38] autobuild: Move "none" environment to samba-none-env
This takes this part of the test out of the main, slow samba task
but also keeps it away from samba-o3 which is up against the 50min
budget on travis-ci.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 8 ++++++++
.travis.yml | 1 +
script/autobuild.py | 17 ++++++++++++++++-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6935c69255f..8b824c5660c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -30,6 +30,14 @@ build_samba_nt4:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-nt4 --verbose --tail --testbase /tmp/samba-testbase
+build_samba_none_env:
+ stage: build
+ tags:
+ - autobuild
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-none-env --verbose --tail --testbase /tmp/samba-testbase
+
build_samba_others:
stage: build
tags:
diff --git a/.travis.yml b/.travis.yml
index db0e00f66df..fb9d4565f22 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -12,6 +12,7 @@ env:
- TASK=samba-libs
- TASK=samba-static
- TASK=samba-o3
+ - TASK=samba-none-env
- TASK=samba-nopython
- TASK=samba-systemkrb5
- TASK=samba-nt4
diff --git a/script/autobuild.py b/script/autobuild.py
index 3ccbd255353..5b26db82e9c 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -32,6 +32,7 @@ builddirs = {
"samba-libs" : ".",
"samba-static" : ".",
"samba-test-only" : ".",
+ "samba-none-env" : ".",
"samba-systemkrb5" : ".",
"samba-nopython" : ".",
"ldb" : "lib/ldb",
@@ -53,6 +54,7 @@ defaulttasks = [ "ctdb",
"samba-ctdb",
"samba-libs",
"samba-static",
+ "samba-none-env",
"samba-systemkrb5",
"samba-nopython",
"ldb",
@@ -92,7 +94,11 @@ tasks = {
# We have 'test' before 'install' because, 'test' should work without 'install (runs ad_dc_ntvfs and all the other envs)'
"samba" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--exclude-env=nt4_dc --exclude-env=nt4_member'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='--exclude-env=nt4_dc "
+ "--exclude-env=nt4_member "
+ "--exclude-env=none'",
+ "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
@@ -175,6 +181,15 @@ tasks = {
("allshared-configure", samba_libs_configure_samba + " --with-shared-modules=ALL", "text/plain"),
("allshared-make", "make -j", "text/plain")],
+ "samba-none-env" : [
+ ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test "
+ "FAIL_IMMEDIATELY=1 "
+ "TESTS='--include-env=none'",
+ "text/plain")],
+
"samba-static" : [
("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
# build with all modules static
--
2.11.0
From f3b18868290efcc3fc1b7c765020a31b0678461f Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sat, 3 Mar 2018 21:03:11 +1300
Subject: [PATCH 09/38] selftest: Do not run smb2.notify against nt4_dc and
ad_dc
This is a slow test and we need to keep the time on ad_dc down to below 50mins total
for travis-ci.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/selftest/tests.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/source3/selftest/tests.py b/source3/selftest/tests.py
index c93abf5e632..a65edcea713 100755
--- a/source3/selftest/tests.py
+++ b/source3/selftest/tests.py
@@ -519,7 +519,6 @@ for t in tests:
plansmbtorture4testsuite(t, "simpleserver", '//$SERVER/tmp -U$USERNAME%$PASSWORD')
elif t == "smb2.notify":
plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/tmp -U$USERNAME%$PASSWORD --signing=required')
- plansmbtorture4testsuite(t, "ad_dc", '//$SERVER/tmp -U$USERNAME%$PASSWORD --signing=required')
elif t == "smb2.dosmode":
plansmbtorture4testsuite(t, "simpleserver", '//$SERVER/dosmode -U$USERNAME%$PASSWORD')
elif t == "smb2.kernel-oplocks":
--
2.11.0
From be205ffe99a7e4d67cc564c9cd22a469a3f56de2 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 30 Jun 2017 11:13:55 +1200
Subject: [PATCH 10/38] autobuild: Run all "ad_dc" environment tests in
samba-ad-dc
This allows us not to run ad_dc tests in the main build, making the
autobuild process faster. The ad_dc tests run in less than 50mins
on travis-ci, which allows this part of the tests to be run.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 8 ++++++++
.travis.yml | 1 +
script/autobuild.py | 10 ++++++++++
3 files changed, 19 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8b824c5660c..aff970c7ba5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -30,6 +30,14 @@ build_samba_nt4:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-nt4 --verbose --tail --testbase /tmp/samba-testbase
+build_samba_ad_dc:
+ stage: build
+ tags:
+ - autobuild
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-ad-dc --verbose --tail --testbase /tmp/samba-testbase
+
build_samba_none_env:
stage: build
tags:
diff --git a/.travis.yml b/.travis.yml
index fb9d4565f22..61b1d8770d7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,6 +16,7 @@ env:
- TASK=samba-nopython
- TASK=samba-systemkrb5
- TASK=samba-nt4
+ - TASK=samba-ad-dc
- TASK=ldb
- TASK=tdb
- TASK=talloc
diff --git a/script/autobuild.py b/script/autobuild.py
index 5b26db82e9c..faec539a3a5 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -33,6 +33,7 @@ builddirs = {
"samba-static" : ".",
"samba-test-only" : ".",
"samba-none-env" : ".",
+ "samba-ad-dc" : ".",
"samba-systemkrb5" : ".",
"samba-nopython" : ".",
"ldb" : "lib/ldb",
@@ -55,6 +56,7 @@ defaulttasks = [ "ctdb",
"samba-libs",
"samba-static",
"samba-none-env",
+ "samba-ad-dc",
"samba-systemkrb5",
"samba-nopython",
"ldb",
@@ -97,6 +99,7 @@ tasks = {
("test", "make test FAIL_IMMEDIATELY=1 "
"TESTS='--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
+ "--exclude-env=ad_dc "
"--exclude-env=none'",
"text/plain"),
("install", "make install", "text/plain"),
@@ -111,6 +114,13 @@ tasks = {
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
+ # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
+ "samba-ad-dc" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_dc'", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+
"samba-test-only" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", 'make test FAIL_IMMEDIATELY=1 TESTS="${TESTS}"',"text/plain") ],
--
2.11.0
From e9d3e6c5e63fe1374dcd27af1f0d8bab1874a97a Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sat, 3 Mar 2018 21:24:47 +1300
Subject: [PATCH 11/38] travis-ci: Only un-shallow for PIDL
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.travis.yml | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/.travis.yml b/.travis.yml
index 61b1d8770d7..2ab086bffb5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,5 +33,7 @@ before_install:
- sudo apt-get install --assume-yes acl attr autoconf bind9utils bison build-essential debhelper dnsutils docbook-xml docbook-xsl flex gdb libjansson-dev krb5-user libacl1-dev libaio-dev libarchive-dev libattr1-dev libblkid-dev libbsd-dev libcap-dev libcups2-dev libgnutls-dev libgpgme11-dev libjson-perl libldap2-dev libncurses5-dev libpam0g-dev libparse-yapp-perl libpopt-dev libreadline-dev nettle-dev perl perl-modules pkg-config python-all-dev python-crypto python-dbg python-dev python-dnspython python3-dnspython python-gpgme python3-gpgme python-markdown python3-markdown python3-dev xsltproc zlib1g-dev
script:
- - git fetch --unshallow
+ - if [ $TASK = "pidl" ]; then
+ git fetch --unshallow;
+ fi
- ./script/autobuild.py --tail --testbase=/tmp $TASK
--
2.11.0
From 1fb239aa4588d623529fa281c172e89d2a1dc6e6 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sat, 3 Mar 2018 23:39:24 +1300
Subject: [PATCH 12/38] travis-ci: Use Gold linker for faster builds
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.travis.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.travis.yml b/.travis.yml
index 2ab086bffb5..bfd60f5588e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -31,6 +31,10 @@ matrix:
before_install:
- sudo apt-get update -qq
- sudo apt-get install --assume-yes acl attr autoconf bind9utils bison build-essential debhelper dnsutils docbook-xml docbook-xsl flex gdb libjansson-dev krb5-user libacl1-dev libaio-dev libarchive-dev libattr1-dev libblkid-dev libbsd-dev libcap-dev libcups2-dev libgnutls-dev libgpgme11-dev libjson-perl libldap2-dev libncurses5-dev libpam0g-dev libparse-yapp-perl libpopt-dev libreadline-dev nettle-dev perl perl-modules pkg-config python-all-dev python-crypto python-dbg python-dev python-dnspython python3-dnspython python-gpgme python3-gpgme python-markdown python3-markdown python3-dev xsltproc zlib1g-dev
+ - sudo apt-get install --assume-yes binutils-gold
+ - sudo update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.gold" 20
+ - sudo update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10
+ - sudo update-alternatives --set ld /usr/bin/ld.gold
script:
- if [ $TASK = "pidl" ]; then
--
2.11.0
From 7682b606d265c9b44ea000fa19026b02c2181add Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sun, 4 Mar 2018 21:32:09 +1300
Subject: [PATCH 13/38] autobuild: Remove fileserver tests from the main build
Again, this is to allow these to run in the 50min timelimit
of travis-ci and so gain test coverage.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 8 ++++++++
.travis.yml | 1 +
script/autobuild.py | 12 +++++++++++-
3 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index aff970c7ba5..ee62aa10563 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -30,6 +30,14 @@ build_samba_nt4:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-nt4 --verbose --tail --testbase /tmp/samba-testbase
+build_samba_fileserver:
+ stage: build
+ tags:
+ - autobuild
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-fileserver --verbose --tail --testbase /tmp/samba-testbase
+
build_samba_ad_dc:
stage: build
tags:
diff --git a/.travis.yml b/.travis.yml
index bfd60f5588e..deca01d6f5a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,6 +16,7 @@ env:
- TASK=samba-nopython
- TASK=samba-systemkrb5
- TASK=samba-nt4
+ - TASK=samba-fileserver
- TASK=samba-ad-dc
- TASK=ldb
- TASK=tdb
diff --git a/script/autobuild.py b/script/autobuild.py
index faec539a3a5..56cd4375898 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -26,6 +26,7 @@ builddirs = {
"ctdb" : "ctdb",
"samba" : ".",
"samba-nt4" : ".",
+ "samba-fileserver" : ".",
"samba-xc" : ".",
"samba-o3" : ".",
"samba-ctdb" : ".",
@@ -50,6 +51,7 @@ builddirs = {
defaulttasks = [ "ctdb",
"samba",
"samba-nt4",
+ "samba-fileserver",
"samba-xc",
"samba-o3",
"samba-ctdb",
@@ -100,7 +102,8 @@ tasks = {
"TESTS='--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
"--exclude-env=ad_dc "
- "--exclude-env=none'",
+ "--exclude-env=none "
+ "--exclude-env=fileserver'",
"text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
@@ -115,6 +118,13 @@ tasks = {
("clean", "make clean", "text/plain") ],
# We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
+ "samba-fileserver" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=fileserver'", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+
+ # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-ad-dc" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
--
2.11.0
From 9fd227069bdf471710bc9581054ecf72c75c1e5d Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sun, 4 Mar 2018 21:20:31 +1300
Subject: [PATCH 14/38] selftest: Move base.delaywrite tests to fileserver
environment
This aims to keep the ad_dc tests well below 50mins for travis CI and
base.delaywrite is very slow.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/selftest/tests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source3/selftest/tests.py b/source3/selftest/tests.py
index a65edcea713..9ac8233aa6c 100755
--- a/source3/selftest/tests.py
+++ b/source3/selftest/tests.py
@@ -405,7 +405,7 @@ tests= base + raw + smb2 + rpc + unix + local + rap + nbt + libsmbclient + idmap
for t in tests:
if t == "base.delaywrite":
- plansmbtorture4testsuite(t, "ad_dc", '//$SERVER/tmp -U$USERNAME%$PASSWORD -k yes --maximum-runtime=900')
+ plansmbtorture4testsuite(t, "fileserver", '//$SERVER/tmp -U$USERNAME%$PASSWORD --maximum-runtime=900')
elif t == "base.createx_access":
plansmbtorture4testsuite(t, "ad_dc", '//$SERVER/tmp -U$USERNAME%$PASSWORD -k yes --maximum-runtime=900')
elif t == "rap.sam":
--
2.11.0
From b745e27eee3a0aad6dc2287f97ba35694a9f77b0 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sun, 4 Mar 2018 21:33:50 +1300
Subject: [PATCH 15/38] Move smbtorture3 tests to fileserver environment
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/selftest/tests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source3/selftest/tests.py b/source3/selftest/tests.py
index 9ac8233aa6c..393615cb77c 100755
--- a/source3/selftest/tests.py
+++ b/source3/selftest/tests.py
@@ -85,7 +85,7 @@ tests = ["FDPASS", "LOCK1", "LOCK2", "LOCK3", "LOCK4", "LOCK5", "LOCK6", "LOCK7"
"BAD-NBT-SESSION"]
for t in tests:
- plantestsuite("samba3.smbtorture_s3.plain(nt4_dc).%s" % t, "nt4_dc", [os.path.join(samba3srcdir, "script/tests/test_smbtorture_s3.sh"), t, '//$SERVER_IP/tmp', '$USERNAME', '$PASSWORD', smbtorture3, "", "-l $LOCAL_PATH"])
+ plantestsuite("samba3.smbtorture_s3.plain(fileserver).%s" % t, "fileserver", [os.path.join(samba3srcdir, "script/tests/test_smbtorture_s3.sh"), t, '//$SERVER_IP/tmp', '$USERNAME', '$PASSWORD', smbtorture3, "", "-l $LOCAL_PATH"])
plantestsuite("samba3.smbtorture_s3.crypt_client(nt4_dc).%s" % t, "nt4_dc", [os.path.join(samba3srcdir, "script/tests/test_smbtorture_s3.sh"), t, '//$SERVER_IP/tmp', '$USERNAME', '$PASSWORD', smbtorture3, "-e", "-l $LOCAL_PATH"])
if t == "TORTURE":
# this is a negative test to verify that the server rejects
--
2.11.0
From 8d6d8904a10ebf9ab2ba533f3e2b6fd53640a1ac Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sun, 4 Mar 2018 21:36:22 +1300
Subject: [PATCH 16/38] autobuild: Try and test different configure options for
new environments
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 56cd4375898..77c01177bb9 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -119,7 +119,7 @@ tasks = {
# We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-fileserver" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
- ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("configure", "./configure.developer --without-ad-dc --without-ads --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=fileserver'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
--
2.11.0
From e556821ca97463445be474f13c37e29ac39295a6 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sun, 4 Mar 2018 22:31:00 +1300
Subject: [PATCH 17/38] libsmb: Use the same #ifdef for is_our_primary_domain()
as the only caller
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/libsmb/namequery_dc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source3/libsmb/namequery_dc.c b/source3/libsmb/namequery_dc.c
index eb34741defa..d08456590f6 100644
--- a/source3/libsmb/namequery_dc.c
+++ b/source3/libsmb/namequery_dc.c
@@ -32,7 +32,7 @@
Is this our primary domain ?
**********************************************************************/
-#ifdef HAVE_KRB5
+#ifdef HAVE_ADS
static bool is_our_primary_domain(const char *domain)
{
int role = lp_server_role();
--
2.11.0
From 77182f631bd504d1787dce033ea5c7ce388d8ca4 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Sun, 4 Mar 2018 22:57:22 +1300
Subject: [PATCH 18/38] s3-libnet: move rpc_join label into HAVE_ADS block with
only caller
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/libnet/libnet_join.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/source3/libnet/libnet_join.c b/source3/libnet/libnet_join.c
index 5db2ca09ccc..bbe604b729e 100644
--- a/source3/libnet/libnet_join.c
+++ b/source3/libnet/libnet_join.c
@@ -2665,9 +2665,10 @@ static WERROR libnet_DomainJoin(TALLOC_CTX *mem_ctx,
DEBUG(5, ("failed to precreate account in ou %s: %s",
r->in.account_ou, ads_errstr(ads_status)));
}
+ rpc_join:
+
#endif /* HAVE_ADS */
- rpc_join:
if ((r->in.join_flags & WKSSVC_JOIN_FLAGS_JOIN_UNSECURE) &&
(r->in.join_flags & WKSSVC_JOIN_FLAGS_MACHINE_PWD_PASSED)) {
status = libnet_join_joindomain_rpc_unsecure(mem_ctx, r, cli);
--
2.11.0
From 6c8096463416f71362783be5160ca543643088e9 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Mon, 5 Mar 2018 08:04:22 +1300
Subject: [PATCH 19/38] selftest: Move slower base.deny1 and base.deny2 to
fileserver environment
This avoids these running in the ad_dc environment which we need to get under 50mins for
travis-ci.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/selftest/tests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source3/selftest/tests.py b/source3/selftest/tests.py
index 393615cb77c..00e04318c7c 100755
--- a/source3/selftest/tests.py
+++ b/source3/selftest/tests.py
@@ -404,7 +404,7 @@ vfs = ["vfs.fruit", "vfs.acl_xattr", "vfs.fruit_netatalk", "vfs.fruit_file_id",
tests= base + raw + smb2 + rpc + unix + local + rap + nbt + libsmbclient + idmap + vfs
for t in tests:
- if t == "base.delaywrite":
+ if t == "base.delaywrite" or t == "base.deny1" or t == "base.deny2":
plansmbtorture4testsuite(t, "fileserver", '//$SERVER/tmp -U$USERNAME%$PASSWORD --maximum-runtime=900')
elif t == "base.createx_access":
plansmbtorture4testsuite(t, "ad_dc", '//$SERVER/tmp -U$USERNAME%$PASSWORD -k yes --maximum-runtime=900')
--
2.11.0
From 1db8770c0d694c76730055a68b24f88313cafb79 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Mon, 5 Mar 2018 17:08:51 +1300
Subject: [PATCH 20/38] selftest: Move samba.tests.samba_tool{.dnscmd,.sites}
to chgdcpass
This helps reduce the runtime of ad_dc which needs to be under 50mins
including build time to run on travis-ci.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source4/selftest/tests.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/source4/selftest/tests.py b/source4/selftest/tests.py
index dc520e9d860..5c5c73caaf7 100755
--- a/source4/selftest/tests.py
+++ b/source4/selftest/tests.py
@@ -611,8 +611,9 @@ planpythontestsuite("ad_dc:local", "samba.tests.samba_tool.ntacl")
planpythontestsuite("none", "samba.tests.samba_tool.provision_password_check")
planpythontestsuite("none", "samba.tests.samba_tool.help")
-planpythontestsuite("ad_dc:local", "samba.tests.samba_tool.sites")
-planpythontestsuite("ad_dc:local", "samba.tests.samba_tool.dnscmd")
+# Run these against chgdcpass to share the runtime load
+planpythontestsuite("chgdcpass:local", "samba.tests.samba_tool.sites")
+planpythontestsuite("chgdcpass:local", "samba.tests.samba_tool.dnscmd")
planpythontestsuite("ad_dc_ntvfs:local", "samba.tests.dcerpc.rpcecho")
planoldpythontestsuite("ad_dc_ntvfs:local", "samba.tests.dcerpc.registry", extra_args=['-U"$USERNAME%$PASSWORD"'])
--
2.11.0
From 2409fb79942a2b6787aa338d0aac3630144288fa Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Tue, 20 Mar 2018 15:07:03 +1300
Subject: [PATCH 21/38] gitlab-ci: Set shared and private tags to allow builds
that need ext4 to pass
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ee62aa10563..a4354dad63c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -18,6 +18,7 @@ build_samba:
stage: build
tags:
- docker
+ - private
script:
# this one takes about 4 hours to finish
- python script/autobuild.py samba --verbose --tail --testbase /tmp/samba-testbase
@@ -25,7 +26,8 @@ build_samba:
build_samba_nt4:
stage: build
tags:
- - autobuild
+ - docker
+ - private
script:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-nt4 --verbose --tail --testbase /tmp/samba-testbase
@@ -33,7 +35,8 @@ build_samba_nt4:
build_samba_fileserver:
stage: build
tags:
- - autobuild
+ - docker
+ - shared
script:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-fileserver --verbose --tail --testbase /tmp/samba-testbase
@@ -41,7 +44,8 @@ build_samba_fileserver:
build_samba_ad_dc:
stage: build
tags:
- - autobuild
+ - docker
+ - private
script:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-ad-dc --verbose --tail --testbase /tmp/samba-testbase
@@ -49,7 +53,8 @@ build_samba_ad_dc:
build_samba_none_env:
stage: build
tags:
- - autobuild
+ - docker
+ - shared
script:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-none-env --verbose --tail --testbase /tmp/samba-testbase
@@ -58,6 +63,7 @@ build_samba_others:
stage: build
tags:
- docker
+ - shared
script:
- python script/autobuild.py samba-nopython --verbose --tail --testbase /tmp/samba-testbase
- python script/autobuild.py samba-systemkrb5 --verbose --tail --testbase /tmp/samba-testbase
@@ -70,6 +76,7 @@ build_ctdb:
stage: build
tags:
- docker
+ - shared
script:
- python script/autobuild.py samba-ctdb --verbose --tail --testbase /tmp/samba-testbase
- python script/autobuild.py ctdb --verbose --tail --testbase /tmp/samba-testbase
@@ -78,6 +85,7 @@ build_others:
stage: build
tags:
- docker
+ - shared
script:
- python script/autobuild.py ldb --verbose --tail --testbase /tmp/samba-testbase
- python script/autobuild.py pidl --verbose --tail --testbase /tmp/samba-testbase
--
2.11.0
From 40a906aa1305a88e3c5b38c54cd9f8ab831908ce Mon Sep 17 00:00:00 2001
From: Jamie McClymont <jamiemcclymont at catalyst.net.nz>
Date: Mon, 29 Jan 2018 18:59:34 +1300
Subject: [PATCH 22/38] selftest: Clear environment before provision
Currently, if an environment is being provisioned after a test which used
ad_member_rfc2307, the provisioning process has all of the following in its
environment:
{
'DC_NETBIOSNAME' => 'LOCALDC',
'DC_PASSWORD' => 'locDCpass1',
'DC_SERVER_IP' => '127.0.0.21',
'DC_SERVER_IPV6' => 'fd00:0000:0000:0000:0000:0000:5357:5f15',
'DC_SERVER' => 'localdc',
'DC_USERNAME' => 'Administrator',
'DOMAIN' => 'SAMBADOMAIN',
'LOCAL_PATH' => '/.../st/ad_member_rfc2307/share',
'LOCK_DIR' => '/.../st/ad_member_rfc2307/lockdir',
'NETBIOSNAME' => 'RFC2307MEMBER',
'NMBD_SOCKET_DIR' => '/.../st/ad_member_rfc2307/nmbd',
'NSS_WRAPPER_GROUP' => '/.../st/ad_member_rfc2307/private/group',
'NSS_WRAPPER_HOSTNAME' => 'rfc2307member.samba.example.com',
'NSS_WRAPPER_HOSTS' => '/.../st/hosts',
'NSS_WRAPPER_MODULE_FN_PREFIX' => 'winbind',
'NSS_WRAPPER_MODULE_SO_PATH' => '/.../bin/default/nsswitch/libnss-wrapper-winbind.so',
'NSS_WRAPPER_PASSWD' => '/.../st/ad_member_rfc2307/private/passwd',
'PASSWORD' => 'loCalMemberPass',
'REALM' => 'SAMBA.EXAMPLE.COM',
'RESOLV_WRAPPER_HOSTS' => '/.../st/dns_host_file',
'SELFTEST_WINBINDD_SOCKET_DIR' => '/.../st/ad_member_rfc2307/winbindd',
'SERVER_IP' => '127.0.0.34',
'SERVER_IPV6' => 'fd00:0000:0000:0000:0000:0000:5357:5f22',
'SERVER' => 'RFC2307MEMBER',
'USERID' => '55668',
'USERNAME' => 'jamiemcclymont',
}
Unsurprisingly, some of these can cause issues for the provisioning process, if
a reduced subset of tests is being run which causes the provision to encounter
never-before-seen pairs of adjacent environments.
For example, a run with only
TESTS='--include-env=vampire_dc --include-env=ad_member_rfc2307'
would fail to start up the vampire_dc with:
Could not find machine account in secrets database:
Failed to fetch machine account password from secrets.ldb:
Could not find entry to match filter:
'(&(flatname=SAMBADOMAIN)(objectclass=primaryDomain))' base: 'cn=Primary Domains': No such object: dsdb_search at ../source4/dsdb/common/util.c:4641
Signed-off-by: Jamie McClymont <jamiemcclymont at catalyst.net.nz>
---
selftest/selftest.pl | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/selftest/selftest.pl b/selftest/selftest.pl
index 42c1e62736f..a772613b8e1 100755
--- a/selftest/selftest.pl
+++ b/selftest/selftest.pl
@@ -942,6 +942,19 @@ sub setup_env($$)
$option = "client" if $option eq "";
+ # Initially clear out the environment for the provision, so previous envs'
+ # variables don't leak in. Provisioning steps must explicitly set their
+ # necessary variables when calling out to other executables
+ foreach (@exported_envvars) {
+ unless ($_ == "NSS_WRAPPER_HOSTS" ||
+ $_ == "RESOLV_WRAPPER_HOSTS")
+ {
+ delete $ENV{$_};
+ }
+ }
+ delete $ENV{SOCKET_WRAPPER_DEFAULT_IFACE};
+ delete $ENV{SMB_CONF_PATH};
+
$ENV{KRB5CCNAME} = "FILE:${selftest_krbt_ccache_path}.${envname}/ignore";
if (defined(get_running_env($envname))) {
--
2.11.0
From c3e01f7e6040c3c219e926a89ddb0b8b7620c345 Mon Sep 17 00:00:00 2001
From: Jamie McClymont <jamiemcclymont at catalyst.net.nz>
Date: Wed, 10 Jan 2018 13:28:13 +1300
Subject: [PATCH 23/38] s4:selftest: explicitly set NSS/RESOLV_WAPPER_* in
wait_for_start
These variables were previously set directly on the selftest process
for the purpose of making this ldbsearch call, allowing them to leak
into other environments.
Signed-off-by: Jamie McClymont <jamiemcclymont at catalyst.net.nz>
Reviewed-by: Gary Lockyer <gary at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
---
selftest/target/Samba4.pm | 24 +++++++++++++++---------
1 file changed, 15 insertions(+), 9 deletions(-)
diff --git a/selftest/target/Samba4.pm b/selftest/target/Samba4.pm
index 842cd1e1bb6..51a175b25e8 100755
--- a/selftest/target/Samba4.pm
+++ b/selftest/target/Samba4.pm
@@ -215,14 +215,6 @@ sub wait_for_start($$)
# Ensure we have the first RID Set before we start tests. This makes the tests more reliable.
if ($testenv_vars->{SERVER_ROLE} eq "domain controller") {
- # Add hosts file for name lookups
- $ENV{NSS_WRAPPER_HOSTS} = $testenv_vars->{NSS_WRAPPER_HOSTS};
- if (defined($testenv_vars->{RESOLV_WRAPPER_CONF})) {
- $ENV{RESOLV_WRAPPER_CONF} = $testenv_vars->{RESOLV_WRAPPER_CONF};
- } else {
- $ENV{RESOLV_WRAPPER_HOSTS} = $testenv_vars->{RESOLV_WRAPPER_HOSTS};
- }
-
print "waiting for working LDAP and a RID Set to be allocated\n";
my $ldbsearch = Samba::bindir_path($self, "ldbsearch");
my $count = 0;
@@ -234,7 +226,21 @@ sub wait_for_start($$)
$search_dn = "cn=RID Set,cn=$testenv_vars->{NETBIOSNAME},ou=domain controllers,$base_dn";
}
my $max_wait = 60;
- my $cmd = "$ldbsearch $testenv_vars->{CONFIGURATION} -H ldap://$testenv_vars->{SERVER} -U$testenv_vars->{USERNAME}%$testenv_vars->{PASSWORD} -s base -b \"$search_dn\"";
+
+ # Add hosts file for name lookups
+ my $cmd = "NSS_WRAPPER_HOSTS='$testenv_vars->{NSS_WRAPPER_HOSTS}' ";
+ if (defined($testenv_vars->{RESOLV_WRAPPER_CONF})) {
+ $cmd .= "RESOLV_WRAPPER_CONF='$testenv_vars->{RESOLV_WRAPPER_CONF}' ";
+ } else {
+ $cmd .= "RESOLV_WRAPPER_HOSTS='$testenv_vars->{RESOLV_WRAPPER_HOSTS}' ";
+ }
+
+ $cmd .= "$ldbsearch ";
+ $cmd .= "$testenv_vars->{CONFIGURATION} ";
+ $cmd .= "-H ldap://$testenv_vars->{SERVER} ";
+ $cmd .= "-U$testenv_vars->{USERNAME}%$testenv_vars->{PASSWORD} ";
+ $cmd .= "-s base ";
+ $cmd .= "-b '$search_dn' ";
while (system("$cmd >/dev/null") != 0) {
$count++;
if ($count > $max_wait) {
--
2.11.0
From 6b624432444bba434e17fb8a12d6671589fa54b5 Mon Sep 17 00:00:00 2001
From: Jamie McClymont <jamiemcclymont at catalyst.net.nz>
Date: Thu, 25 Jan 2018 17:23:06 +1300
Subject: [PATCH 24/38] selftest: consistently produce high-res UTC time
Currently some subunit reporters throughout the codebase provide low-res time,
meaning timestamps jump back and forth in the subunit file. Also, some subunit
reporters produce UTC timestamps while others produce local time. UTC was chosen
as the standard for this commit since all of the timestamps end with a Z (= Zulu
= UTC).
Signed-off-by: Jamie McClymont <jamiemcclymont at catalyst.net.nz>
---
lib/torture/subunit.c | 4 ++--
selftest/Subunit.pm | 6 ++++--
selftest/selftest.pl | 7 ++++---
source4/torture/rpc/witness.c | 4 ++--
4 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/lib/torture/subunit.c b/lib/torture/subunit.c
index 46f1b65053d..deb96ffa3a9 100644
--- a/lib/torture/subunit.c
+++ b/lib/torture/subunit.c
@@ -62,9 +62,9 @@ static void torture_subunit_report_time(struct torture_context *tctx)
return;
}
- tmp = localtime(&tp.tv_sec);
+ tmp = gmtime(&tp.tv_sec);
if (!tmp) {
- perror("localtime");
+ perror("gmtime");
return;
}
diff --git a/selftest/Subunit.pm b/selftest/Subunit.pm
index 1cc0e721696..46b0fda503c 100644
--- a/selftest/Subunit.pm
+++ b/selftest/Subunit.pm
@@ -16,6 +16,7 @@
package Subunit;
use POSIX;
+use Time::HiRes;
require Exporter;
@ISA = qw(Exporter);
@@ -43,10 +44,11 @@ sub end_test($$;$)
}
}
-sub report_time($)
+sub report_time()
{
my ($time) = @_;
- my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst) = localtime($time);
+ $time = Time::HiRes::time() unless (defined($time));
+ my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst) = gmtime($time);
$sec = ($time - int($time) + $sec);
my $msg = sprintf("%f", $sec);
if (substr($msg, 1, 1) eq ".") {
diff --git a/selftest/selftest.pl b/selftest/selftest.pl
index a772613b8e1..995abd71a04 100755
--- a/selftest/selftest.pl
+++ b/selftest/selftest.pl
@@ -28,6 +28,7 @@ use lib "$RealBin";
use Subunit;
use SocketWrapper;
use target::Samba;
+use Time::HiRes qw(time);
eval {
require Time::HiRes;
@@ -150,9 +151,9 @@ sub run_testsuite($$$$$)
Subunit::start_testsuite($name);
Subunit::progress_push();
- Subunit::report_time(time());
+ Subunit::report_time();
system($cmd);
- Subunit::report_time(time());
+ Subunit::report_time();
Subunit::progress_pop();
if ($? == -1) {
@@ -781,7 +782,7 @@ my $suitestotal = $#todo + 1;
unless ($opt_list) {
Subunit::progress($suitestotal);
- Subunit::report_time(time());
+ Subunit::report_time();
}
my $i = 0;
diff --git a/source4/torture/rpc/witness.c b/source4/torture/rpc/witness.c
index 4b44e91dbcf..4e7c682a7ee 100644
--- a/source4/torture/rpc/witness.c
+++ b/source4/torture/rpc/witness.c
@@ -790,9 +790,9 @@ static void torture_subunit_report_time(struct torture_context *tctx)
return;
}
- tmp = localtime(&tp.tv_sec);
+ tmp = gmtime(&tp.tv_sec);
if (!tmp) {
- torture_comment(tctx, "failed to call localtime");
+ torture_comment(tctx, "failed to call gmtime");
return;
}
--
2.11.0
From 45c16d8c4aea34c3d5bba4dd802cdef3627b8907 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Wed, 21 Mar 2018 07:02:42 +1300
Subject: [PATCH 25/38] autobuild: Split up the build further with
samba-ad-dc-2
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 9 +++++++++
.travis.yml | 1 +
script/autobuild.py | 19 +++++++++++++++++++
3 files changed, 29 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index a4354dad63c..653ebbd8d6b 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -50,6 +50,15 @@ build_samba_ad_dc:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-ad-dc --verbose --tail --testbase /tmp/samba-testbase
+build_samba_ad_dc_2:
+ stage: build
+ tags:
+ - docker
+ - private
+ script:
+ # this one takes about 1 hours to finish
+ - python script/autobuild.py samba-ad-dc-2 --verbose --tail --testbase /tmp/samba-testbase
+
build_samba_none_env:
stage: build
tags:
diff --git a/.travis.yml b/.travis.yml
index deca01d6f5a..4c56f598c85 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -18,6 +18,7 @@ env:
- TASK=samba-nt4
- TASK=samba-fileserver
- TASK=samba-ad-dc
+ - TASK=samba-ad-dc-2
- TASK=ldb
- TASK=tdb
- TASK=talloc
diff --git a/script/autobuild.py b/script/autobuild.py
index 77c01177bb9..3257027fae1 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -35,6 +35,7 @@ builddirs = {
"samba-test-only" : ".",
"samba-none-env" : ".",
"samba-ad-dc" : ".",
+ "samba-ad-dc-2" : ".",
"samba-systemkrb5" : ".",
"samba-nopython" : ".",
"ldb" : "lib/ldb",
@@ -59,6 +60,7 @@ defaulttasks = [ "ctdb",
"samba-static",
"samba-none-env",
"samba-ad-dc",
+ "samba-ad-dc-2",
"samba-systemkrb5",
"samba-nopython",
"ldb",
@@ -102,6 +104,9 @@ tasks = {
"TESTS='--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
"--exclude-env=ad_dc "
+ "--exclude-env=chgdcpass "
+ "--exclude-env=vampire_2000_dc "
+ "--exclude-env=fl2000dc "
"--exclude-env=none "
"--exclude-env=fileserver'",
"text/plain"),
@@ -131,6 +136,20 @@ tasks = {
("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_dc'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+ # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
+ "samba-ad-dc-2" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=chgdcpass --include-env=vampire_2000_dc --include-env=fl2000dc'", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+
+ # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
+ "samba-ad-dc-2" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=chgdcpass --include-env=vampire_2000_dc --include-env=fl2000dc'", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+
"samba-test-only" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", 'make test FAIL_IMMEDIATELY=1 TESTS="${TESTS}"',"text/plain") ],
--
2.11.0
From ea93742b8a79d4de73dc2f908106317c853e3824 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Wed, 21 Mar 2018 07:24:23 +1300
Subject: [PATCH 26/38] gitlab: Run fileserver tests on "private" not "shared"
This might make the delaywrite tests pass
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 653ebbd8d6b..faf04e3e2f7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -36,7 +36,7 @@ build_samba_fileserver:
stage: build
tags:
- docker
- - shared
+ - private
script:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-fileserver --verbose --tail --testbase /tmp/samba-testbase
--
2.11.0
From 0c20b4f2e10737446fdda3c56764bfaa07780c05 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Wed, 21 Mar 2018 20:43:10 +1300
Subject: [PATCH 27/38] winbindd: Add a cache of the samr and lsa handles for
the passdb domain
This domain is very close, in AD DC configurations over a internal ncacn_np pipe
and otherwise in the same process via C linking. It is however very expensive
to re-create the binding handle per SID->name lookup, so keep a cache.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/winbindd/winbindd_samr.c | 272 +++++++++++++++++++++++----------------
1 file changed, 162 insertions(+), 110 deletions(-)
diff --git a/source3/winbindd/winbindd_samr.c b/source3/winbindd/winbindd_samr.c
index aedb77bdee9..68064d725a2 100644
--- a/source3/winbindd/winbindd_samr.c
+++ b/source3/winbindd/winbindd_samr.c
@@ -40,6 +40,21 @@
#undef DBGC_CLASS
#define DBGC_CLASS DBGC_WINBIND
+/*
+ * The other end of this won't go away easily, so we can trust it
+ *
+ * It is either a long-lived process with the same lifetime as
+ * winbindd or a part of this process
+ */
+struct winbind_internal_pipe_private
+{
+ struct rpc_pipe_client *samr_pipe;
+ struct policy_handle samr_domain_hnd;
+ struct rpc_pipe_client *lsa_pipe;
+ struct policy_handle lsa_hnd;
+};
+
+
NTSTATUS open_internal_samr_conn(TALLOC_CTX *mem_ctx,
struct winbindd_domain *domain,
struct rpc_pipe_client **samr_pipe,
@@ -101,6 +116,70 @@ NTSTATUS open_internal_lsa_conn(TALLOC_CTX *mem_ctx,
return status;
}
+
+static NTSTATUS open_cached_internal_pipe_conn(struct winbindd_domain *domain,
+ struct rpc_pipe_client **samr_pipe,
+ struct policy_handle *samr_domain_hnd,
+ struct rpc_pipe_client **lsa_pipe,
+ struct policy_handle *lsa_hnd)
+{
+ struct winbind_internal_pipe_private *private_data;
+
+ if (domain->private_data == NULL) {
+ TALLOC_CTX *frame = talloc_stackframe();
+ NTSTATUS status;
+
+ private_data = talloc(frame,
+ struct winbind_internal_pipe_private);
+
+ status = open_internal_samr_conn(private_data,
+ domain,
+ &private_data->samr_pipe,
+ &private_data->samr_domain_hnd);
+
+ if (!NT_STATUS_IS_OK(status)) {
+ TALLOC_FREE(frame);
+ return status;
+ }
+
+ status = open_internal_lsa_conn(private_data,
+ &private_data->lsa_pipe,
+ &private_data->lsa_hnd);
+
+ if (!NT_STATUS_IS_OK(status)) {
+ TALLOC_FREE(frame);
+ return status;
+ }
+
+ domain->private_data = talloc_steal(domain, private_data);
+
+ TALLOC_FREE(frame);
+
+ } else {
+ private_data =
+ talloc_get_type_abort(domain->private_data,
+ struct winbind_internal_pipe_private);
+ }
+
+ if (samr_domain_hnd) {
+ *samr_domain_hnd = private_data->samr_domain_hnd;
+ }
+
+ if (samr_pipe) {
+ *samr_pipe = private_data->samr_pipe;
+ }
+
+ if (lsa_hnd) {
+ *lsa_hnd = private_data->lsa_hnd;
+ }
+
+ if (lsa_pipe) {
+ *lsa_pipe = private_data->lsa_pipe;
+ }
+
+ return NT_STATUS_OK;
+}
+
/*********************************************************************
SAM specific functions.
*********************************************************************/
@@ -116,8 +195,7 @@ static NTSTATUS sam_enum_dom_groups(struct winbindd_domain *domain,
struct wb_acct_info *info = NULL;
uint32_t num_info = 0;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("sam_enum_dom_groups\n"));
@@ -130,20 +208,24 @@ static NTSTATUS sam_enum_dom_groups(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
- goto error;
+ TALLOC_FREE(tmp_ctx);
+ return status;
}
- b = samr_pipe->binding_handle;
-
status = rpc_enum_dom_groups(tmp_ctx,
samr_pipe,
&dom_pol,
&num_info,
&info);
if (!NT_STATUS_IS_OK(status)) {
- goto error;
+ TALLOC_FREE(tmp_ctx);
+ return status;
}
if (pnum_info) {
@@ -154,10 +236,6 @@ static NTSTATUS sam_enum_dom_groups(struct winbindd_domain *domain,
*pinfo = talloc_move(mem_ctx, &info);
}
-error:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -171,8 +249,7 @@ static NTSTATUS sam_query_user_list(struct winbindd_domain *domain,
struct policy_handle dom_pol = { 0 };
uint32_t *rids = NULL;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("samr_query_user_list\n"));
@@ -181,13 +258,15 @@ static NTSTATUS sam_query_user_list(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = samr_pipe->binding_handle;
-
status = rpc_query_user_list(tmp_ctx,
samr_pipe,
&dom_pol,
@@ -202,10 +281,6 @@ static NTSTATUS sam_query_user_list(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
-
TALLOC_FREE(rids);
TALLOC_FREE(tmp_ctx);
return status;
@@ -221,8 +296,7 @@ static NTSTATUS sam_trusted_domains(struct winbindd_domain *domain,
struct netr_DomainTrust *trusts = NULL;
uint32_t num_trusts = 0;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("samr: trusted domains\n"));
@@ -235,13 +309,15 @@ static NTSTATUS sam_trusted_domains(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_lsa_conn(tmp_ctx, &lsa_pipe, &lsa_policy);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = lsa_pipe->binding_handle;
-
status = rpc_trusted_domains(tmp_ctx,
lsa_pipe,
&lsa_policy,
@@ -257,10 +333,6 @@ static NTSTATUS sam_trusted_domains(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&lsa_policy)) {
- dcerpc_lsa_Close(b, mem_ctx, &lsa_policy, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -284,9 +356,8 @@ static NTSTATUS sam_lookup_groupmem(struct winbindd_domain *domain,
uint32_t *name_types = NULL;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
-
+ NTSTATUS status;
+
DEBUG(3,("sam_lookup_groupmem\n"));
/* Paranoia check */
@@ -304,13 +375,15 @@ static NTSTATUS sam_lookup_groupmem(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = samr_pipe->binding_handle;
-
status = rpc_lookup_groupmem(tmp_ctx,
samr_pipe,
&dom_pol,
@@ -340,10 +413,6 @@ static NTSTATUS sam_lookup_groupmem(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -398,8 +467,7 @@ static NTSTATUS sam_enum_local_groups(struct winbindd_domain *domain,
struct wb_acct_info *info = NULL;
uint32_t num_info = 0;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("samr: enum local groups\n"));
@@ -412,13 +480,15 @@ static NTSTATUS sam_enum_local_groups(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = samr_pipe->binding_handle;
-
status = rpc_enum_local_groups(mem_ctx,
samr_pipe,
&dom_pol,
@@ -437,10 +507,6 @@ static NTSTATUS sam_enum_local_groups(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -459,8 +525,7 @@ static NTSTATUS sam_name_to_sid(struct winbindd_domain *domain,
struct dom_sid sid;
enum lsa_SidType type;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("sam_name_to_sid\n"));
@@ -469,13 +534,15 @@ static NTSTATUS sam_name_to_sid(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_lsa_conn(tmp_ctx, &lsa_pipe, &lsa_policy);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = lsa_pipe->binding_handle;
-
status = rpc_name_to_sid(tmp_ctx,
lsa_pipe,
&lsa_policy,
@@ -496,10 +563,6 @@ static NTSTATUS sam_name_to_sid(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&lsa_policy)) {
- dcerpc_lsa_Close(b, mem_ctx, &lsa_policy, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -518,8 +581,7 @@ static NTSTATUS sam_sid_to_name(struct winbindd_domain *domain,
char *name = NULL;
enum lsa_SidType type;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("sam_sid_to_name\n"));
@@ -543,13 +605,15 @@ static NTSTATUS sam_sid_to_name(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_lsa_conn(tmp_ctx, &lsa_pipe, &lsa_policy);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = lsa_pipe->binding_handle;
-
status = rpc_sid_to_name(tmp_ctx,
lsa_pipe,
&lsa_policy,
@@ -572,9 +636,6 @@ static NTSTATUS sam_sid_to_name(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&lsa_policy)) {
- dcerpc_lsa_Close(b, mem_ctx, &lsa_policy, &result);
- }
TALLOC_FREE(tmp_ctx);
return status;
@@ -595,8 +656,7 @@ static NTSTATUS sam_rids_to_names(struct winbindd_domain *domain,
char *domain_name = NULL;
char **names = NULL;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("sam_rids_to_names for %s\n", domain->name));
@@ -616,13 +676,15 @@ static NTSTATUS sam_rids_to_names(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_lsa_conn(tmp_ctx, &lsa_pipe, &lsa_policy);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = lsa_pipe->binding_handle;
-
status = rpc_rids_to_names(tmp_ctx,
lsa_pipe,
&lsa_policy,
@@ -650,10 +712,6 @@ static NTSTATUS sam_rids_to_names(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&lsa_policy)) {
- dcerpc_lsa_Close(b, mem_ctx, &lsa_policy, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -676,13 +734,17 @@ static NTSTATUS sam_lockout_policy(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto error;
}
b = samr_pipe->binding_handle;
-
+
status = dcerpc_samr_QueryDomainInfo(b,
mem_ctx,
&dom_pol,
@@ -700,10 +762,6 @@ static NTSTATUS sam_lockout_policy(struct winbindd_domain *domain,
*lockout_policy = info->info12;
error:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -726,7 +784,11 @@ static NTSTATUS sam_password_policy(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto error;
}
@@ -750,10 +812,6 @@ static NTSTATUS sam_password_policy(struct winbindd_domain *domain,
*passwd_policy = info->info1;
error:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
-
TALLOC_FREE(tmp_ctx);
return status;
}
@@ -770,8 +828,7 @@ static NTSTATUS sam_lookup_usergroups(struct winbindd_domain *domain,
struct dom_sid *user_grpsids = NULL;
uint32_t num_groups = 0;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("sam_lookup_usergroups\n"));
@@ -786,13 +843,15 @@ static NTSTATUS sam_lookup_usergroups(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = samr_pipe->binding_handle;
-
status = rpc_lookup_usergroups(tmp_ctx,
samr_pipe,
&dom_pol,
@@ -813,9 +872,6 @@ static NTSTATUS sam_lookup_usergroups(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
TALLOC_FREE(tmp_ctx);
return status;
@@ -833,8 +889,7 @@ static NTSTATUS sam_lookup_useraliases(struct winbindd_domain *domain,
uint32_t num_aliases = 0;
uint32_t *alias_rids = NULL;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("sam_lookup_useraliases\n"));
@@ -847,13 +902,15 @@ static NTSTATUS sam_lookup_useraliases(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = samr_pipe->binding_handle;
-
status = rpc_lookup_useraliases(tmp_ctx,
samr_pipe,
&dom_pol,
@@ -874,9 +931,6 @@ static NTSTATUS sam_lookup_useraliases(struct winbindd_domain *domain,
}
done:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, mem_ctx, &dom_pol, &result);
- }
TALLOC_FREE(tmp_ctx);
return status;
@@ -890,8 +944,7 @@ static NTSTATUS sam_sequence_number(struct winbindd_domain *domain,
struct policy_handle dom_pol = { 0 };
uint32_t seq = DOM_SEQUENCE_NONE;
TALLOC_CTX *tmp_ctx;
- NTSTATUS status, result;
- struct dcerpc_binding_handle *b = NULL;
+ NTSTATUS status;
DEBUG(3,("samr: sequence number\n"));
@@ -904,13 +957,15 @@ static NTSTATUS sam_sequence_number(struct winbindd_domain *domain,
return NT_STATUS_NO_MEMORY;
}
- status = open_internal_samr_conn(tmp_ctx, domain, &samr_pipe, &dom_pol);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
- b = samr_pipe->binding_handle;
-
status = rpc_sequence_number(tmp_ctx,
samr_pipe,
&dom_pol,
@@ -923,11 +978,8 @@ static NTSTATUS sam_sequence_number(struct winbindd_domain *domain,
if (pseq) {
*pseq = seq;
}
-done:
- if (b && is_valid_policy_hnd(&dom_pol)) {
- dcerpc_samr_Close(b, tmp_ctx, &dom_pol, &result);
- }
+done:
TALLOC_FREE(tmp_ctx);
return status;
}
--
2.11.0
From 673c270cbd146243c350203df740cab3b1cb75ab Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Wed, 21 Mar 2018 20:44:31 +1300
Subject: [PATCH 28/38] winbindd: Do re-connect if the RPC call fails in the
passdb case
This is very, very unlikely but possible as in the AD case the RPC server is in
another process that may eventually be able to restart.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/winbindd/winbindd_samr.c | 309 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 308 insertions(+), 1 deletion(-)
diff --git a/source3/winbindd/winbindd_samr.c b/source3/winbindd/winbindd_samr.c
index 68064d725a2..70e07fa30a6 100644
--- a/source3/winbindd/winbindd_samr.c
+++ b/source3/winbindd/winbindd_samr.c
@@ -28,6 +28,7 @@
#include "winbindd_rpc.h"
#include "lib/util_unixsids.h"
#include "rpc_client/rpc_client.h"
+#include "rpc_client/cli_pipe.h"
#include "../librpc/gen_ndr/ndr_samr_c.h"
#include "rpc_client/cli_samr.h"
#include "../librpc/gen_ndr/ndr_lsa_c.h"
@@ -223,6 +224,29 @@ static NTSTATUS sam_enum_dom_groups(struct winbindd_domain *domain,
&dom_pol,
&num_info,
&info);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ TALLOC_FREE(tmp_ctx);
+ return status;
+ }
+
+ status = rpc_enum_dom_groups(tmp_ctx,
+ samr_pipe,
+ &dom_pol,
+ &num_info,
+ &info);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
TALLOC_FREE(tmp_ctx);
return status;
@@ -272,6 +296,28 @@ static NTSTATUS sam_query_user_list(struct winbindd_domain *domain,
&dom_pol,
&domain->sid,
&rids);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_query_user_list(tmp_ctx,
+ samr_pipe,
+ &dom_pol,
+ &domain->sid,
+ &rids);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -323,6 +369,27 @@ static NTSTATUS sam_trusted_domains(struct winbindd_domain *domain,
&lsa_policy,
&num_trusts,
&trusts);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(lsa_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_trusted_domains(tmp_ctx,
+ lsa_pipe,
+ &lsa_policy,
+ &num_trusts,
+ &trusts);
+ }
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -396,6 +463,34 @@ static NTSTATUS sam_lookup_groupmem(struct winbindd_domain *domain,
&names,
&name_types);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_lookup_groupmem(tmp_ctx,
+ samr_pipe,
+ &dom_pol,
+ domain->name,
+ &domain->sid,
+ group_sid,
+ type,
+ &num_names,
+ &sid_mem,
+ &names,
+ &name_types);
+ }
+
if (pnum_names) {
*pnum_names = num_names;
}
@@ -494,6 +589,28 @@ static NTSTATUS sam_enum_local_groups(struct winbindd_domain *domain,
&dom_pol,
&num_info,
&info);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_enum_local_groups(mem_ctx,
+ samr_pipe,
+ &dom_pol,
+ &num_info,
+ &info);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -551,6 +668,30 @@ static NTSTATUS sam_name_to_sid(struct winbindd_domain *domain,
flags,
&sid,
&type);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(lsa_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_name_to_sid(tmp_ctx,
+ lsa_pipe,
+ &lsa_policy,
+ domain_name,
+ name,
+ flags,
+ &sid,
+ &type);
+ }
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -622,7 +763,32 @@ static NTSTATUS sam_sid_to_name(struct winbindd_domain *domain,
&domain_name,
&name,
&type);
-
+
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(lsa_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_sid_to_name(tmp_ctx,
+ lsa_pipe,
+ &lsa_policy,
+ domain,
+ sid,
+ &domain_name,
+ &name,
+ &type);
+ }
+
if (ptype) {
*ptype = type;
}
@@ -695,6 +861,32 @@ static NTSTATUS sam_rids_to_names(struct winbindd_domain *domain,
&domain_name,
&names,
&types);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(lsa_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ NULL,
+ NULL,
+ &lsa_pipe,
+ &lsa_policy);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_rids_to_names(tmp_ctx,
+ lsa_pipe,
+ &lsa_policy,
+ domain,
+ domain_sid,
+ rids,
+ num_rids,
+ &domain_name,
+ &names,
+ &types);
+ }
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -751,6 +943,28 @@ static NTSTATUS sam_lockout_policy(struct winbindd_domain *domain,
DomainLockoutInformation,
&info,
&result);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto error;
+ }
+
+ status = dcerpc_samr_QueryDomainInfo(b,
+ mem_ctx,
+ &dom_pol,
+ DomainLockoutInformation,
+ &info,
+ &result);
+ }
if (!NT_STATUS_IS_OK(status)) {
goto error;
}
@@ -801,6 +1015,29 @@ static NTSTATUS sam_password_policy(struct winbindd_domain *domain,
DomainPasswordInformation,
&info,
&result);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto error;
+ }
+
+ status = dcerpc_samr_QueryDomainInfo(b,
+ mem_ctx,
+ &dom_pol,
+ DomainPasswordInformation,
+ &info,
+ &result);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
goto error;
}
@@ -859,6 +1096,30 @@ static NTSTATUS sam_lookup_usergroups(struct winbindd_domain *domain,
user_sid,
&num_groups,
&user_grpsids);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_lookup_usergroups(tmp_ctx,
+ samr_pipe,
+ &dom_pol,
+ &domain->sid,
+ user_sid,
+ &num_groups,
+ &user_grpsids);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -918,6 +1179,30 @@ static NTSTATUS sam_lookup_useraliases(struct winbindd_domain *domain,
sids,
&num_aliases,
&alias_rids);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_lookup_useraliases(tmp_ctx,
+ samr_pipe,
+ &dom_pol,
+ num_sids,
+ sids,
+ &num_aliases,
+ &alias_rids);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
@@ -971,6 +1256,28 @@ static NTSTATUS sam_sequence_number(struct winbindd_domain *domain,
&dom_pol,
domain->name,
&seq);
+ /*
+ * Very unlikely, but before checking the result,
+ * if we disconnected try again
+ */
+ if (!rpccli_is_connected(samr_pipe)) {
+ TALLOC_FREE(domain->private_data);
+ status = open_cached_internal_pipe_conn(domain,
+ &samr_pipe,
+ &dom_pol,
+ NULL,
+ NULL);
+ if (!NT_STATUS_IS_OK(status)) {
+ goto done;
+ }
+
+ status = rpc_sequence_number(tmp_ctx,
+ samr_pipe,
+ &dom_pol,
+ domain->name,
+ &seq);
+ }
+
if (!NT_STATUS_IS_OK(status)) {
goto done;
}
--
2.11.0
From 832edc5aea7438820684a4011083208fea73a28d Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Wed, 21 Mar 2018 21:23:13 +1300
Subject: [PATCH 29/38] winbindd: Use talloc_zero_array for consistency with
other winbindd_domain allocators
The other allocator for this structure uses talloc_zero()
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/winbindd/wb_seqnums.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/source3/winbindd/wb_seqnums.c b/source3/winbindd/wb_seqnums.c
index 2a4cdc930e8..3c9af01e40f 100644
--- a/source3/winbindd/wb_seqnums.c
+++ b/source3/winbindd/wb_seqnums.c
@@ -56,8 +56,8 @@ struct tevent_req *wb_seqnums_send(TALLOC_CTX *mem_ctx,
state->subreqs = talloc_array(state, struct tevent_req *,
state->num_domains);
- state->domains = talloc_array(state, struct winbindd_domain *,
- state->num_domains);
+ state->domains = talloc_zero_array(state, struct winbindd_domain *,
+ state->num_domains);
state->stati = talloc_array(state, NTSTATUS, state->num_domains);
state->seqnums = talloc_array(state, uint32_t, state->num_domains);
--
2.11.0
From 4baf9c80920c1a97981d703ab69ebc4a01a7a64a Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 08:25:41 +1300
Subject: [PATCH 30/38] autobuild: Run all envs that depend on ad_dc in the
ad_dc job
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 3257027fae1..db908fc718c 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -104,6 +104,11 @@ tasks = {
"TESTS='--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
"--exclude-env=ad_dc "
+ "--exclude-env=fl2003dc "
+ "--exclude-env=fl2008r2dc "
+ "--exclude-env=ad_member "
+ "--exclude-env=ad_member_idmap_rid "
+ "--exclude-env=ad_member_idmap_ad "
"--exclude-env=chgdcpass "
"--exclude-env=vampire_2000_dc "
"--exclude-env=fl2000dc "
@@ -133,7 +138,13 @@ tasks = {
"samba-ad-dc" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_dc'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='"
+ "--include-env=ad_dc "
+ "--include-env=fl2003dc "
+ "--include-env=fl2008r2dc "
+ "--include-env=ad_member "
+ "--include-env=ad_member_idmap_rid "
+ "--include-env=ad_member_idmap_ad'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
# We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
--
2.11.0
From 7b8c8cf332606446318c48a588c874f5960f23da Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 12:02:16 +1300
Subject: [PATCH 31/38] selftest: Do not run raw.notify, smb2.oplock and
raw.oplock twice
These are slower tests that do not need to be run against the ad_dc configuration
in particular.
This saves time in the ad_dc job which needs to stay under 50mins to
pass on travis-ci (and faster tests are better for everyone anyway).
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/selftest/tests.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/source3/selftest/tests.py b/source3/selftest/tests.py
index 00e04318c7c..a4ded32195b 100755
--- a/source3/selftest/tests.py
+++ b/source3/selftest/tests.py
@@ -517,7 +517,8 @@ for t in tests:
plansmbtorture4testsuite(t, env, '//$SERVER/tmp -U$USERNAME%$PASSWORD')
elif t == "smb2.change_notify_disabled":
plansmbtorture4testsuite(t, "simpleserver", '//$SERVER/tmp -U$USERNAME%$PASSWORD')
- elif t == "smb2.notify":
+ elif t == "smb2.notify" or t == "raw.notify" or t == "smb2.oplock" or t == "raw.oplock":
+ # These tests are a little slower so don't duplicate them with ad_dc
plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/tmp -U$USERNAME%$PASSWORD --signing=required')
elif t == "smb2.dosmode":
plansmbtorture4testsuite(t, "simpleserver", '//$SERVER/dosmode -U$USERNAME%$PASSWORD')
--
2.11.0
From 75a314ea850480eff3fe381ef2d9b09fe69d701c Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 12:21:52 +1300
Subject: [PATCH 32/38] selftest: Do not run krb5.kdc machine account test
against ad_dc
This code is already well tested against fl2008r2dc and just as per
8f1557a2c43e287c07723c16be78e1d858f4111d this test is slow and we can afford to be
more selective here.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source4/selftest/tests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source4/selftest/tests.py b/source4/selftest/tests.py
index 5c5c73caaf7..2a7cf163a3b 100755
--- a/source4/selftest/tests.py
+++ b/source4/selftest/tests.py
@@ -972,7 +972,7 @@ plansmbtorture4testsuite('krb5.kdc', env, ['ncacn_np:$SERVER_IP', "-k", "yes", '
"samba4.krb5.kdc with specified account")
-for env in ["rodc", "promoted_dc", "ad_dc", "fl2000dc", "fl2008r2dc"]:
+for env in ["rodc", "promoted_dc", "fl2000dc", "fl2008r2dc"]:
if env == "rodc":
# The machine account is cached at the RODC, as it is the local account
extra_options = ['--option=torture:expect_rodc=true', '--option=torture:expect_cached_at_rodc=true']
--
2.11.0
From b2a6bfeb2ab55d1a5c20ce2f7db0d0af561cb1f2 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 13:15:34 +1300
Subject: [PATCH 33/38] selftest: Align cleanup of tmpkpasswdscript with
scripts that use it
Some scripst removed it without creating it and the others created it
and did not remove it.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
testprogs/blackbox/test_chgdcpass.sh | 2 +-
testprogs/blackbox/test_kinit_mit.sh | 2 +-
testprogs/blackbox/test_kinit_trusts_heimdal.sh | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/testprogs/blackbox/test_chgdcpass.sh b/testprogs/blackbox/test_chgdcpass.sh
index 3830cb4e31d..6ac4c882e32 100755
--- a/testprogs/blackbox/test_chgdcpass.sh
+++ b/testprogs/blackbox/test_chgdcpass.sh
@@ -110,5 +110,5 @@ test_smbclient "Test login with kerberos ccache with fresh kinit" 'ls' "$unc" -k
rm -f $KRB5CCNAME
-rm -f $PREFIX/tmpccache tmpccfile tmppassfile tmpuserpassfile tmpuserccache tmpkpasswdscript
+rm -f $PREFIX/tmpccache tmpccfile tmppassfile tmpuserpassfile tmpuserccache
exit $failed
diff --git a/testprogs/blackbox/test_kinit_mit.sh b/testprogs/blackbox/test_kinit_mit.sh
index 3e07281b8c7..d178ba4c710 100755
--- a/testprogs/blackbox/test_kinit_mit.sh
+++ b/testprogs/blackbox/test_kinit_mit.sh
@@ -306,5 +306,5 @@ $samba_kdestroy
rm -f $KRB5CCNAME_PATH
rm -f $PREFIX/tmpkinituserpassscript
rm -f $PREFIX/tmpkinitscript
-
+rm -f $PREFIX/tmpkpasswdscript
exit $failed
diff --git a/testprogs/blackbox/test_kinit_trusts_heimdal.sh b/testprogs/blackbox/test_kinit_trusts_heimdal.sh
index e67f77361a4..4a1c0f7f213 100755
--- a/testprogs/blackbox/test_kinit_trusts_heimdal.sh
+++ b/testprogs/blackbox/test_kinit_trusts_heimdal.sh
@@ -94,5 +94,5 @@ testit "wbinfo check outgoing trust pw" $VALGRIND $wbinfo --check-secret --domai
test_smbclient "Test user login with the changed outgoing secret" 'ls' "$unc" -k yes -U$USERNAME@$REALM%$PASSWORD || failed=`expr $failed + 1`
-rm -f $PREFIX/tmpccache tmpccfile tmppassfile tmpuserpassfile tmpuserccache tmpkpasswdscript
+rm -f $PREFIX/tmpccache tmpccfile tmppassfile tmpuserpassfile tmpuserccache
exit $failed
--
2.11.0
From aa9e84ec2d10943734eab9bd10024ba1d3675239 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 13:18:39 +1300
Subject: [PATCH 34/38] selftest: Ensure tmpkpasswdscript is always under
$PREFIX
Otherwise it might not be removed at the end of the test
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
testprogs/blackbox/test_kpasswd_heimdal.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/testprogs/blackbox/test_kpasswd_heimdal.sh b/testprogs/blackbox/test_kpasswd_heimdal.sh
index 7e3daed8467..5cb6dab9fbe 100755
--- a/testprogs/blackbox/test_kpasswd_heimdal.sh
+++ b/testprogs/blackbox/test_kpasswd_heimdal.sh
@@ -109,7 +109,7 @@ testit "kpasswd check short user password" \
###########################################################
echo "check that a short password is rejected"
-cat > ./tmpkpasswdscript <<EOF
+cat > $PREFIX/tmpkpasswdscript <<EOF
expect Password
password ${TEST_PASSWORD}\n
expect New password
@@ -126,7 +126,7 @@ testit "kpasswd check weak user password" \
### check that a strong password is accepted
###########################################################
-cat > ./tmpkpasswdscript <<EOF
+cat > $PREFIX/tmpkpasswdscript <<EOF
expect Password
password ${TEST_PASSWORD}\n
expect New password
--
2.11.0
From c7d1ec21031529cd0d0eddd9faf4a741bd71e73d Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 15:13:18 +1300
Subject: [PATCH 35/38] gitlab-ci: Split up build_samba_others and build_ctdb
tasks
These make too much output and the shared runners on GitLab CI object to
sending more than 4MB of output.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.gitlab-ci.yml | 44 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 43 insertions(+), 1 deletion(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index faf04e3e2f7..8220fb236d9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -68,17 +68,52 @@ build_samba_none_env:
# this one takes about 1 hours to finish
- python script/autobuild.py samba-none-env --verbose --tail --testbase /tmp/samba-testbase
-build_samba_others:
+build_samba_nopython:
stage: build
tags:
- docker
- shared
script:
- python script/autobuild.py samba-nopython --verbose --tail --testbase /tmp/samba-testbase
+
+build_samba_systemkrb5:
+ stage: build
+ tags:
+ - docker
+ - shared
+ script:
- python script/autobuild.py samba-systemkrb5 --verbose --tail --testbase /tmp/samba-testbase
+
+build_samba_xc:
+ stage: build
+ tags:
+ - docker
+ - shared
+ script:
- python script/autobuild.py samba-xc --verbose --tail --testbase /tmp/samba-testbase
+
+build_samba_o3:
+ stage: build
+ tags:
+ - docker
+ - shared
+ script:
- python script/autobuild.py samba-o3 --verbose --tail --testbase /tmp/samba-testbase
+
+build_samba_libs:
+ stage: build
+ tags:
+ - docker
+ - shared
+ script:
- python script/autobuild.py samba-libs --verbose --tail --testbase /tmp/samba-testbase
+
+build_samba_static:
+ stage: build
+ tags:
+ - docker
+ - shared
+ script:
- python script/autobuild.py samba-static --verbose --tail --testbase /tmp/samba-testbase
build_ctdb:
@@ -88,6 +123,13 @@ build_ctdb:
- shared
script:
- python script/autobuild.py samba-ctdb --verbose --tail --testbase /tmp/samba-testbase
+
+build_samba_ctdb:
+ stage: build
+ tags:
+ - docker
+ - shared
+ script:
- python script/autobuild.py ctdb --verbose --tail --testbase /tmp/samba-testbase
build_others:
--
2.11.0
From 1996c0c0a3d1dfa4f03be3e340ed615a626c3868 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 16:25:59 +1300
Subject: [PATCH 36/38] selftest: Move slow raw_protocol test to chgdcpass
environment
The ad_dc environment is busy and we need to keep it under 50mins for travis CI, so run
this on a different environment with a shorter runtime.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source4/selftest/tests.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/source4/selftest/tests.py b/source4/selftest/tests.py
index 2a7cf163a3b..475355b3db7 100755
--- a/source4/selftest/tests.py
+++ b/source4/selftest/tests.py
@@ -619,7 +619,7 @@ planpythontestsuite("ad_dc_ntvfs:local", "samba.tests.dcerpc.rpcecho")
planoldpythontestsuite("ad_dc_ntvfs:local", "samba.tests.dcerpc.registry", extra_args=['-U"$USERNAME%$PASSWORD"'])
planoldpythontestsuite("ad_dc_ntvfs", "samba.tests.dcerpc.dnsserver", extra_args=['-U"$USERNAME%$PASSWORD"'])
planoldpythontestsuite("ad_dc", "samba.tests.dcerpc.dnsserver", extra_args=['-U"$USERNAME%$PASSWORD"'])
-planoldpythontestsuite("ad_dc", "samba.tests.dcerpc.raw_protocol", extra_args=['-U"$USERNAME%$PASSWORD"'])
+planoldpythontestsuite("chgdcpass", "samba.tests.dcerpc.raw_protocol", extra_args=['-U"$USERNAME%$PASSWORD"'])
if have_heimdal_support:
planoldpythontestsuite("ad_dc:local", "samba.tests.auth_log", extra_args=['-U"$USERNAME%$PASSWORD"'],
environ={'CLIENT_IP': '127.0.0.11',
--
2.11.0
From e2089b481380cf96634f87f2333b3496a752c180 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 16:45:05 +1300
Subject: [PATCH 37/38] selftest: Do not run *.lock tests against both nt4_dc
and ad_dc
This part of the protocol is not changed by being an AD DC.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source3/selftest/tests.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/source3/selftest/tests.py b/source3/selftest/tests.py
index a4ded32195b..a9b7c209393 100755
--- a/source3/selftest/tests.py
+++ b/source3/selftest/tests.py
@@ -488,7 +488,8 @@ for t in tests:
elif t == "smb2.lock":
plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/aio -U$USERNAME%$PASSWORD', 'aio')
plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/tmp -U$USERNAME%$PASSWORD')
- plansmbtorture4testsuite(t, "ad_dc", '//$SERVER/tmp -U$USERNAME%$PASSWORD')
+ elif t == "raw.lock" or t == "base.lock":
+ plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/tmp -U$USERNAME%$PASSWORD')
elif t == "raw.read":
plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/tmp -U$USERNAME%$PASSWORD')
plansmbtorture4testsuite(t, "nt4_dc", '//$SERVER_IP/aio -U$USERNAME%$PASSWORD', 'aio')
--
2.11.0
From 8b9b0dd463448a807deb3ab336aeb5dd3a5b3b26 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Mar 2018 16:49:53 +1300
Subject: [PATCH 38/38] selftest: Run net.api.become.dc against less roles
This test is slower than many and need not be run five times.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
source4/selftest/tests.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/source4/selftest/tests.py b/source4/selftest/tests.py
index 475355b3db7..1415611ee6e 100755
--- a/source4/selftest/tests.py
+++ b/source4/selftest/tests.py
@@ -262,6 +262,8 @@ for env in ["ad_dc_ntvfs", "fl2000dc", "fl2003dc", "fl2008r2dc", "ad_dc"]:
for bindoptions in ["connect", "krb5", "krb5,sign", "krb5,seal", "spnego", "spnego,sign", "spnego,seal"] + validate_list + ["padcheck", "bigendian", "bigendian,seal"]:
echooptions = "--option=socket:testnonblock=True --option=torture:quick=yes -k yes"
plansmbtorture4testsuite('rpc.echo', env, ["%s:$SERVER[%s]" % (transport, bindoptions), echooptions, '-U$USERNAME%$PASSWORD', '--workgroup=$DOMAIN'], "samba4.rpc.echo on %s with %s and %s" % (transport, bindoptions, echooptions))
+
+for env in ["fl2000dc", "fl2008r2dc"]:
plansmbtorture4testsuite("net.api.become.dc", env, '$SERVER[%s] -U$USERNAME%%$PASSWORD -W$DOMAIN' % validate)
for bindoptions in ["sign", "seal"]:
--
2.11.0
More information about the samba-technical
mailing list