[WIP][PATCH] Run make test envs in parallel in autobuild
Andrew Bartlett
abartlet at samba.org
Fri Jul 21 09:17:17 UTC 2017
On Tue, 2017-05-09 at 22:15 +0200, Andreas Schneider via samba-
technical wrote:
> On Friday, 5 May 2017 22:41:45 CEST Andrew Bartlett via samba-technical wrote:
> > On Tue, 2017-02-28 at 21:58 +1300, Andrew Bartlett wrote:
> > > This patch was inspired by the fact that even our basic python tests do
> > > not run in the travis-ci triggered by github pull requests.
> > >
> > > I also wanted to see what time we could save by running some tests from
> > > the separated environments in a different autobuild task (which maps to
> > > distinct travis-ci tasks).
> > >
> > > On the Catalyst cloud, this decreases overall time on a 4 CPU server by
> > > 80mins, to 180min from 260min. It may be making the flapping tests
> > > flap more also however, and these need to be understood.
> > >
> > > Tasks that run in under 45mins can run in travis-ci, which is a good
> > > thing.
> > >
> > > Anyway, for now it is food for thought and a possible way to reduce the
> > > ever-expanding build and test times, and ensure more of our tests run
> > > for github users.
> >
> > I'm giving this another go, after comments at the team meeting.
> >
> > Hopefully with less flapping tests overall, this becomes practical.
>
> Thanks for looking into this Andrew!
Just a status update on this. It seems with my figures above I may
have made autobuild faster simply by removing all the nt4_dc tests,
rather than making things more parallel. Oops.
Before I push anything I'll get more stats and I'll be counting the
number of tests run to ensure we haven't lost any.
The attached patch series makes us parallel while, I hope, keeping the
nt4_dc and nt4_member tests.
Also attached is a patch I have under test to reverse a regression I
noticed where in a preparitory patch for this, started under-testing
the samba-systemkrb5 autobuild target, oops!
Thanks,
Andrew Bartlett
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
-------------- next part --------------
From d3525f0b60a5c8bd7ce2217024e38f5dc8e92c32 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 21 Jul 2017 20:10:43 +1200
Subject: [PATCH] selftest: Make --include-env and --exclude-env use the base
env name
The code as deployed would have required (eg) '--include-env=ktest
--include-env=ktest:local' which was not done in autobuild, causing
tests to be skipped. This patch restores the intended behaviour.
This causes 33 testsuites to run, one more test (the newly added
samba.tests.ntlmauth) than the old regex provided (before
602772159dfd1213385f42ecbf31136f57693b63).
(The regression dropped us down to matching only 7 tests).
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12922
---
selftest/selftest.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/selftest/selftest.pl b/selftest/selftest.pl
index 38de96d..b3ef658 100755
--- a/selftest/selftest.pl
+++ b/selftest/selftest.pl
@@ -1085,19 +1085,19 @@ $envvarstr
my $cmd = $$_[2];
my $name = $$_[0];
my $envname = $$_[1];
-
+ my ($env_basename, $env_localpart) = split(/:/, $envname);
my $envvars = "SKIP";
if (@opt_include_env) {
foreach my $env (@opt_include_env) {
- if ($envname eq $env) {
+ if ($env_basename eq $env) {
$envvars = setup_env($envname, $prefix);
}
}
} elsif (@opt_exclude_env) {
my $excluded = 0;
foreach my $env (@opt_exclude_env) {
- if ($envname eq $env) {
+ if ($env_basename eq $env) {
$excluded = 1;
}
}
--
2.9.4
-------------- next part --------------
From d3525f0b60a5c8bd7ce2217024e38f5dc8e92c32 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 21 Jul 2017 20:10:43 +1200
Subject: [PATCH 1/5] selftest: Make --include-env and --exclude-env use the
base env name
The code as deployed would have required (eg) '--include-env=ktest
--include-env=ktest:local' which was not done in autobuild, causing
tests to be skipped. This patch restores the intended behaviour.
This causes 33 testsuites to run, one more test (the newly added
samba.tests.ntlmauth) than the old regex provided (before
602772159dfd1213385f42ecbf31136f57693b63).
(The regression dropped us down to matching only 7 tests).
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12922
---
selftest/selftest.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/selftest/selftest.pl b/selftest/selftest.pl
index 38de96d..b3ef658 100755
--- a/selftest/selftest.pl
+++ b/selftest/selftest.pl
@@ -1085,19 +1085,19 @@ $envvarstr
my $cmd = $$_[2];
my $name = $$_[0];
my $envname = $$_[1];
-
+ my ($env_basename, $env_localpart) = split(/:/, $envname);
my $envvars = "SKIP";
if (@opt_include_env) {
foreach my $env (@opt_include_env) {
- if ($envname eq $env) {
+ if ($env_basename eq $env) {
$envvars = setup_env($envname, $prefix);
}
}
} elsif (@opt_exclude_env) {
my $excluded = 0;
foreach my $env (@opt_exclude_env) {
- if ($envname eq $env) {
+ if ($env_basename eq $env) {
$excluded = 1;
}
}
--
2.9.4
From 4427d7e57efaf51db4acdad67e4d0611b243f5d4 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 21 Jul 2017 19:54:36 +1200
Subject: [PATCH 2/5] autobuild: Move defaulttasks to one-per-line
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index a4ad544..2ae05a5 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -44,7 +44,21 @@ builddirs = {
"retry" : "."
}
-defaulttasks = [ "ctdb", "samba", "samba-xc", "samba-o3", "samba-ctdb", "samba-libs", "samba-static", "samba-systemkrb5", "samba-nopython", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ]
+defaulttasks = [ "ctdb",
+ "samba",
+ "samba-xc",
+ "samba-o3",
+ "samba-ctdb",
+ "samba-libs",
+ "samba-static",
+ "samba-systemkrb5",
+ "samba-nopython",
+ "ldb",
+ "tdb",
+ "talloc",
+ "replace",
+ "tevent",
+ "pidl" ]
if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
defaulttasks.remove("samba-o3")
--
2.9.4
From b841051af6cac4f508fd6e244555d17090284fb5 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 30 Jun 2017 11:13:55 +1200
Subject: [PATCH 3/5] autobuild: Run "none" environment tests in samba-o3
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 2ae05a5..6ebc818 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -90,7 +90,7 @@ tasks = {
# We have 'test' before 'install' because, 'test' should work without 'install'
"samba" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--exclude-env=none'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
@@ -107,11 +107,11 @@ tasks = {
" --cross-answers=./bin-xe/cross-answers.txt --with-selftest-prefix=./bin-xa/ab" + samba_configure_params, "text/plain"),
("compare-results", "script/compare_cc_results.py ./bin/c4che/default.cache.py ./bin-xe/c4che/default.cache.py ./bin-xa/c4che/default.cache.py", "text/plain")],
- # test build with -O3 -- catches extra warnings and bugs, tests the ad_dc environments
+ # test build with -O3 -- catches extra warnings and bugs, tests the ad_dc and none environments
"samba-o3" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
("configure", "ADDITIONAL_CFLAGS='-O3' ./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make quicktest FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_dc'", "text/plain"),
+ ("test", "make quicktest FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_dc --include-env=none'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
--
2.9.4
From 73acd71ea86062c36578ea3275cb99cb96efaab2 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 30 Jun 2017 11:15:40 +1200
Subject: [PATCH 4/5] autobuild: Run nt4_dc and nt4_member tests in parallel
These do not interact with the main AD DC environments, so can run in parallel
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
script/autobuild.py | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 6ebc818..6127fb3 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -25,6 +25,7 @@ cleanup_list = []
builddirs = {
"ctdb" : "ctdb",
"samba" : ".",
+ "samba-nt4" : ".",
"samba-xc" : ".",
"samba-o3" : ".",
"samba-ctdb" : ".",
@@ -46,6 +47,7 @@ builddirs = {
defaulttasks = [ "ctdb",
"samba",
+ "samba-nt4",
"samba-xc",
"samba-o3",
"samba-ctdb",
@@ -87,14 +89,22 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
- # We have 'test' before 'install' because, 'test' should work without 'install'
+ # We have 'test' before 'install' because, 'test' should work without 'install (runs ad_dc_ntvfs and all the other envs)'
"samba" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
- ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--exclude-env=none'", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--exclude-env=nt4_dc --exclude-env=nt4_member --exclude-env=ad_dc --exclude-env=none'", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
+ # We split out this so the isolated nt4_dc tests do not wait for ad_dc or ad_dc_ntvfs tests (which are long)
+ "samba-nt4" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 TESTS='--include-env=nt4_dc --include-env=nt4_member'", "text/plain"),
+ ("install", "make install", "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
+ ("clean", "make clean", "text/plain") ],
+
"samba-test-only" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab --abi-check-disable" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", "make test FAIL_IMMEDIATELY=1 TESTS=${TESTS}", "text/plain") ],
--
2.9.4
From 2a32b150d64e1a2100e0bea37b69b9b76c331b99 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Fri, 30 Jun 2017 11:44:58 +1200
Subject: [PATCH 5/5] travis-ci: Run new samba-nt4 environment
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
---
.travis.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.travis.yml b/.travis.yml
index 4c68c72..7d5b4cd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,6 +14,7 @@ env:
- TASK=samba-o3
- TASK=samba-nopython
- TASK=samba-systemkrb5
+ - TASK=samba-nt4
- TASK=ldb
- TASK=tdb
- TASK=talloc
--
2.9.4
More information about the samba-technical
mailing list