[PR PATCH] Cloud build
Stefan Metzmacher
metze at samba.org
Mon Jan 4 08:13:07 UTC 2016
Hi Andrew,
I've reordered the commits a bit, improved the commit message
a little bit and squashed the two travis related commit.
Is it ok to push the attached patches?
metze
Am 04.01.2016 um 04:52 schrieb github at samba.org:
> There is a new pull request by abartlet against master on the Samba Samba Github repository
>
> https://github.com/abartlet/samba cloud-build-final
> https://github.com/samba-team/samba/pull/49
>
> Cloud build
> I would like to have Samba start being tested on travis-ci.org, so that we know if pull requests break the tree before they are reviewed, not after.
>
> I realise that the make -j in the static and samba-libs builds may be an issue, but otherwise these come far to close to the 50min timeout (they take 23mins with -j). I have already split up the samba-libs build into samba-libs and samba-static for similar reasons.
>
> The main samba build and test is not done at this time, as we need to come up with a new quicktest that can run in 20mins or so.
>
> The autobuild.py script has been adjusted to use cp --link rather than git checkout to reduce disk use.
>
> The CTDB and samba builds have been omitted until they are reliable.
>
> Please review/push
>
>
> A patch file from https://github.com/samba-team/samba/pull/49.patch is attached
>
-------------- next part --------------
From 67d4fa72bc843166ab10714a4f04c9e0c8f5cb70 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Wed, 21 Oct 2015 14:35:33 +1300
Subject: [PATCH 1/5] autobuild: Give a clearer failure message
This helps when autobuild.py is used in --tail mode and
where there is neither e-mail nor access to the logs.tar.gz
Working back to find where the error happened is typically
quite difficult, as many failures are actually due to the
cleanup.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
---
script/autobuild.py | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/script/autobuild.py b/script/autobuild.py
index 2d3af2a..8d1060e 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -720,6 +720,24 @@ blist.tarlogs("logs.tar.gz")
if options.email is not None:
email_failure(status, failed_task, failed_stage, failed_tag, errstr,
elapsed_time, log_base=options.log_base)
+else:
+ elapsed_minutes = elapsed_time / 60.0
+ print '''
+
+####################################################################
+
+AUTOBUILD FAILURE
+
+Your autobuild on %s failed after %.1f minutes
+when trying to test %s with the following error:
+
+ %s
+
+the autobuild has been abandoned. Please fix the error and resubmit.
+
+####################################################################
+
+''' % (platform.node(), elapsed_minutes, failed_task, errstr)
cleanup()
print(errstr)
--
1.9.1
From c90fa4e5842189e80f31a24bb18355354c7935d9 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Mon, 4 Jan 2016 14:20:54 +1300
Subject: [PATCH 2/5] autobuild: Use cp --recursive --link --archive
This works on waf based builds as source files are not modified during the build
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
---
script/autobuild.py | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 8d1060e..4b8f772 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -220,7 +220,7 @@ def run_cmd(cmd, dir=".", show=None, output=False, checkfail=True):
class builder(object):
'''handle build of one directory'''
- def __init__(self, name, sequence):
+ def __init__(self, name, sequence, cp=True):
self.name = name
self.dir = builddirs[name]
@@ -243,7 +243,10 @@ class builder(object):
cleanup_list.append(self.prefix)
os.makedirs(self.sdir)
run_cmd("rm -rf %s" % self.sdir)
- run_cmd("git clone --recursive --shared %s %s" % (test_master, self.sdir), dir=test_master, show=True)
+ if cp:
+ run_cmd("cp --recursive --link --archive %s %s" % (test_master, self.sdir), dir=test_master, show=True)
+ else:
+ run_cmd("git clone --recursive --shared %s %s" % (test_master, self.sdir), dir=test_master, show=True)
self.start_next()
def start_next(self):
@@ -282,7 +285,7 @@ class buildlist(object):
os.environ['AUTOBUILD_RANDOM_SLEEP_OVERRIDE'] = '1'
for n in tasknames:
- b = builder(n, tasks[n])
+ b = builder(n, tasks[n], cp=n is not "pidl")
self.tlist.append(b)
if options.retry:
rebase_remote = "rebaseon"
--
1.9.1
From 022053254af6a62497e43739ab99895accde1b10 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Oct 2015 10:35:47 +1300
Subject: [PATCH 3/5] autobuild: Put the static build in another stanza
This will allow it to run on a different VM in travis-ci and so keep
us below the 50min limit.
Because we do not rebuild the libs (and real world static builds are
likely to have internal libs), this changes us to build them
internally.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
---
script/autobuild.py | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 4b8f772..15cee1c3 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -26,6 +26,7 @@ builddirs = {
"samba-xc" : ".",
"samba-ctdb" : ".",
"samba-libs" : ".",
+ "samba-static" : ".",
"ldb" : "lib/ldb",
"tdb" : "lib/tdb",
"talloc" : "lib/talloc",
@@ -37,7 +38,7 @@ builddirs = {
"retry" : "."
}
-defaulttasks = [ "ctdb", "samba", "samba-xc", "samba-ctdb", "samba-libs", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ]
+defaulttasks = [ "ctdb", "samba", "samba-xc", "samba-ctdb", "samba-libs", "samba-static", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ]
samba_configure_params = " --picky-developer ${PREFIX} --with-profiling-data"
@@ -121,21 +122,22 @@ tasks = {
# retry with all modules shared
("allshared-distclean", "make distclean", "text/plain"),
("allshared-configure", samba_libs_configure_samba + " --with-shared-modules=ALL", "text/plain"),
- ("allshared-make", "make", "text/plain"),
+ ("allshared-make", "make", "text/plain")],
- # retry with all modules static
- ("allstatic-distclean", "make distclean", "text/plain"),
- ("allstatic-configure", samba_libs_configure_samba + " --with-static-modules=ALL", "text/plain"),
+ "samba-static" : [
+ ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ # build with all modules static
+ ("allstatic-configure", "./configure.developer " + samba_configure_params + " --with-static-modules=ALL", "text/plain"),
("allstatic-make", "make", "text/plain"),
# retry without any required modules
("none-distclean", "make distclean", "text/plain"),
- ("none-configure", samba_libs_configure_samba + " --with-static-modules=!FORCED,!DEFAULT --with-shared-modules=!FORCED,!DEFAULT", "text/plain"),
+ ("none-configure", "./configure.developer " + samba_configure_params + " --with-static-modules=!FORCED,!DEFAULT --with-shared-modules=!FORCED,!DEFAULT", "text/plain"),
("none-make", "make", "text/plain"),
# retry with nonshared smbd and smbtorture
("nonshared-distclean", "make distclean", "text/plain"),
- ("nonshared-configure", samba_libs_configure_base + " --bundled-libraries=talloc,tdb,pytdb,ldb,pyldb,tevent,pytevent --with-static-modules=ALL --nonshared-binary=smbtorture,smbd/smbd", "text/plain"),
+ ("nonshared-configure", "./configure.developer " + samba_configure_params + " --bundled-libraries=talloc,tdb,pytdb,ldb,pyldb,tevent,pytevent --with-static-modules=ALL --nonshared-binary=smbtorture,smbd/smbd", "text/plain"),
("nonshared-make", "make", "text/plain")],
"ldb" : [
--
1.9.1
From 61e8fcc25beac47ef8060b3da1a0f3d998db8899 Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Thu, 22 Oct 2015 14:57:10 +1300
Subject: [PATCH 4/5] autobuild: Use make -j on samba-libs/samba-static build
as well
We need this build to be fast when split out as a seperate VM
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
---
script/autobuild.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/script/autobuild.py b/script/autobuild.py
index 15cee1c3..d75931f 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -122,23 +122,23 @@ tasks = {
# retry with all modules shared
("allshared-distclean", "make distclean", "text/plain"),
("allshared-configure", samba_libs_configure_samba + " --with-shared-modules=ALL", "text/plain"),
- ("allshared-make", "make", "text/plain")],
+ ("allshared-make", "make -j", "text/plain")],
"samba-static" : [
("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
# build with all modules static
("allstatic-configure", "./configure.developer " + samba_configure_params + " --with-static-modules=ALL", "text/plain"),
- ("allstatic-make", "make", "text/plain"),
+ ("allstatic-make", "make -j", "text/plain"),
# retry without any required modules
("none-distclean", "make distclean", "text/plain"),
("none-configure", "./configure.developer " + samba_configure_params + " --with-static-modules=!FORCED,!DEFAULT --with-shared-modules=!FORCED,!DEFAULT", "text/plain"),
- ("none-make", "make", "text/plain"),
+ ("none-make", "make -j", "text/plain"),
# retry with nonshared smbd and smbtorture
("nonshared-distclean", "make distclean", "text/plain"),
("nonshared-configure", "./configure.developer " + samba_configure_params + " --bundled-libraries=talloc,tdb,pytdb,ldb,pyldb,tevent,pytevent --with-static-modules=ALL --nonshared-binary=smbtorture,smbd/smbd", "text/plain"),
- ("nonshared-make", "make", "text/plain")],
+ ("nonshared-make", "make -j", "text/plain")],
"ldb" : [
("random-sleep", "../../script/random-sleep.sh 60 600", "text/plain"),
--
1.9.1
From 60e4cd9aa9d47bbf715bd00dc08bc8321a0347ab Mon Sep 17 00:00:00 2001
From: Andrew Bartlett <abartlet at samba.org>
Date: Mon, 4 May 2015 13:44:45 +1200
Subject: [PATCH 5/5] travis: Add metadata file for the Travis CI Open Source
cloud build/test service
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
---
.travis.yml | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
create mode 100644 .travis.yml
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 0000000..9d12665
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,31 @@
+language: c
+dist: trusty
+
+sudo: true
+
+cache: ccache
+
+# Everything except samba and ctdb (long tests)
+env:
+ - TASK=samba-xc
+ - TASK=samba-ctdb
+ - TASK=samba-libs
+ - TASK=samba-static
+ - TASK=ldb
+ - TASK=tdb
+ - TASK=talloc
+ - TASK=replace
+ - TASK=tevent
+ - TASK=pidl
+
+# Fail everything after the first job fails
+matrix:
+ fast_finish: true
+
+before_install:
+ - sudo apt-get update -qq
+ - sudo apt-get install --assume-yes screen git build-essential libacl1-dev libattr1-dev libblkid-dev libgnutls-dev libreadline-dev python-dev libpam0g-dev python-dnspython gdb pkg-config libpopt-dev libldap2-dev dnsutils libbsd-dev attr krb5-user docbook-xsl libcups2-dev acl bison debhelper docbook-xml docbook-xsl flex libaio-dev libbsd-dev libcap-dev libcups2-dev libncurses5-dev libpam0g-dev libpopt-dev libreadline-dev perl perl-modules libparse-yapp-perl pkg-config python-all-dev python-dnspython xsltproc zlib1g-dev autoconf
+
+script:
+ - git fetch --unshallow
+ - ./script/autobuild.py --tail --testbase=/tmp $TASK
--
1.9.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20160104/34b68642/signature.sig>
More information about the samba-technical
mailing list