[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Wed May 20 06:11:03 MDT 2015
The branch, master has been updated
via 88d1b44 autobuild: Add test for cross-compilation infrastructure
via 4a0d599 waf: add a script that compares configurations
via 13ce285 waf: add a cross-execute program that runs natively
via 1c58fee waf: add a sample run-on-target script
via 25b1721 waf: Fix parsing of cross-answers file in case answer includes a colon
via 56114bb waf: allow cross-execute program to have arguments with spaces
via c244e7d waf: improve readability of cross-answers generated by cross-execute
via eb696df Adds a new mode to samba cross-compiling.
via 3fb6173 waf: sanitize and fix added cross answer
via 843c373 waf: always close cross answers file
from 0dcf535 Factor out submodule presence checking.
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 88d1b44d150c5eaeea32c08f98c2b26f44dbeede
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Mon May 4 10:21:53 2015 +0300
autobuild: Add test for cross-compilation infrastructure
With added option to generate cross-answers by cross-execute script,
a test is added to verify that at least for tested build environments,
the infrastructure is self-consistent.
The test runs samba configuration in three modes:
1. Native
2. Cross compile with cross-execute, using the build machine as its own
target
3. Cross compile with cross-answers, using the answers generated in
step 2
The build environments created by the three processes are then tested to
be equal.
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed May 20 14:10:05 CEST 2015 on sn-devel-104
commit 4a0d599518bc6312e887d1bb4c2b9063c1c78d83
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Tue May 19 11:50:41 2015 +0300
waf: add a script that compares configurations
This script is intended for use with the cross-build infrastructure
self-test during autobuild.
The script takes two or more "default.cache.py" files which are the
result of the waf configure process, and performs an intelligent
comparison - allowing for a limited set of variables to be different.
If the configurations are identical (apart from allowed differences) then
the script exit with 0 code.
If there are differences, the script outputs them to standard output
in unified diff format.
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit 13ce285b24d4f1a3e9e1e2f6eb45a53ff9f2d642
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Tue May 19 11:44:03 2015 +0300
waf: add a cross-execute program that runs natively
This added script is for supporting self-testing of the cross-build
infrastructure using autobuild.
The script can be used as the --cross-execute parameter
to samba's configure process, but it actually runs the program
natively on the build machine. This can be done if the build is
actually a native build disguised as a cross-build, i.e. native
toolchain is being used.
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit 1c58fee706e02410966d3f9db89a7525fd2cb8a7
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Mon May 4 22:03:54 2015 +0300
waf: add a sample run-on-target script
This is a sample script that can be used with the --cross-execute
configure switch to run configure tests on a remote target for which
samba is being cross-compiled.
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit 25b17210753117170ba1e40a3cbda0b97e208199
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Mon May 4 09:12:45 2015 +0300
waf: Fix parsing of cross-answers file in case answer includes a colon
The answer provided in the cross-answers file may include a colon,
as in:
Checking uname version type: "#57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014"
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit 56114bb6778a3515cf8da34a1413b86fe1f904ca
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Tue May 19 11:09:55 2015 +0300
waf: allow cross-execute program to have arguments with spaces
When cross-compiling samba using the cross-execute method,
allow the cross-excute command to have arguments that contain
spaces by quoting them.
For example:
./configure --cross-compile '--cross-execute=ce-program "par am"'
In this case, for each cross test whose binary is /path/testprog,
waf shall run the equivalent of running from a shell:
ce-program "par am" /path/testprog
This is useful for passing an arbitrary argument list to a subprocess
of the cross-execute program, such as in buildtools/examples/run_on_target.py
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit c244e7dc09e9c98f678cbe716fe2e640de0bd4c0
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Sun May 3 22:56:15 2015 +0300
waf: improve readability of cross-answers generated by cross-execute
When generating a result for cross-answers from the (retcode, retstring) tuple:
- (0, "output") indicated as "output"
- 1 is interpreted as generic fail code, instead of 255, because most
if not all tests fail with 1 as exit code rather than 255
- For failing test, use NO instead of FAIL, because that's not
necessarily a failure (it could mean that something is NOT
broken)
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit eb696df4936bc48bb8fa7077bb5b45e5e2e6cb01
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Mon May 18 21:15:19 2015 +0300
Adds a new mode to samba cross-compiling.
When both --cross-answers and --cross-execute are set, this means:
- Use cross-answers
- If answer is unknown, then instead of adding UNKNOWN to the cross-answers
file and failing configure, the new mode runs cross-execute to determine the
answer and adds that to the cross-answers file.
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit 3fb617310df73795fbb761253f27052ae5afa40b
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Mon May 18 21:12:06 2015 +0300
waf: sanitize and fix added cross answer
When configuring samba for cross-compilation using the cross-answers
method, the function add_answer receives the standard output and exit code
of a configuration test and updates the cross-answers file accordingly.
This patch sanitizes the standard output to conform to the cross-answers
file format - one line of output. It also adds a missing newline.
(Note - at this point add_answer is only ever called with empty output
but this change is significant for the reminder of this patchset)
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
commit 843c373d008cd51940f08a7a291976b6e80a5fbf
Author: Uri Simchoni <urisimchoni at gmail.com>
Date: Mon May 18 20:40:11 2015 +0300
waf: always close cross answers file
When configuring samba for cross-compilation using the
cross-answers file, the file is not closed in a couple of
cases - fix that.
Signed-off-by: Uri Simchoni <urisimchoni at gmail.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Alexander Bokovoy <ab at samba.org>
-----------------------------------------------------------------------
Summary of changes:
buildtools/examples/run_on_target.py | 148 +++++++++++++++++++++++++++++++++++
buildtools/wafsamba/samba_cross.py | 75 +++++++++++++-----
script/autobuild.py | 18 ++++-
script/compare_cc_results.py | 40 ++++++++++
script/identity_cc.sh | 10 +++
5 files changed, 269 insertions(+), 22 deletions(-)
create mode 100755 buildtools/examples/run_on_target.py
create mode 100755 script/compare_cc_results.py
create mode 100755 script/identity_cc.sh
Changeset truncated at 500 lines:
diff --git a/buildtools/examples/run_on_target.py b/buildtools/examples/run_on_target.py
new file mode 100755
index 0000000..8322759
--- /dev/null
+++ b/buildtools/examples/run_on_target.py
@@ -0,0 +1,148 @@
+#!/usr/bin/env python
+
+#
+# Sample run-on-target script
+# This is a script that can be used as cross-execute parameter to samba
+# configuration process, running the command on a remote target for which
+# the cross-compiled configure test was compiled.
+#
+# To use:
+# ./configure \
+# --cross-compile \
+# '--cross-execute=./buildtools/example/run_on_target.py --host=<host>'
+#
+# A more elaborate example:
+# ./configure \
+# --cross-compile \
+# '--cross-execute=./buildtools/example/run_on_target.py --host=<host> --user=<user> "--ssh=ssh -i <some key file>" --destdir=/path/to/dir'
+#
+# Typically this is to be used also with --cross-answers, so that the
+# cross answers file gets built and further builds can be made without
+# the help of a remote target.
+#
+# The following assumptions are made:
+# 1. rsync is available on build machine and target machine
+# 2. A running ssh service on target machine with password-less shell login
+# 3. A directory writable by the password-less login user
+# 4. The tests on the target can run and provide reliable results
+# from the login account's home directory. This is significant
+# for example in locking tests which
+# create files in the current directory. As a workaround to this
+# assumption, the TESTDIR environment variable can be set on the target
+# (using ssh command line or server config) and the tests shall
+# chdir to that directory.
+#
+
+import sys
+import os
+import subprocess
+from optparse import OptionParser
+
+# those are defaults, but can be overidden using command line
+SSH = 'ssh'
+USER = None
+HOST = 'localhost'
+
+
+def xfer_files(ssh, srcdir, host, user, targ_destdir):
+ """Transfer executable files to target
+
+ Use rsync to copy the directory containing program to run
+ INTO a destination directory on the target. An exact copy
+ of the source directory is created on the target machine,
+ possibly deleting files on the target machine which do not
+ exist on the source directory.
+
+ The idea is that the test may include files in addition to
+ the compiled binary, and all of those files reside alongside
+ the binary in a source directory.
+
+ For example, if the test to run is /foo/bar/test and the
+ destination directory on the target is /tbaz, then /tbaz/bar
+ on the target shall be an exact copy of /foo/bar on the source,
+ including deletion of files inside /tbaz/bar which do not exist
+ on the source.
+ """
+
+ userhost = host
+ if user:
+ userhost = '%s@%s' % (user, host)
+
+ cmd = 'rsync --verbose -rl --ignore-times --delete -e "%s" %s %s:%s/' % \
+ (ssh, srcdir, userhost, targ_destdir)
+ p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ (out, err) = p.communicate()
+ if p.returncode != 0:
+ raise Exception('failed syncing files\n stdout:\n%s\nstderr:%s\n'
+ % (out, err))
+
+
+def exec_remote(ssh, host, user, destdir, targdir, prog, args):
+ """Run a test on the target
+
+ Using password-less ssh, run the compiled binary on the target.
+
+ An assumption is that there's no need to cd into the target dir,
+ same as there's no need to do it on a native build.
+ """
+ userhost = host
+ if user:
+ userhost = '%s@%s' % (user, host)
+
+ cmd = '%s %s %s/%s/%s' % (ssh, userhost, destdir, targdir, prog)
+ if args:
+ cmd = cmd + ' ' + ' '.join(args)
+ p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ (out, err) = p.communicate()
+ return (p.returncode, out)
+
+
+def main(argv):
+ usage = "usage: %prog [options] <prog> [args]"
+ parser = OptionParser(usage)
+
+ parser.add_option('--ssh', help="SSH client and additional flags",
+ default=SSH)
+ parser.add_option('--host', help="target host name or IP address",
+ default=HOST)
+ parser.add_option('--user', help="login user on target",
+ default=USER)
+ parser.add_option('--destdir', help="work directory on target",
+ default='~')
+
+ (options, args) = parser.parse_args(argv)
+ if len(args) < 1:
+ parser.error("please supply test program to run")
+
+ progpath = args[0]
+
+ # assume that a test that was not compiled fails (e.g. getconf)
+ if progpath[0] != '/':
+ return (1, "")
+
+ progdir = os.path.dirname(progpath)
+ prog = os.path.basename(progpath)
+ targ_progdir = os.path.basename(progdir)
+
+ xfer_files(
+ options.ssh,
+ progdir,
+ options.host,
+ options.user,
+ options.destdir)
+
+ (rc, out) = exec_remote(options.ssh,
+ options.host,
+ options.user,
+ options.destdir,
+ targ_progdir,
+ prog, args[1:])
+ return (rc, out)
+
+
+if __name__ == '__main__':
+ (rc, out) = main(sys.argv[1:])
+ sys.stdout.write(out)
+ sys.exit(rc)
diff --git a/buildtools/wafsamba/samba_cross.py b/buildtools/wafsamba/samba_cross.py
index 3838e34..ed3af1e 100644
--- a/buildtools/wafsamba/samba_cross.py
+++ b/buildtools/wafsamba/samba_cross.py
@@ -2,11 +2,12 @@
import Utils, Logs, sys, os, Options, re
from Configure import conf
+import shlex
real_Popen = None
ANSWER_UNKNOWN = (254, "")
-ANSWER_FAIL = (255, "")
+ANSWER_NO = (1, "")
ANSWER_OK = (0, "")
cross_answers_incomplete = False
@@ -19,15 +20,27 @@ def add_answer(ca_file, msg, answer):
except:
Logs.error("Unable to open cross-answers file %s" % ca_file)
sys.exit(1)
+ (retcode, retstring) = answer
+ # if retstring is more than one line then we probably
+ # don't care about its actual content (the tests should
+ # yield one-line output in order to comply with the cross-answer
+ # format)
+ retstring = retstring.strip()
+ if len(retstring.split('\n')) > 1:
+ retstring = ''
+ answer = (retcode, retstring)
+
if answer == ANSWER_OK:
f.write('%s: OK\n' % msg)
elif answer == ANSWER_UNKNOWN:
f.write('%s: UNKNOWN\n' % msg)
- elif answer == ANSWER_FAIL:
- f.write('%s: FAIL\n' % msg)
+ elif answer == ANSWER_NO:
+ f.write('%s: NO\n' % msg)
else:
- (retcode, retstring) = answer
- f.write('%s: (%d, "%s")' % (msg, retcode, retstring))
+ if retcode == 0:
+ f.write('%s: "%s"\n' % (msg, retstring))
+ else:
+ f.write('%s: (%d, "%s")\n' % (msg, retcode, retstring))
f.close()
@@ -36,14 +49,13 @@ def cross_answer(ca_file, msg):
try:
f = open(ca_file, 'r')
except:
- add_answer(ca_file, msg, ANSWER_UNKNOWN)
return ANSWER_UNKNOWN
for line in f:
line = line.strip()
if line == '' or line[0] == '#':
continue
if line.find(':') != -1:
- a = line.split(':')
+ a = line.split(':', 1)
thismsg = a[0].strip()
if thismsg != msg:
continue
@@ -56,10 +68,12 @@ def cross_answer(ca_file, msg):
return ANSWER_UNKNOWN
elif ans == "FAIL" or ans == "NO":
f.close()
- return ANSWER_FAIL
+ return ANSWER_NO
elif ans[0] == '"':
+ f.close()
return (0, ans.strip('"'))
elif ans[0] == "'":
+ f.close()
return (0, ans.strip("'"))
else:
m = re.match('\(\s*(-?\d+)\s*,\s*\"(.*)\"\s*\)', ans)
@@ -69,7 +83,6 @@ def cross_answer(ca_file, msg):
else:
raise Utils.WafError("Bad answer format '%s' in %s" % (line, ca_file))
f.close()
- add_answer(ca_file, msg, ANSWER_UNKNOWN)
return ANSWER_UNKNOWN
@@ -77,24 +90,47 @@ class cross_Popen(Utils.pproc.Popen):
'''cross-compilation wrapper for Popen'''
def __init__(*k, **kw):
(obj, args) = k
-
- if '--cross-execute' in args:
- # when --cross-execute is set, then change the arguments
- # to use the cross emulator
- i = args.index('--cross-execute')
- newargs = args[i+1].split()
- newargs.extend(args[0:i])
- args = newargs
- elif '--cross-answers' in args:
+ use_answers = False
+ ans = ANSWER_UNKNOWN
+
+ # Three possibilities:
+ # 1. Only cross-answers - try the cross-answers file, and if
+ # there's no corresponding answer, add to the file and mark
+ # the configure process as unfinished.
+ # 2. Only cross-execute - get the answer from cross-execute
+ # 3. Both - try the cross-answers file, and if there is no
+ # corresponding answer - use cross-execute to get an answer,
+ # and add that answer to the file.
+ if '--cross-answers' in args:
# when --cross-answers is set, then change the arguments
# to use the cross answers if available
+ use_answers = True
i = args.index('--cross-answers')
ca_file = args[i+1]
msg = args[i+2]
ans = cross_answer(ca_file, msg)
+
+ if '--cross-execute' in args and ans == ANSWER_UNKNOWN:
+ # when --cross-execute is set, then change the arguments
+ # to use the cross emulator
+ i = args.index('--cross-execute')
+ newargs = shlex.split(args[i+1])
+ newargs.extend(args[0:i])
+ if use_answers:
+ p = real_Popen(newargs,
+ stdout=Utils.pproc.PIPE,
+ stderr=Utils.pproc.PIPE)
+ ce_out, ce_err = p.communicate()
+ ans = (p.returncode, ce_out)
+ add_answer(ca_file, msg, ans)
+ else:
+ args = newargs
+
+ if use_answers:
if ans == ANSWER_UNKNOWN:
global cross_answers_incomplete
cross_answers_incomplete = True
+ add_answer(ca_file, msg, ans)
(retcode, retstring) = ans
args = ['/bin/sh', '-c', "echo -n '%s'; exit %d" % (retstring, retcode)]
real_Popen.__init__(*(obj, args), **kw)
@@ -115,7 +151,8 @@ def SAMBA_CROSS_ARGS(conf, msg=None):
if conf.env.CROSS_EXECUTE:
ret.extend(['--cross-execute', conf.env.CROSS_EXECUTE])
- elif conf.env.CROSS_ANSWERS:
+
+ if conf.env.CROSS_ANSWERS:
if msg is None:
raise Utils.WafError("Cannot have NULL msg in cross-answers")
ret.extend(['--cross-answers', os.path.join(Options.launch_dir, conf.env.CROSS_ANSWERS), msg])
diff --git a/script/autobuild.py b/script/autobuild.py
index b2303f7..c697aad 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -23,6 +23,7 @@ cleanup_list = []
builddirs = {
"ctdb" : "ctdb",
"samba" : ".",
+ "samba-xc" : ".",
"samba-ctdb" : ".",
"samba-libs" : ".",
"ldb" : "lib/ldb",
@@ -36,7 +37,9 @@ builddirs = {
"retry" : "."
}
-defaulttasks = [ "ctdb", "samba", "samba-ctdb", "samba-libs", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ]
+defaulttasks = [ "ctdb", "samba", "samba-xc", "samba-ctdb", "samba-libs", "ldb", "tdb", "talloc", "replace", "tevent", "pidl" ]
+
+samba_configure_params = " --picky-developer ${PREFIX} --with-profiling-data"
tasks = {
"ctdb" : [ ("random-sleep", "../script/random-sleep.sh 60 600", "text/plain"),
@@ -48,13 +51,22 @@ tasks = {
("clean", "make clean", "text/plain") ],
# We have 'test' before 'install' because, 'test' should work without 'install'
- "samba" : [ ("configure", "./configure.developer --picky-developer ${PREFIX} --with-selftest-prefix=./bin/ab --with-profiling-data", "text/plain"),
+ "samba" : [ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", "make test FAIL_IMMEDIATELY=1", "text/plain"),
("install", "make install", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain") ],
+ # Test cross-compile infrastructure
+ "samba-xc" : [ ("configure-native", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("configure-cross-execute", "./configure.developer -b ./bin-xe --cross-compile --cross-execute=script/identity_cc.sh" \
+ " --cross-answers=./bin-xe/cross-answers.txt --with-selftest-prefix=./bin-xe/ab" + samba_configure_params, "text/plain"),
+ ("configure-cross-answers", "./configure.developer -b ./bin-xa --cross-compile" \
+ " --cross-answers=./bin-xe/cross-answers.txt --with-selftest-prefix=./bin-xa/ab" + samba_configure_params, "text/plain"),
+ ("compare-results", "script/compare_cc_results.py ./bin/c4che/default.cache.py ./bin-xe/c4che/default.cache.py ./bin-xa/c4che/default.cache.py", "text/plain")],
+
+
"samba-ctdb" : [ ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
# make sure we have tdb around:
@@ -519,7 +531,7 @@ A summary of the autobuild process is here:
%s/autobuild.log
''' % (platform.node(), failed_task, errstr, log_base)
-
+
if failed_task != 'rebase':
text += '''
You can see logs of the failed task here:
diff --git a/script/compare_cc_results.py b/script/compare_cc_results.py
new file mode 100755
index 0000000..b8ed399
--- /dev/null
+++ b/script/compare_cc_results.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python
+
+#
+# Compare the results of native and cross-compiled configure tests
+#
+
+import sys
+import difflib
+
+exceptions = ['BUILD_DIRECTORY', 'CROSS_COMPILE', 'CROSS_ANSWERS',
+ 'CROSS_EXECUTE', 'SELFTEST_PREFIX']
+
+base_lines = list()
+base_fname = ''
+
+found_diff = False
+
+for fname in sys.argv[1:]:
+ lines = list()
+ f = open(fname, 'r')
+ for line in f:
+ if len(line.split('=', 1)) == 2:
+ key = line.split('=', 1)[0].strip()
+ if key in exceptions:
+ continue
+ lines.append(line)
+ f.close()
+ if base_fname:
+ diff = list(difflib.unified_diff(base_lines,lines,base_fname,fname))
+ if diff:
+ print 'configuration files %s and %s do not match' % (base_fname, fname)
+ for l in diff:
+ sys.stdout.write(l)
+ found_diff = True
+ else:
+ base_fname = fname
+ base_lines = lines
+
+if found_diff:
+ sys.exit(1)
diff --git a/script/identity_cc.sh b/script/identity_cc.sh
new file mode 100755
index 0000000..8b5118e
--- /dev/null
+++ b/script/identity_cc.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+#An "identity cross-execute" script
+#It can be used for testing the cross-build infrastructure
+#as follows:
+#./configure --cross-compile --cross-execute=./script/identity_cc.sh
+#If the build is actually a native build, then the configuration
+#result should be just like running ./configure without --cross-compile.
+
+eval "$@"
--
Samba Shared Repository
More information about the samba-cvs
mailing list