[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Wed Dec 11 04:22:02 UTC 2019
The branch, master has been updated
via 545711ffeaa lib/fuzzing: Fix argument order to ldb_filter_from_tree in fuzz_ldb_parse_tree
via e6fc8e79aee lib/fuzzing: Split up automatically build fuzzers into TYPE_{IN,OUT,STRUCT}
via 6e5aefc2d30 lib/fuzzing: Ensure mem_ctx is freed each time fuzz_ldb_parse_tree is run
via 0be0c044b67 autobuild: extend autobuild with samba-fuzz job to build the fuzzers in AFL mode using oss-fuzz scripts
via d349d344f88 autobuild.py: Avoid listing jobs twice
via df38d511045 autobuild.py: Add missing samba-simpleserver job
via de02a553999 .gitlab-ci.yml: Align tasks with "pages" dependency to get comprehensive code coverage
from fbd97ee8223 smbd: Fix a leases.tdb record leak
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 545711ffeaaa12689a4e9a14f23ba5a4afef2180
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Dec 11 14:09:25 2019 +1300
lib/fuzzing: Fix argument order to ldb_filter_from_tree in fuzz_ldb_parse_tree
Found by the oss-fuzz CI tooling.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed Dec 11 04:21:28 UTC 2019 on sn-devel-184
commit e6fc8e79aee289f79b5fba7b7c2974d476702b1e
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Dec 11 13:03:43 2019 +1300
lib/fuzzing: Split up automatically build fuzzers into TYPE_{IN,OUT,STRUCT}
The advise is that a fuzz target should be as small as possible
so we split this up. Splitting up by function would build too
many fuzzers, but this should help a little.
See for example:
https://github.com/google/fuzzing/blob/master/docs/good-fuzz-target.md#large-apis
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
commit 6e5aefc2d3095f0fb6535e6ab3a4ec12b898794f
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Dec 11 12:48:24 2019 +1300
lib/fuzzing: Ensure mem_ctx is freed each time fuzz_ldb_parse_tree is run
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
commit 0be0c044b6769da39b0feb8d7b655a2760413086
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Dec 4 22:52:06 2019 +1300
autobuild: extend autobuild with samba-fuzz job to build the fuzzers in AFL mode using oss-fuzz scripts
This helps ensure the build_samba.sh file keeps working and the fuzzers build
(because they are excluded from the main build).
This is not in the default autobuild because it uses too much
space on sn-devel (4GB).
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
commit d349d344f8872e25526fdb76a38523477baa4366
Author: Andrew Bartlett <abartlet at samba.org>
Date: Tue Dec 10 10:59:49 2019 +1300
autobuild.py: Avoid listing jobs twice
We use the tasks table instead, to avoid the issue shown in the previous commit.
Now we just have to keep .gitlab-ci.yml and the tasks table in sync.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Andreas Schneider <asn at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
commit df38d511045311b9b7c68c94aad800c69088e331
Author: Andrew Bartlett <abartlet at samba.org>
Date: Tue Dec 10 10:58:47 2019 +1300
autobuild.py: Add missing samba-simpleserver job
This was missed when the job was split out in f0e8dd1a08698884209873bb84002d7b34db016c.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Andreas Schneider <asn at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
commit de02a553999903058dd69ea8aaff2e248a6ad55c
Author: Andrew Bartlett <abartlet at samba.org>
Date: Tue Dec 10 10:47:12 2019 +1300
.gitlab-ci.yml: Align tasks with "pages" dependency to get comprehensive code coverage
These two lists can get out of skew very easily.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Andreas Schneider <asn at samba.org>
Reviewed-by: Gary Lockyer <gary at samba.org>
-----------------------------------------------------------------------
Summary of changes:
.gitlab-ci.yml | 39 ++++++++++++++++++----------
buildtools/wafsamba/samba_pidl.py | 14 +++++++++-
lib/fuzzing/fuzz_ldb_parse_tree.c | 7 ++---
lib/fuzzing/fuzz_ndr_X.c | 14 ++++++++++
lib/fuzzing/oss-fuzz/build_samba.sh | 7 ++++-
lib/fuzzing/oss-fuzz/check_build.sh | 25 ++++++++++++++++++
lib/fuzzing/wscript_build | 8 +++---
script/autobuild.py | 52 ++++++++++++++-----------------------
8 files changed, 112 insertions(+), 54 deletions(-)
create mode 100755 lib/fuzzing/oss-fuzz/check_build.sh
Changeset truncated at 500 lines:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 515fb117902..78710f55796 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -99,6 +99,10 @@ include:
- echo "Running cmd script/autobuild.py $AUTOBUILD_JOB_NAME $SAMBA_CI_AUTOBUILD_ENABLE_COVERAGE --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase"
- script/autobuild.py $AUTOBUILD_JOB_NAME $SAMBA_CI_AUTOBUILD_ENABLE_COVERAGE --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+# Ensure when adding a new job below that you also add it to
+# the dependencies for 'pages' below for the code coverage page
+# generation.
+
others:
extends: .shared_template
script:
@@ -151,6 +155,9 @@ samba-libs:
samba-static:
extends: .shared_template
+samba-fuzz:
+ extends: .shared_template
+
ctdb:
extends: .shared_template
@@ -211,31 +218,35 @@ pages:
- docker
- shared
dependencies: # tell gitlab to download artifacts for these jobs
+ - others
- samba
- - samba-ad-dc-1
- - samba-ad-dc-1-mitkrb5
+ - samba-mitkrb5
+ - samba-none-env
+ - samba-nopython
+ - samba-nopython-py2
+ - samba-xc
+ - samba-admem
- samba-ad-dc-2
- samba-ad-dc-3
- samba-ad-dc-4
- - samba-ad-dc-4-mitkrb5
- samba-ad-dc-5
- samba-ad-dc-6
- - samba-ad-dc-backup
+ - samba-libs
+ - samba-static
+ - samba-fuzz
+ # - ctdb # TODO
+ - samba-ctdb
- samba-ad-dc-ntvfs
- - samba-admem
- samba-admem-mit
- - samba-ctdb
+ - samba-ad-dc-4-mitkrb5
+ - samba-ad-dc-backup
+ - samba-simpleserver
- samba-fileserver
- - samba-libs
- - samba-none-env
- - samba-nopython
- - samba-nopython-py2
+ - samba-ktest-heimdal
+ - samba-ad-dc-1
- samba-nt4
- samba-schemaupgrade
- - samba-static
- - samba-xc
- # - ctdb # TODO
- - others
+ - samba-ad-dc-1-mitkrb5
script:
- ./configure.developer
- make -j
diff --git a/buildtools/wafsamba/samba_pidl.py b/buildtools/wafsamba/samba_pidl.py
index b92120edd23..d7e1247aa20 100644
--- a/buildtools/wafsamba/samba_pidl.py
+++ b/buildtools/wafsamba/samba_pidl.py
@@ -123,7 +123,19 @@ def SAMBA_PIDL_LIST(bld, name, source,
# the fuzzers rely
if generate_tables and generate_fuzzers:
interface = p[0:-4] # strip off the .idl suffix
- bld.SAMBA_NDR_FUZZ(interface, auto_deps=True)
+ bld.SAMBA_NDR_FUZZ(interface,
+ auto_deps=True,
+ fuzz_type="TYPE_STRUCT")
+
+ # Only generate the TYPE_STRUCT fuzzer if this isn't
+ # really DCE/RPC
+ if '--client' in options:
+ bld.SAMBA_NDR_FUZZ(interface,
+ auto_deps=True,
+ fuzz_type="TYPE_IN")
+ bld.SAMBA_NDR_FUZZ(interface,
+ auto_deps=True,
+ fuzz_type="TYPE_OUT")
Build.BuildContext.SAMBA_PIDL_LIST = SAMBA_PIDL_LIST
diff --git a/lib/fuzzing/fuzz_ldb_parse_tree.c b/lib/fuzzing/fuzz_ldb_parse_tree.c
index bbcdeab2e67..e22dd776110 100644
--- a/lib/fuzzing/fuzz_ldb_parse_tree.c
+++ b/lib/fuzzing/fuzz_ldb_parse_tree.c
@@ -33,19 +33,20 @@ int LLVMFuzzerTestOneInput(uint8_t *buf, size_t len)
char *filter;
if (len < 1) {
- return 0;
+ goto out;
}
filter = talloc_strndup(mem_ctx, (const char*)buf, len);
if (filter == NULL) {
- return 0;
+ goto out;
}
tree = ldb_parse_tree(mem_ctx, filter);
- (void)ldb_filter_from_tree(tree, mem_ctx);
+ (void)ldb_filter_from_tree(mem_ctx, tree);
+out:
talloc_free(mem_ctx);
return 0;
diff --git a/lib/fuzzing/fuzz_ndr_X.c b/lib/fuzzing/fuzz_ndr_X.c
index cdc9de50a8c..5fc21dcef26 100644
--- a/lib/fuzzing/fuzz_ndr_X.c
+++ b/lib/fuzzing/fuzz_ndr_X.c
@@ -192,6 +192,20 @@ int LLVMFuzzerTestOneInput(uint8_t *data, size_t size) {
function = SVAL(data, 2);
type = fuzz_packet_flags & 3;
+
+#ifdef FUZZ_TYPE
+ /*
+ * Fuzz targets should have as small an interface as possible.
+ * This allows us to create 3 binaries for most pipes,
+ * TYPE_IN, TYPE_OUT and TYPE_STRUCT
+ *
+ * We keep the header format, and just exit early if it does
+ * not match.
+ */
+ if (type != FUZZ_TYPE) {
+ return 0;
+ }
+#endif
#endif
switch (type) {
diff --git a/lib/fuzzing/oss-fuzz/build_samba.sh b/lib/fuzzing/oss-fuzz/build_samba.sh
index e4783442b20..63b81af0810 100755
--- a/lib/fuzzing/oss-fuzz/build_samba.sh
+++ b/lib/fuzzing/oss-fuzz/build_samba.sh
@@ -11,6 +11,9 @@
#
# CFLAGS are supplied by the caller, eg the oss-fuzz compile command
#
+# Additional arguments are passed to configure, to allow this to be
+# tested in autobuild.py
+#
ADDITIONAL_CFLAGS="$CFLAGS"
export ADDITIONAL_CFLAGS
CFLAGS=""
@@ -42,7 +45,9 @@ esac
--disable-warnings-as-errors \
--abi-check-disable \
--fuzz-target-ldflags="$LIB_FUZZING_ENGINE" \
- --nonshared-binary=ALL LINK_CC="$CXX"
+ --nonshared-binary=ALL \
+ "$@" \
+ LINK_CC="$CXX"
make -j
diff --git a/lib/fuzzing/oss-fuzz/check_build.sh b/lib/fuzzing/oss-fuzz/check_build.sh
new file mode 100755
index 00000000000..cc69cf26418
--- /dev/null
+++ b/lib/fuzzing/oss-fuzz/check_build.sh
@@ -0,0 +1,25 @@
+#!/bin/sh -eux
+#
+# A very simple check script to confirm we still provide binaries
+# that look like the targets oss-fuzz wants.
+#
+# A much stronger check is availble in oss-fuzz via
+# infra/helper.py check_build samba
+#
+
+# oss-fuzz provides an OUT variable, so for clarity this script
+# uses the same. See build_samba.sh
+OUT=$1
+
+# build_samba.sh will have put a non-zero number of fuzzers here. If
+# there are none, this will fail as it becomes literally fuzz_*
+for bin in $OUT/fuzz_*
+do
+ # Confirm that the chrpath was reset to lib/ in the same directory
+ # as the binary
+ chrpath -l $bin | grep 'RUNPATH=$ORIGIN/lib'
+
+ # Confirm that we link to at least some libraries in this
+ # directory (shows that the libraries were found and copied).
+ ldd $bin | grep "$OUT/lib"
+done
diff --git a/lib/fuzzing/wscript_build b/lib/fuzzing/wscript_build
index 191aa69b6d7..e77eea88df5 100644
--- a/lib/fuzzing/wscript_build
+++ b/lib/fuzzing/wscript_build
@@ -59,10 +59,12 @@ def SAMBA_NDR_FUZZ(bld, interface, auto_deps=False,
fuzz_src = os.path.join(fuzz_reldir, 'fuzz_ndr_X.c')
cflags = "-D FUZZ_PIPE_TABLE=ndr_table_%s" % interface
+ if fuzz_type:
+ name += "_%s" % (fuzz_type)
+ cflags += " -D FUZZ_TYPE=%s " % (fuzz_type)
if fuzz_type and fuzz_function:
- name += "_%s_%d" % (fuzz_type, fuzz_function)
- cflags += " -D FUZZ_TYPE=%s -DFUZZ_FUNCTION=%d" % (fuzz_type,
- fuzz_function)
+ name += "_%d" % (fuzz_function)
+ cflags += " -D FUZZ_FUNCTION=%d" % (fuzz_function)
fuzz_named_src = os.path.join(fuzz_reldir,
'%s.c' % (name))
diff --git a/script/autobuild.py b/script/autobuild.py
index 85167cfa993..2a0b7da53e5 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -121,32 +121,6 @@ cleanup_list = []
builddirs = {
"ctdb": "ctdb",
- "samba": ".",
- "samba-mitkrb5": ".",
- "samba-nt4": ".",
- "samba-fileserver": ".",
- "samba-ktest-heimdal": ".",
- "samba-admem": ".",
- "samba-admem-mit": ".",
- "samba-xc": ".",
- "samba-o3": ".",
- "samba-ctdb": ".",
- "samba-libs": ".",
- "samba-static": ".",
- "samba-none-env": ".",
- "samba-ad-dc-1": ".",
- "samba-ad-dc-1-mitkrb5": ".",
- "samba-ad-dc-2": ".",
- "samba-ad-dc-3": ".",
- "samba-ad-dc-4": ".",
- "samba-ad-dc-4-mitkrb5": ".",
- "samba-ad-dc-5": ".",
- "samba-ad-dc-6": ".",
- "samba-ad-dc-ntvfs": ".",
- "samba-ad-dc-backup": ".",
- "samba-nopython": ".",
- "samba-nopython-py2": ".",
- "samba-schemaupgrade": ".",
"ldb": "lib/ldb",
"tdb": "lib/tdb",
"talloc": "lib/talloc",
@@ -155,11 +129,6 @@ builddirs = {
"pidl": "pidl"
}
-defaulttasks = list(builddirs.keys())
-
-if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
- defaulttasks.remove("samba-o3")
-
ctdb_configure_params = " --enable-developer ${PREFIX}"
samba_configure_params = " ${ENABLE_COVERAGE} ${PREFIX} --with-profiling-data"
@@ -207,6 +176,9 @@ def make_test(
return ' '.join([cmd] + _options)
+# When updating this list, also update .gitlab-ci.yml to add the job
+# and to make it a dependency of 'page' for the coverage report.
+
tasks = {
"ctdb": [
("random-sleep", random_sleep(300, 900)),
@@ -652,7 +624,14 @@ tasks = {
# retry with nonshared smbd and smbtorture
("nonshared-distclean", "make distclean"),
("nonshared-configure", "./configure.developer " + samba_configure_params + " --bundled-libraries=talloc,tdb,pytdb,ldb,pyldb,tevent,pytevent --with-static-modules=ALL --nonshared-binary=smbtorture,smbd/smbd"),
- ("nonshared-make", "make -j"),
+ ("nonshared-make", "make -j")
+ ],
+
+ "samba-fuzz": [
+ # build the fuzzers (static) via the oss-fuzz script
+ ("fuzzers-mkdir-prefix", "mkdir -p ${PREFIX_DIR}"),
+ ("fuzzers-build", "OUT=${PREFIX_DIR} LIB_FUZZING_ENGINE= SANITIZER=address CXX= CFLAGS= ./lib/fuzzing/oss-fuzz/build_samba.sh --enable-afl"),
+ ("fuzzers-check", "./lib/fuzzing/oss-fuzz/check_build.sh ${PREFIX_DIR}")
],
# Test Samba without python still builds. When this test fails
@@ -815,6 +794,15 @@ tasks = {
'fail': [("fail", 'echo failing && /bin/false')],
}
+defaulttasks = list(tasks.keys())
+
+defaulttasks.remove("pass")
+defaulttasks.remove("fail")
+defaulttasks.remove("samba-test-only")
+defaulttasks.remove("samba-fuzz")
+if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
+ defaulttasks.remove("samba-o3")
+
def do_print(msg):
print("%s" % msg)
--
Samba Shared Repository
More information about the samba-cvs
mailing list