[SCM] Samba Shared Repository - branch master updated

Andrew Bartlett abartlet at samba.org
Wed Dec 11 04:22:02 UTC 2019


The branch, master has been updated
       via  545711ffeaa lib/fuzzing: Fix argument order to ldb_filter_from_tree in fuzz_ldb_parse_tree
       via  e6fc8e79aee lib/fuzzing: Split up automatically build fuzzers into TYPE_{IN,OUT,STRUCT}
       via  6e5aefc2d30 lib/fuzzing: Ensure mem_ctx is freed each time fuzz_ldb_parse_tree is run
       via  0be0c044b67 autobuild: extend autobuild with samba-fuzz job to build the fuzzers in AFL mode using oss-fuzz scripts
       via  d349d344f88 autobuild.py: Avoid listing jobs twice
       via  df38d511045 autobuild.py: Add missing samba-simpleserver job
       via  de02a553999 .gitlab-ci.yml: Align tasks with "pages" dependency to get comprehensive code coverage
      from  fbd97ee8223 smbd: Fix a leases.tdb record leak

https://git.samba.org/?p=samba.git;a=shortlog;h=master


- Log -----------------------------------------------------------------
commit 545711ffeaaa12689a4e9a14f23ba5a4afef2180
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Wed Dec 11 14:09:25 2019 +1300

    lib/fuzzing: Fix argument order to ldb_filter_from_tree in fuzz_ldb_parse_tree
    
    Found by the oss-fuzz CI tooling.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>
    
    Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
    Autobuild-Date(master): Wed Dec 11 04:21:28 UTC 2019 on sn-devel-184

commit e6fc8e79aee289f79b5fba7b7c2974d476702b1e
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Wed Dec 11 13:03:43 2019 +1300

    lib/fuzzing: Split up automatically build fuzzers into TYPE_{IN,OUT,STRUCT}
    
    The advise is that a fuzz target should be as small as possible
    so we split this up.  Splitting up by function would build too
    many fuzzers, but this should help a little.
    
    See for example:
    https://github.com/google/fuzzing/blob/master/docs/good-fuzz-target.md#large-apis
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>

commit 6e5aefc2d3095f0fb6535e6ab3a4ec12b898794f
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Wed Dec 11 12:48:24 2019 +1300

    lib/fuzzing: Ensure mem_ctx is freed each time fuzz_ldb_parse_tree is run
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>

commit 0be0c044b6769da39b0feb8d7b655a2760413086
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Wed Dec 4 22:52:06 2019 +1300

    autobuild: extend autobuild with samba-fuzz job to build the fuzzers in AFL mode using oss-fuzz scripts
    
    This helps ensure the build_samba.sh file keeps working and the fuzzers build
    (because they are excluded from the main build).
    
    This is not in the default autobuild because it uses too much
    space on sn-devel (4GB).
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>

commit d349d344f8872e25526fdb76a38523477baa4366
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Tue Dec 10 10:59:49 2019 +1300

    autobuild.py: Avoid listing jobs twice
    
    We use the tasks table instead, to avoid the issue shown in the previous commit.
    
    Now we just have to keep .gitlab-ci.yml and the tasks table in sync.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Andreas Schneider <asn at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>

commit df38d511045311b9b7c68c94aad800c69088e331
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Tue Dec 10 10:58:47 2019 +1300

    autobuild.py: Add missing samba-simpleserver job
    
    This was missed when the job was split out in f0e8dd1a08698884209873bb84002d7b34db016c.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Andreas Schneider <asn at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>

commit de02a553999903058dd69ea8aaff2e248a6ad55c
Author: Andrew Bartlett <abartlet at samba.org>
Date:   Tue Dec 10 10:47:12 2019 +1300

    .gitlab-ci.yml: Align tasks with "pages" dependency to get comprehensive code coverage
    
    These two lists can get out of skew very easily.
    
    Signed-off-by: Andrew Bartlett <abartlet at samba.org>
    Reviewed-by: Andreas Schneider <asn at samba.org>
    Reviewed-by: Gary Lockyer <gary at samba.org>

-----------------------------------------------------------------------

Summary of changes:
 .gitlab-ci.yml                      | 39 ++++++++++++++++++----------
 buildtools/wafsamba/samba_pidl.py   | 14 +++++++++-
 lib/fuzzing/fuzz_ldb_parse_tree.c   |  7 ++---
 lib/fuzzing/fuzz_ndr_X.c            | 14 ++++++++++
 lib/fuzzing/oss-fuzz/build_samba.sh |  7 ++++-
 lib/fuzzing/oss-fuzz/check_build.sh | 25 ++++++++++++++++++
 lib/fuzzing/wscript_build           |  8 +++---
 script/autobuild.py                 | 52 ++++++++++++++-----------------------
 8 files changed, 112 insertions(+), 54 deletions(-)
 create mode 100755 lib/fuzzing/oss-fuzz/check_build.sh


Changeset truncated at 500 lines:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 515fb117902..78710f55796 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -99,6 +99,10 @@ include:
     - echo "Running cmd script/autobuild.py $AUTOBUILD_JOB_NAME $SAMBA_CI_AUTOBUILD_ENABLE_COVERAGE --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase"
     - script/autobuild.py $AUTOBUILD_JOB_NAME $SAMBA_CI_AUTOBUILD_ENABLE_COVERAGE  --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
 
+# Ensure when adding a new job below that you also add it to
+# the dependencies for 'pages' below for the code coverage page
+# generation.
+
 others:
   extends: .shared_template
   script:
@@ -151,6 +155,9 @@ samba-libs:
 samba-static:
   extends: .shared_template
 
+samba-fuzz:
+  extends: .shared_template
+
 ctdb:
   extends: .shared_template
 
@@ -211,31 +218,35 @@ pages:
     - docker
     - shared
   dependencies:  # tell gitlab to download artifacts for these jobs
+    - others
     - samba
-    - samba-ad-dc-1
-    - samba-ad-dc-1-mitkrb5
+    - samba-mitkrb5
+    - samba-none-env
+    - samba-nopython
+    - samba-nopython-py2
+    - samba-xc
+    - samba-admem
     - samba-ad-dc-2
     - samba-ad-dc-3
     - samba-ad-dc-4
-    - samba-ad-dc-4-mitkrb5
     - samba-ad-dc-5
     - samba-ad-dc-6
-    - samba-ad-dc-backup
+    - samba-libs
+    - samba-static
+    - samba-fuzz
+    # - ctdb  # TODO
+    - samba-ctdb
     - samba-ad-dc-ntvfs
-    - samba-admem
     - samba-admem-mit
-    - samba-ctdb
+    - samba-ad-dc-4-mitkrb5
+    - samba-ad-dc-backup
+    - samba-simpleserver
     - samba-fileserver
-    - samba-libs
-    - samba-none-env
-    - samba-nopython
-    - samba-nopython-py2
+    - samba-ktest-heimdal
+    - samba-ad-dc-1
     - samba-nt4
     - samba-schemaupgrade
-    - samba-static
-    - samba-xc
-    # - ctdb  # TODO
-    - others
+    - samba-ad-dc-1-mitkrb5
   script:
     - ./configure.developer
     - make -j
diff --git a/buildtools/wafsamba/samba_pidl.py b/buildtools/wafsamba/samba_pidl.py
index b92120edd23..d7e1247aa20 100644
--- a/buildtools/wafsamba/samba_pidl.py
+++ b/buildtools/wafsamba/samba_pidl.py
@@ -123,7 +123,19 @@ def SAMBA_PIDL_LIST(bld, name, source,
         # the fuzzers rely
         if generate_tables and generate_fuzzers:
             interface = p[0:-4] # strip off the .idl suffix
-            bld.SAMBA_NDR_FUZZ(interface, auto_deps=True)
+            bld.SAMBA_NDR_FUZZ(interface,
+                               auto_deps=True,
+                               fuzz_type="TYPE_STRUCT")
+
+            # Only generate the TYPE_STRUCT fuzzer if this isn't
+            # really DCE/RPC
+            if '--client' in options:
+                bld.SAMBA_NDR_FUZZ(interface,
+                                   auto_deps=True,
+                                   fuzz_type="TYPE_IN")
+                bld.SAMBA_NDR_FUZZ(interface,
+                                   auto_deps=True,
+                                   fuzz_type="TYPE_OUT")
 Build.BuildContext.SAMBA_PIDL_LIST = SAMBA_PIDL_LIST
 
 
diff --git a/lib/fuzzing/fuzz_ldb_parse_tree.c b/lib/fuzzing/fuzz_ldb_parse_tree.c
index bbcdeab2e67..e22dd776110 100644
--- a/lib/fuzzing/fuzz_ldb_parse_tree.c
+++ b/lib/fuzzing/fuzz_ldb_parse_tree.c
@@ -33,19 +33,20 @@ int LLVMFuzzerTestOneInput(uint8_t *buf, size_t len)
 	char *filter;
 
 	if (len < 1) {
-		return 0;
+		goto out;
 	}
 
 	filter = talloc_strndup(mem_ctx, (const char*)buf, len);
 
 	if (filter == NULL) {
-		return 0;
+		goto out;
 	}
 
 	tree = ldb_parse_tree(mem_ctx, filter);
 
-	(void)ldb_filter_from_tree(tree, mem_ctx);
+	(void)ldb_filter_from_tree(mem_ctx, tree);
 
+out:
 	talloc_free(mem_ctx);
 
 	return 0;
diff --git a/lib/fuzzing/fuzz_ndr_X.c b/lib/fuzzing/fuzz_ndr_X.c
index cdc9de50a8c..5fc21dcef26 100644
--- a/lib/fuzzing/fuzz_ndr_X.c
+++ b/lib/fuzzing/fuzz_ndr_X.c
@@ -192,6 +192,20 @@ int LLVMFuzzerTestOneInput(uint8_t *data, size_t size) {
 	function = SVAL(data, 2);
 
 	type = fuzz_packet_flags & 3;
+
+#ifdef FUZZ_TYPE
+	/*
+	 * Fuzz targets should have as small an interface as possible.
+	 * This allows us to create 3 binaries for most pipes,
+	 * TYPE_IN, TYPE_OUT and TYPE_STRUCT
+	 *
+	 * We keep the header format, and just exit early if it does
+	 * not match.
+	 */
+	if (type != FUZZ_TYPE) {
+		return 0;
+	}
+#endif
 #endif
 
 	switch (type) {
diff --git a/lib/fuzzing/oss-fuzz/build_samba.sh b/lib/fuzzing/oss-fuzz/build_samba.sh
index e4783442b20..63b81af0810 100755
--- a/lib/fuzzing/oss-fuzz/build_samba.sh
+++ b/lib/fuzzing/oss-fuzz/build_samba.sh
@@ -11,6 +11,9 @@
 #
 # CFLAGS are supplied by the caller, eg the oss-fuzz compile command
 #
+# Additional arguments are passed to configure, to allow this to be
+# tested in autobuild.py
+#
 ADDITIONAL_CFLAGS="$CFLAGS"
 export ADDITIONAL_CFLAGS
 CFLAGS=""
@@ -42,7 +45,9 @@ esac
 	    --disable-warnings-as-errors \
 	    --abi-check-disable \
 	    --fuzz-target-ldflags="$LIB_FUZZING_ENGINE" \
-	    --nonshared-binary=ALL LINK_CC="$CXX"
+	    --nonshared-binary=ALL \
+	    "$@" \
+	    LINK_CC="$CXX"
 
 make -j
 
diff --git a/lib/fuzzing/oss-fuzz/check_build.sh b/lib/fuzzing/oss-fuzz/check_build.sh
new file mode 100755
index 00000000000..cc69cf26418
--- /dev/null
+++ b/lib/fuzzing/oss-fuzz/check_build.sh
@@ -0,0 +1,25 @@
+#!/bin/sh -eux
+#
+# A very simple check script to confirm we still provide binaries
+# that look like the targets oss-fuzz wants.
+#
+# A much stronger check is availble in oss-fuzz via
+# infra/helper.py check_build samba
+#
+
+# oss-fuzz provides an OUT variable, so for clarity this script
+# uses the same.  See build_samba.sh
+OUT=$1
+
+# build_samba.sh will have put a non-zero number of fuzzers here.  If
+# there are none, this will fail as it becomes literally fuzz_*
+for bin in $OUT/fuzz_*
+do
+    # Confirm that the chrpath was reset to lib/ in the same directory
+    # as the binary
+    chrpath -l $bin | grep 'RUNPATH=$ORIGIN/lib'
+
+    # Confirm that we link to at least some libraries in this
+    # directory (shows that the libraries were found and copied).
+    ldd $bin | grep "$OUT/lib"
+done
diff --git a/lib/fuzzing/wscript_build b/lib/fuzzing/wscript_build
index 191aa69b6d7..e77eea88df5 100644
--- a/lib/fuzzing/wscript_build
+++ b/lib/fuzzing/wscript_build
@@ -59,10 +59,12 @@ def SAMBA_NDR_FUZZ(bld, interface, auto_deps=False,
     fuzz_src = os.path.join(fuzz_reldir, 'fuzz_ndr_X.c')
 
     cflags = "-D FUZZ_PIPE_TABLE=ndr_table_%s" % interface
+    if fuzz_type:
+        name += "_%s" % (fuzz_type)
+        cflags += " -D FUZZ_TYPE=%s " % (fuzz_type)
     if fuzz_type and fuzz_function:
-        name += "_%s_%d" % (fuzz_type, fuzz_function)
-        cflags += " -D FUZZ_TYPE=%s -DFUZZ_FUNCTION=%d" % (fuzz_type,
-                                                           fuzz_function)
+        name += "_%d" % (fuzz_function)
+        cflags += " -D FUZZ_FUNCTION=%d" % (fuzz_function)
 
     fuzz_named_src = os.path.join(fuzz_reldir,
                                   '%s.c' % (name))
diff --git a/script/autobuild.py b/script/autobuild.py
index 85167cfa993..2a0b7da53e5 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -121,32 +121,6 @@ cleanup_list = []
 
 builddirs = {
     "ctdb": "ctdb",
-    "samba": ".",
-    "samba-mitkrb5": ".",
-    "samba-nt4": ".",
-    "samba-fileserver": ".",
-    "samba-ktest-heimdal": ".",
-    "samba-admem": ".",
-    "samba-admem-mit": ".",
-    "samba-xc": ".",
-    "samba-o3": ".",
-    "samba-ctdb": ".",
-    "samba-libs": ".",
-    "samba-static": ".",
-    "samba-none-env": ".",
-    "samba-ad-dc-1": ".",
-    "samba-ad-dc-1-mitkrb5": ".",
-    "samba-ad-dc-2": ".",
-    "samba-ad-dc-3": ".",
-    "samba-ad-dc-4": ".",
-    "samba-ad-dc-4-mitkrb5": ".",
-    "samba-ad-dc-5": ".",
-    "samba-ad-dc-6": ".",
-    "samba-ad-dc-ntvfs": ".",
-    "samba-ad-dc-backup": ".",
-    "samba-nopython": ".",
-    "samba-nopython-py2": ".",
-    "samba-schemaupgrade": ".",
     "ldb": "lib/ldb",
     "tdb": "lib/tdb",
     "talloc": "lib/talloc",
@@ -155,11 +129,6 @@ builddirs = {
     "pidl": "pidl"
 }
 
-defaulttasks = list(builddirs.keys())
-
-if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
-    defaulttasks.remove("samba-o3")
-
 ctdb_configure_params = " --enable-developer ${PREFIX}"
 samba_configure_params = " ${ENABLE_COVERAGE} ${PREFIX} --with-profiling-data"
 
@@ -207,6 +176,9 @@ def make_test(
     return ' '.join([cmd] + _options)
 
 
+# When updating this list, also update .gitlab-ci.yml to add the job
+# and to make it a dependency of 'page' for the coverage report.
+
 tasks = {
     "ctdb": [
         ("random-sleep", random_sleep(300, 900)),
@@ -652,7 +624,14 @@ tasks = {
         # retry with nonshared smbd and smbtorture
         ("nonshared-distclean", "make distclean"),
         ("nonshared-configure", "./configure.developer " + samba_configure_params + " --bundled-libraries=talloc,tdb,pytdb,ldb,pyldb,tevent,pytevent --with-static-modules=ALL --nonshared-binary=smbtorture,smbd/smbd"),
-        ("nonshared-make", "make -j"),
+        ("nonshared-make", "make -j")
+        ],
+
+    "samba-fuzz": [
+        # build the fuzzers (static) via the oss-fuzz script
+        ("fuzzers-mkdir-prefix", "mkdir -p ${PREFIX_DIR}"),
+        ("fuzzers-build", "OUT=${PREFIX_DIR} LIB_FUZZING_ENGINE= SANITIZER=address CXX= CFLAGS= ./lib/fuzzing/oss-fuzz/build_samba.sh --enable-afl"),
+        ("fuzzers-check", "./lib/fuzzing/oss-fuzz/check_build.sh ${PREFIX_DIR}")
         ],
 
     # Test Samba without python still builds.  When this test fails
@@ -815,6 +794,15 @@ tasks = {
     'fail': [("fail", 'echo failing && /bin/false')],
 }
 
+defaulttasks = list(tasks.keys())
+
+defaulttasks.remove("pass")
+defaulttasks.remove("fail")
+defaulttasks.remove("samba-test-only")
+defaulttasks.remove("samba-fuzz")
+if os.environ.get("AUTOBUILD_SKIP_SAMBA_O3", "0") == "1":
+    defaulttasks.remove("samba-o3")
+
 
 def do_print(msg):
     print("%s" % msg)


-- 
Samba Shared Repository



More information about the samba-cvs mailing list