[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Wed Aug 7 07:26:03 UTC 2019
The branch, master has been updated
via 2a902020525 charset: add tests for Unicode NFC <-> NFD conversion
via 107020793c7 charset: add support for Unicode normalisation with libicu
via 323f8521475 torture: add torture_assert_errno_equal_goto()
via 39e2f6d59fa Add fuzzing binary for oLschema2ldif
via 404278d9472 Add fuzzing binary for tiniparser
via dd5f8732d8b Add fuzzing support to build system
from aab17124785 smbd: Assert that INTERNAL_OPEN_ONLY never gets real oplocks
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 2a90202052558c945e02675d1331e65aeb15f9fa
Author: Ralph Boehme <slow at samba.org>
Date: Tue Apr 9 13:34:39 2019 +0200
charset: add tests for Unicode NFC <-> NFD conversion
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed Aug 7 07:25:39 UTC 2019 on sn-devel-184
commit 107020793c7ea44e5e776a1401bbf4f8ccb9bd85
Author: Ralph Boehme <slow at samba.org>
Date: Tue Apr 9 11:21:57 2019 +0200
charset: add support for Unicode normalisation with libicu
This adds a direct conversion hook using libicu to perform NFC <-> NFD
conversion on UTF8 strings. The defined charset strings are "UTF8-NFC" and
"UTF8-NFD", to convert from one to the other the caller calls smb_iconv_open()
with the desired source and target charsets, eg
smb_iconv_open("UTF8-NFD", "UTF8-NFC");
for converting from NFC to NFD.
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 323f852147524b950bab4628a2824aa2273cb577
Author: Ralph Boehme <slow at samba.org>
Date: Tue Apr 23 16:47:45 2019 +0200
torture: add torture_assert_errno_equal_goto()
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 39e2f6d59fa2e839c8ef40934790b34dbdfc8f29
Author: Michael Hanselmann <public at hansmi.ch>
Date: Thu Apr 4 01:10:11 2019 +0200
Add fuzzing binary for oLschema2ldif
Use the oLschema2ldif library functions introduced in commit
0c7c44a284a26790081c000f5b8f4ed32f9f21d7 to implement a fuzzing utility.
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
commit 404278d9472c28fedc200d385667a0d2cc1c992d
Author: Michael Hanselmann <public at hansmi.ch>
Date: Thu Apr 4 01:03:58 2019 +0200
Add fuzzing binary for tiniparser
The "tiniparser_load" function is made into a wrapper for the newly
added "tiniparser_load_stream" function which accepts a FILE pointer.
This way no actual files have to be opened for fuzzing (memfd_create(2)
isn't readily available on all systems yet).
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
commit dd5f8732d8b2ac6cd86e5d7dc8a9498d2b951a43
Author: Michael Hanselmann <public at hansmi.ch>
Date: Thu Apr 4 00:23:07 2019 +0200
Add fuzzing support to build system
LibFuzzer, Honggfuzz and other programs implement simple interfaces for
fuzzing appropriately prepared code. Samba contains quite a lot of
parsing code, often a good target for fuzzing.
With this change the build system is amended to support building fuzzing
binaries (added in later changes).
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
-----------------------------------------------------------------------
Summary of changes:
buildtools/wafsamba/samba_autoconf.py | 5 +
buildtools/wafsamba/wscript | 7 +
lib/fuzzing/README.md | 35 ++++
.../sec_ctx1.c => lib/fuzzing/fuzz_oLschema2ldif.c | 47 +++--
.../ndr/ndr_rap.c => lib/fuzzing/fuzz_tiniparser.c | 27 ++-
.../libnet/grouptest.h => lib/fuzzing/fuzzing.c | 7 +-
librpc/ndr/ndr_krb5pac.h => lib/fuzzing/fuzzing.h | 16 +-
lib/fuzzing/wscript_build | 20 ++
lib/torture/torture.h | 12 ++
lib/util/charset/iconv.c | 164 +++++++++++++++
lib/util/charset/tests/convert_string.c | 228 +++++++++++++++++++++
lib/util/charset/wscript_build | 2 +-
lib/util/charset/wscript_configure | 13 ++
lib/util/tiniparser.c | 24 ++-
lib/util/tiniparser.h | 1 +
wscript_build | 1 +
16 files changed, 567 insertions(+), 42 deletions(-)
create mode 100644 lib/fuzzing/README.md
copy testsuite/smbd/sec_ctx1.c => lib/fuzzing/fuzz_oLschema2ldif.c (52%)
copy librpc/ndr/ndr_rap.c => lib/fuzzing/fuzz_tiniparser.c (65%)
copy source4/torture/libnet/grouptest.h => lib/fuzzing/fuzzing.c (84%)
copy librpc/ndr/ndr_krb5pac.h => lib/fuzzing/fuzzing.h (67%)
create mode 100644 lib/fuzzing/wscript_build
Changeset truncated at 500 lines:
diff --git a/buildtools/wafsamba/samba_autoconf.py b/buildtools/wafsamba/samba_autoconf.py
index 352b5ca31d3..ee15e34b5ec 100644
--- a/buildtools/wafsamba/samba_autoconf.py
+++ b/buildtools/wafsamba/samba_autoconf.py
@@ -938,6 +938,11 @@ def SETUP_CONFIGURE_CACHE(conf, enable):
@conf
def SAMBA_CHECK_UNDEFINED_SYMBOL_FLAGS(conf):
+ if Options.options.address_sanitizer or Options.options.enable_libfuzzer:
+ # Sanitizers can rely on symbols undefined at library link time and the
+ # symbols used for fuzzers are only defined by compiler wrappers.
+ return
+
if not sys.platform.startswith("openbsd"):
# we don't want any libraries or modules to rely on runtime
# resolution of symbols
diff --git a/buildtools/wafsamba/wscript b/buildtools/wafsamba/wscript
index e5017a4a02f..ce5e0b48fc1 100644
--- a/buildtools/wafsamba/wscript
+++ b/buildtools/wafsamba/wscript
@@ -129,6 +129,9 @@ def options(opt):
action="store_true",
dest='undefined_sanitizer',
default=False)
+ gr.add_option('--enable-libfuzzer',
+ help=("Build fuzzing binaries (requires compiler options for libFuzzer or compiler wrapper such as honggfuzz/hfuzz-cc)"),
+ action="store_true", dest='enable_libfuzzer', default=False)
gr.add_option('--abi-check',
help=("Check ABI signatures for libraries"),
@@ -590,6 +593,10 @@ struct foo bar = { .y = 'X', .x = 1 };
eprintf("bla", "bar")
''', define='HAVE__VA_ARGS__MACRO')
+ conf.env.enable_libfuzzer = Options.options.enable_libfuzzer
+ if conf.env.enable_libfuzzer:
+ conf.DEFINE('ENABLE_LIBFUZZER', 1)
+
conf.SAMBA_BUILD_ENV()
diff --git a/lib/fuzzing/README.md b/lib/fuzzing/README.md
new file mode 100644
index 00000000000..3848838ba02
--- /dev/null
+++ b/lib/fuzzing/README.md
@@ -0,0 +1,35 @@
+# Fuzzing Samba
+
+Fuzzing supplies valid, invalid, unexpected or random data as input to a piece
+of code. Instrumentation, usually compiler-implemented, is used to monitor for
+exceptions such as crashes, assertions or memory corruption.
+
+See [Wikipedia article on fuzzing](https://en.wikipedia.org/wiki/Fuzzing) for
+more information.
+
+
+## Configure with fuzzing
+
+Example command line to build binaries for use with
+[honggfuzz](https://github.com/google/honggfuzz/):
+
+```sh
+buildtools/bin/waf -C --without-gettext --enable-debug --enable-developer \
+ --address-sanitizer --enable-libfuzzer \
+ CC=.../honggfuzz/hfuzz_cc/hfuzz-clang configure \
+ LINK_CC=.../honggfuzz/hfuzz_cc/hfuzz-clang
+```
+
+
+## Fuzzing tiniparser
+
+Example for fuzzing `tiniparser` using `honggfuzz` (see `--help` for more
+options):
+
+```sh
+buildtools/bin/waf --targets=fuzz_tiniparser build && \
+.../honggfuzz/honggfuzz --sanitizers --timeout 3 --max_file_size 256 \
+ --rlimit_rss 100 -f .../tiniparser-corpus -- bin/fuzz_tiniparser
+```
+
+# vim: set sw=8 sts=8 ts=8 tw=79 :
diff --git a/testsuite/smbd/sec_ctx1.c b/lib/fuzzing/fuzz_oLschema2ldif.c
similarity index 52%
copy from testsuite/smbd/sec_ctx1.c
copy to lib/fuzzing/fuzz_oLschema2ldif.c
index ab85ae16e5b..4dd5668e673 100644
--- a/testsuite/smbd/sec_ctx1.c
+++ b/lib/fuzzing/fuzz_oLschema2ldif.c
@@ -1,39 +1,52 @@
-/*
- Unix SMB/Netbios implementation.
- Version 1.9.
- Security context tests
- Copyright (C) Tim Potter 2000
-
+/*
+ Fuzzing for oLschema2ldif
+ Copyright (C) Michael Hanselmann 2019
+
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.
-
+
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
-
+
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "includes.h"
+#include "fuzzing.h"
+#include "utils/oLschema2ldif/lib.h"
-void exit_server(char *reason) {}
+static FILE *devnull;
-int main (int argc, char **argv)
+int LLVMFuzzerInitialize(int *argc, char ***argv)
{
- /* Become a non-root user */
+ devnull = fopen("/dev/null", "w");
+
+ return 0;
+}
+
+int LLVMFuzzerTestOneInput(uint8_t *buf, size_t len)
+{
+ TALLOC_CTX *mem_ctx;
+ struct conv_options opt;
+
+ mem_ctx = talloc_init(__FUNCTION__);
+
+ opt.in = fmemopen(buf, len, "r");
+ opt.out = devnull;
+ opt.ldb_ctx = ldb_init(mem_ctx, NULL);
+
+ opt.basedn = ldb_dn_new(mem_ctx, opt.ldb_ctx, "");
- samba_setuid(1);
- samba_setgid(1);
+ process_file(mem_ctx, &opt);
- /* Try to push a security context. This should fail with a
- smb_assert() error. */
+ fclose(opt.in);
- push_sec_ctx(2, 2);
- printf("FAIL\n");
+ talloc_free(mem_ctx);
return 0;
}
diff --git a/librpc/ndr/ndr_rap.c b/lib/fuzzing/fuzz_tiniparser.c
similarity index 65%
copy from librpc/ndr/ndr_rap.c
copy to lib/fuzzing/fuzz_tiniparser.c
index ea18a088650..a6e2ef7c2fe 100644
--- a/librpc/ndr/ndr_rap.c
+++ b/lib/fuzzing/fuzz_tiniparser.c
@@ -1,9 +1,6 @@
/*
- Unix SMB/CIFS implementation.
-
- routines for marshalling/unmarshalling special rap types
-
- Copyright (C) Guenther Deschner 2010
+ Fuzzing for trivial smb.conf parsing code.
+ Copyright (C) Michael Hanselmann 2019
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -20,9 +17,23 @@
*/
#include "includes.h"
-#include "librpc/gen_ndr/ndr_rap.h"
+#include "fuzzing.h"
+#include "lib/util/tiniparser.h"
+
+int LLVMFuzzerInitialize(int *argc, char ***argv)
+{
+ return 0;
+}
-_PUBLIC_ void ndr_print_rap_status(struct ndr_print *ndr, const char *name, enum rap_status r)
+int LLVMFuzzerTestOneInput(uint8_t *buf, size_t len)
{
- ndr_print_WERROR(ndr, name, W_ERROR(r));
+ FILE *fp;
+
+ fp = fmemopen(buf, len, "r");
+
+ tiniparser_load_stream(fp);
+
+ fclose(fp);
+
+ return 0;
}
diff --git a/source4/torture/libnet/grouptest.h b/lib/fuzzing/fuzzing.c
similarity index 84%
copy from source4/torture/libnet/grouptest.h
copy to lib/fuzzing/fuzzing.c
index 8b65e6e57aa..f0d7fb4f46b 100644
--- a/source4/torture/libnet/grouptest.h
+++ b/lib/fuzzing/fuzzing.c
@@ -1,7 +1,7 @@
/*
Unix SMB/CIFS implementation.
-
- Copyright (C) Rafal Szczesniak 2007
+ Fuzzing utility functions
+ Copyright (C) Michael Hanselmann 2019
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -17,4 +17,5 @@
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
-#define TEST_GROUPNAME "libnetgrptest"
+#include "includes.h"
+#include "fuzzing/fuzzing.h"
diff --git a/librpc/ndr/ndr_krb5pac.h b/lib/fuzzing/fuzzing.h
similarity index 67%
copy from librpc/ndr/ndr_krb5pac.h
copy to lib/fuzzing/fuzzing.h
index f53916142b9..67e49c3fbe0 100644
--- a/librpc/ndr/ndr_krb5pac.h
+++ b/lib/fuzzing/fuzzing.h
@@ -1,9 +1,7 @@
/*
Unix SMB/CIFS implementation.
-
- routines for marshalling/unmarshalling spoolss subcontext buffer structures
-
- Copyright (C) Stefan Metzmacher 2005
+ Fuzzing utility functions
+ Copyright (C) Michael Hanselmann 2019
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -19,8 +17,14 @@
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
+#ifndef _SAMBA_FUZZING_H
+#define _SAMBA_FUZZING_H
-#include "librpc/gen_ndr/ndr_krb5pac.h"
+#include <stddef.h>
+#include <stdint.h>
-size_t _ndr_size_PAC_INFO(const union PAC_INFO *r, uint32_t level, int flags);
+/* Prototypes for fuzzing interface */
+int LLVMFuzzerInitialize(int *argc, char ***argv);
+int LLVMFuzzerTestOneInput(uint8_t * buf, size_t len);
+#endif /* _SAMBA_FUZZING_H */
diff --git a/lib/fuzzing/wscript_build b/lib/fuzzing/wscript_build
new file mode 100644
index 00000000000..9c73c59c259
--- /dev/null
+++ b/lib/fuzzing/wscript_build
@@ -0,0 +1,20 @@
+#!/usr/bin/env python
+
+bld.SAMBA_SUBSYSTEM('fuzzing',
+ source='fuzzing.c',
+ deps='talloc',
+ enabled=bld.env.enable_libfuzzer,
+ )
+
+bld.SAMBA_BINARY('fuzz_tiniparser',
+ source='fuzz_tiniparser.c',
+ deps='fuzzing tiniparser talloc',
+ install=False,
+ enabled=bld.env.enable_libfuzzer)
+
+bld.SAMBA_BINARY('fuzz_oLschema2ldif',
+ source='fuzz_oLschema2ldif.c',
+ deps='fuzzing oLschema2ldif-lib',
+ install=False,
+ enabled=bld.env.enable_libfuzzer,
+ )
diff --git a/lib/torture/torture.h b/lib/torture/torture.h
index a7f3f471b3a..9f55c164ac1 100644
--- a/lib/torture/torture.h
+++ b/lib/torture/torture.h
@@ -579,6 +579,18 @@ static inline void torture_dump_data_str_cb(const char *buf, void *private_data)
} \
} while(0)
+#define torture_assert_errno_equal_goto(torture_ctx,expected,ret,label,cmt)\
+ do { int __expected = (expected); \
+ if (errno != __expected) { \
+ torture_result(torture_ctx, TORTURE_FAIL, \
+ __location__": errno was %d (%s), expected %d: %s: %s", \
+ errno, strerror(errno), __expected, \
+ strerror(__expected), cmt); \
+ ret = false; \
+ goto label; \
+ } \
+ } while(0)
+
#define torture_assert_guid_equal(torture_ctx,got,expected,cmt)\
do {const struct GUID __got = (got), __expected = (expected); \
if (!GUID_equal(&__got, &__expected)) { \
diff --git a/lib/util/charset/iconv.c b/lib/util/charset/iconv.c
index 4fae09fda52..14a1f8652e3 100644
--- a/lib/util/charset/iconv.c
+++ b/lib/util/charset/iconv.c
@@ -28,6 +28,11 @@
#include "libcli/util/ntstatus.h"
#include "lib/util/util_str_hex.h"
+#ifdef HAVE_ICU_I18N
+#include <unicode/ustring.h>
+#include <unicode/utrans.h>
+#endif
+
#ifdef strcasecmp
#undef strcasecmp
#endif
@@ -165,6 +170,109 @@ static size_t sys_iconv(void *cd,
}
#endif
+#ifdef HAVE_ICU_I18N
+static size_t sys_uconv(void *cd,
+ const char **inbuf,
+ size_t *inbytesleft,
+ char **outbuf,
+ size_t *outbytesleft)
+{
+ UTransliterator *t = (UTransliterator *)cd;
+ size_t bufsize = *inbytesleft * 2;
+ UChar ustr[bufsize];
+ UChar *up = NULL;
+ char *p = NULL;
+ int32_t ustrlen;
+ int32_t limit;
+ int32_t converted_len;
+ size_t inbuf_consumed;
+ size_t outbut_consumed;
+ UErrorCode ue;
+
+ /* Convert from UTF8 to UCS2 */
+ ue = 0;
+ up = u_strFromUTF8(ustr, /* dst */
+ bufsize, /* dst buflen */
+ &converted_len, /* dst written */
+ *inbuf, /* src */
+ *inbytesleft, /* src length */
+ &ue);
+ if (up == NULL || U_FAILURE(ue)) {
+ return -1;
+ }
+ if (converted_len > bufsize) {
+ /*
+ * u_strFromUTF8() returns the required size in
+ * converted_len. In theory this should never overflow as the
+ * ustr[] array is allocated with a size twice as big as
+ * inbytesleft and converted_len should be equal to inbytesleft,
+ * but you never know...
+ */
+ errno = EOVERFLOW;
+ return -1;
+ }
+ inbuf_consumed = converted_len;
+
+ /*
+ * The following transliteration function takes two parameters, the
+ * lenght of the text to be converted (converted_len) and a limit which
+ * may be smaller then converted_len. We just set limit to converted_len
+ * and also ignore the value returned in limit.
+ */
+ limit = converted_len;
+
+ /* Inplace transliteration */
+ utrans_transUChars(t,
+ ustr, /* text */
+ &converted_len, /* text length */
+ bufsize, /* text buflen */
+ 0, /* start */
+ &limit, /* limit */
+ &ue);
+ if (U_FAILURE(ue)) {
+ return -1;
+ }
+ if (converted_len > bufsize) {
+ /*
+ * In theory this should never happen as the ustr[] array is
+ * allocated with a size twice as big as inbytesleft and
+ * converted_len should be equal to inbytesleft, but you never
+ * know...
+ */
+ errno = EOVERFLOW;
+ return -1;
+ }
+ ustrlen = converted_len;
+
+ /* Convert from UCS2 back to UTF8 */
+ ue = 0;
+ p = u_strToUTF8(*outbuf, /* dst */
+ *outbytesleft, /* dst buflen */
+ &converted_len, /* dst required length */
+ ustr, /* src */
+ ustrlen, /* src length */
+ &ue);
+ if (p == NULL || U_FAILURE(ue)) {
+ return -1;
+ }
+
+ outbut_consumed = converted_len;
+ if (converted_len > *outbytesleft) {
+ /*
+ * The caller's result buffer is too small...
+ */
+ outbut_consumed = *outbytesleft;
+ }
+
+ *inbuf += inbuf_consumed;
+ *inbytesleft -= inbuf_consumed;
+ *outbuf += outbut_consumed;
+ *outbytesleft -= outbut_consumed;
+
+ return converted_len;
+}
+#endif
+
/**
* This is a simple portable iconv() implementaion.
*
@@ -228,6 +336,16 @@ static bool is_utf16(const char *name)
static int smb_iconv_t_destructor(smb_iconv_t hwd)
{
+#ifdef HAVE_ICU_I18N
+ /*
+ * This has to come first, as the cd_direct member won't be an iconv
+ * handle and must not be passed to iconv_close().
+ */
+ if (hwd->direct == sys_uconv) {
+ utrans_close(hwd->cd_direct);
+ return 0;
+ }
+#endif
#ifdef HAVE_NATIVE_ICONV
if (hwd->cd_pull != NULL && hwd->cd_pull != (iconv_t)-1)
iconv_close(hwd->cd_pull);
@@ -302,6 +420,52 @@ _PUBLIC_ smb_iconv_t smb_iconv_open_ex(TALLOC_CTX *mem_ctx, const char *tocode,
}
#endif
+#ifdef HAVE_ICU_I18N
+ if (strcasecmp(fromcode, "UTF8-NFD") == 0 &&
+ strcasecmp(tocode, "UTF8-NFC") == 0)
+ {
+ U_STRING_DECL(t, "any-nfc", 7);
+ UErrorCode ue = 0;
+
+ U_STRING_INIT(t, "any-nfc", 7);
+
+ ret->cd_direct = utrans_openU(t,
+ strlen("any-nfc"),
+ UTRANS_FORWARD,
+ NULL,
+ 0,
+ NULL,
+ &ue);
+ if (U_FAILURE(ue)) {
+ return (smb_iconv_t)-1;
+ }
+ ret->direct = sys_uconv;
+ return ret;
+ }
+
+ if (strcasecmp(fromcode, "UTF8-NFC") == 0 &&
+ strcasecmp(tocode, "UTF8-NFD") == 0)
+ {
+ U_STRING_DECL(tname, "any-nfd", 7);
+ UErrorCode ue = 0;
+
+ U_STRING_INIT(tname, "any-nfd", 7);
+
+ ret->cd_direct = utrans_openU(tname,
+ 7,
+ UTRANS_FORWARD,
+ NULL,
+ 0,
+ NULL,
+ &ue);
+ if (U_FAILURE(ue)) {
--
Samba Shared Repository
More information about the samba-cvs
mailing list