[SCM] Samba Shared Repository - branch master updated
Amitay Isaacs
amitay at samba.org
Mon Nov 17 22:38:02 MST 2014
The branch, master has been updated
via 7f377cf ctdb-scripts: Fix stack dumping when debugging hung scripts
via 4cd5be8 ctdb-scripts: Try to restart statd after every 10 failures
via f51672f ctdb-scripts: Add rpc.statd stack dumping to Ganesha restart
via 968401c ctdb-scripts: Dump stack traces for hung mountd, rquotad, statd processes
via 1f49e1a ctdb-scripts: Add optional program name argument to nfs_dump_some_threads()
via 2ebc305 ctdb-scripts: Factor out new function program_stack_traces()
via 0062a2f s4:dns_server: allocate substructures of struct dns_request_state on the correct TALLOC_CTX
via 7e301c7 s4:dns_server: add some const to dns_server_process_update/dns_update_allowed arguments
from a62cc2c samba: pass down size_t instead of int to add_string_to_array().
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 7f377cf26ecec10cd77f28c1993c48337279892d
Author: Martin Schwenke <martin at meltin.net>
Date: Mon Nov 17 14:15:14 2014 +1100
ctdb-scripts: Fix stack dumping when debugging hung scripts
There are parentheses missing that stop the default pattern from
matching commands with trailing garbage (e.g. "exportfs.orig").
A careful check of POSIX (and running GNU sed with --posix) suggests
that "\|" isn't a supported way of specifying alternation in a regular
expression. Therefore, it is clearer to switch to extended regular
expressions so that this has a chance of being portable (even though
the point is to print /proc/<pid>/stack, which only works on Linux).
Signed-off-by: Martin Schwenke <martin at meltin.net>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay at samba.org>
Autobuild-Date(master): Tue Nov 18 06:37:45 CET 2014 on sn-devel-104
commit 4cd5be87daf531cb8a67f31b91cceeaf2c488127
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 14 16:42:01 2014 +1100
ctdb-scripts: Try to restart statd after every 10 failures
Also add and update tests for statd stack dumps. Update the existing
60.ganesha statd test to do more iterations. Duplicate the result as
a new test for 60.nfs.
Signed-off-by: Martin Schwenke <martin at meltin.net>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit f51672f5149110025088ef6d1fc59fe7208d2aae
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 14 16:39:07 2014 +1100
ctdb-scripts: Add rpc.statd stack dumping to Ganesha restart
Signed-off-by: Martin Schwenke <martin at meltin.net>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 968401ccdc217d0addb6235739b84dbb9d23e651
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 14 13:59:16 2014 +1100
ctdb-scripts: Dump stack traces for hung mountd, rquotad, statd processes
Add a corresponding new unit test for statd.
Signed-off-by: Martin Schwenke <martin at meltin.net>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 1f49e1ab5b317812c0ad482404fb224368726846
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 14 13:48:16 2014 +1100
ctdb-scripts: Add optional program name argument to nfs_dump_some_threads()
Signed-off-by: Martin Schwenke <martin at meltin.net>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 2ebc305be64cd59ad8cb4ccb6beb6ec6e66bf07a
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 14 13:31:03 2014 +1100
ctdb-scripts: Factor out new function program_stack_traces()
In the process, fix a bug where an extra trace would be printed.
Signed-off-by: Martin Schwenke <martin at meltin.net>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 0062a2f5fb464b7c0991ccefd2f9d146fca54a0e
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Oct 14 09:34:29 2014 +0200
s4:dns_server: allocate substructures of struct dns_request_state on the correct TALLOC_CTX
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 7e301c76ce0264bfec0545f9e677e1b3d2b62120
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Oct 14 09:30:43 2014 +0200
s4:dns_server: add some const to dns_server_process_update/dns_update_allowed arguments
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
-----------------------------------------------------------------------
Summary of changes:
ctdb/config/debug-hung-script.sh | 9 +++--
ctdb/config/events.d/60.ganesha | 1 +
ctdb/config/functions | 47 ++++++++++++++--------
ctdb/config/nfs-rpc-checks.d/10.statd.check | 1 +
ctdb/doc/ctdbd.conf.5.xml | 8 ++--
ctdb/tests/complex/90_debug_hung_script.sh | 2 +-
ctdb/tests/eventscripts/60.ganesha.monitor.141.sh | 18 ++++++++-
ctdb/tests/eventscripts/60.nfs.monitor.143.sh | 15 +++++++
...anesha.monitor.141.sh => 60.nfs.monitor.144.sh} | 20 ++++++++-
ctdb/tests/eventscripts/scripts/local.sh | 12 +++++-
ctdb/tests/eventscripts/stubs/pidof | 3 ++
source4/dns_server/dns_crypto.c | 4 +-
source4/dns_server/dns_query.c | 2 +-
source4/dns_server/dns_server.c | 1 +
source4/dns_server/dns_server.h | 5 ++-
source4/dns_server/dns_update.c | 6 +--
16 files changed, 118 insertions(+), 36 deletions(-)
create mode 100755 ctdb/tests/eventscripts/60.nfs.monitor.143.sh
copy ctdb/tests/eventscripts/{60.ganesha.monitor.141.sh => 60.nfs.monitor.144.sh} (57%)
Changeset truncated at 500 lines:
diff --git a/ctdb/config/debug-hung-script.sh b/ctdb/config/debug-hung-script.sh
index 34e957c..3f800fc 100755
--- a/ctdb/config/debug-hung-script.sh
+++ b/ctdb/config/debug-hung-script.sh
@@ -1,5 +1,8 @@
#!/bin/sh
+# This script only works on Linux. Please modify (and submit patches)
+# for other operating systems.
+
[ -n "$CTDB_BASE" ] || \
export CTDB_BASE=$(cd -P $(dirname "$0") ; echo "$PWD")
@@ -28,12 +31,12 @@ fi
# Check for processes matching a regular expression and print
# stack staces. This could help confirm that certain processes
# are stuck in certain places such as the cluster filesystem. The
- # regexp should separate items with "\|" and should not contain
+ # regexp must separate items with "|" and must not contain
# parentheses. The default pattern can be replaced for testing.
- default_pat='exportfs\|rpcinfo'
+ default_pat='exportfs|rpcinfo'
pat="${CTDB_DEBUG_HUNG_SCRIPT_STACKPAT:-${default_pat}}"
echo "$out" |
- sed -n "s at .*-\(.*${pat}.*\),\([0-9]*\).*@\2 \1 at p" |
+ sed -r -n "s at .*-(.*(${pat}).*),([0-9]*).*@\3 \1 at p" |
while read pid name ; do
trace=$(cat "/proc/${pid}/stack" 2>/dev/null)
if [ $? -eq 0 ] ; then
diff --git a/ctdb/config/events.d/60.ganesha b/ctdb/config/events.d/60.ganesha
index df0912d..150be1f 100755
--- a/ctdb/config/events.d/60.ganesha
+++ b/ctdb/config/events.d/60.ganesha
@@ -230,6 +230,7 @@ case "$1" in
p="rpc.statd"
which $p >/dev/null 2>/dev/null && \
nfs_check_rpc_service "statd" \
+ % 10 "verbose restart:b unhealthy" \
-ge 6 "verbose unhealthy" \
-eq 4 "verbose restart" \
-eq 2 "restart:b"
diff --git a/ctdb/config/functions b/ctdb/config/functions
index 53a36dc..77c3ffb 100755
--- a/ctdb/config/functions
+++ b/ctdb/config/functions
@@ -211,6 +211,27 @@ get_proc ()
}
######################################################
+# Print up to $_max kernel stack traces for processes named $_program
+program_stack_traces ()
+{
+ _prog="$1"
+ _max="${2:-1}"
+
+ _count=1
+ for _pid in $(pidof "$_prog") ; do
+ [ $_count -le $_max ] || break
+
+ # Do this first to avoid racing with process exit
+ _stack=$(get_proc "${_pid}/stack" 2>/dev/null)
+ if [ -n "$_stack" ] ; then
+ echo "Stack trace for ${_prog}[${_pid}]:"
+ echo "$_stack"
+ _count=$(($_count + 1))
+ fi
+ done
+}
+
+######################################################
# Check that an RPC service is healthy -
# this includes allowing a certain number of failures
# before marking the NFS service unhealthy.
@@ -381,11 +402,13 @@ _nfs_restart_rpc_service ()
mountd)
echo "Trying to restart $_prog_name [${_p}]"
killall -q -9 "$_p"
+ nfs_dump_some_threads "$_p"
$_maybe_background $_p ${MOUNTD_PORT:+-p} $MOUNTD_PORT
;;
rquotad)
echo "Trying to restart $_prog_name [${_p}]"
killall -q -9 "$_p"
+ nfs_dump_some_threads "$_p"
$_maybe_background $_p ${RQUOTAD_PORT:+-p} $RQUOTAD_PORT
;;
lockd)
@@ -395,6 +418,7 @@ _nfs_restart_rpc_service ()
statd)
echo "Trying to restart $_prog_name [${_p}]"
killall -q -9 "$_p"
+ nfs_dump_some_threads "$_p"
$_maybe_background $_p \
${STATD_HOSTNAME:+-n} $STATD_HOSTNAME \
${STATD_PORT:+-p} $STATD_PORT \
@@ -678,7 +702,9 @@ startstop_ganesha()
service "$_service_name" stop
;;
restart)
- service "$_service_name" restart
+ service "$_service_name" stop
+ nfs_dump_some_threads "rpc.statd"
+ service "$_service_name" start
;;
esac
}
@@ -745,23 +771,12 @@ startstop_nfs() {
# Dump up to the configured number of nfsd thread backtraces.
nfs_dump_some_threads ()
{
- [ -n "$CTDB_NFS_DUMP_STUCK_THREADS" ] || CTDB_NFS_DUMP_STUCK_THREADS=5
+ _prog="${1:-nfsd}"
- # Optimisation to avoid running an unnecessary pidof
- [ $CTDB_NFS_DUMP_STUCK_THREADS -gt 0 ] || return 0
+ _num="${CTDB_NFS_DUMP_STUCK_THREADS:-5}"
+ [ $_num -gt 0 ] || return 0
- _count=0
- for _pid in $(pidof nfsd) ; do
- [ $_count -le $CTDB_NFS_DUMP_STUCK_THREADS ] || break
-
- # Do this first to avoid racing with thread exit
- _stack=$(get_proc "${_pid}/stack" 2>/dev/null)
- if [ -n "$_stack" ] ; then
- echo "Stack trace for stuck nfsd thread [${_pid}]:"
- echo "$_stack"
- _count=$(($_count + 1))
- fi
- done
+ program_stack_traces "$_prog" $_num
}
########################################################
diff --git a/ctdb/config/nfs-rpc-checks.d/10.statd.check b/ctdb/config/nfs-rpc-checks.d/10.statd.check
index d738a32..526e238 100644
--- a/ctdb/config/nfs-rpc-checks.d/10.statd.check
+++ b/ctdb/config/nfs-rpc-checks.d/10.statd.check
@@ -1,3 +1,4 @@
+% 10 verbose restart:b unhealthy
-ge 6 verbose unhealthy
-eq 4 verbose restart
-eq 2 restart:b
diff --git a/ctdb/doc/ctdbd.conf.5.xml b/ctdb/doc/ctdbd.conf.5.xml
index cb48d72..cf71a35 100644
--- a/ctdb/doc/ctdbd.conf.5.xml
+++ b/ctdb/doc/ctdbd.conf.5.xml
@@ -1521,11 +1521,13 @@ CTDB_SET_MonitorInterval=20
<para>
REGEXP specifies interesting processes for which stack
traces should be logged when debugging hung eventscripts
- and those processes are matched in pstree output. See
- also <citetitle>CTDB_DEBUG_HUNG_SCRIPT</citetitle>.
+ and those processes are matched in pstree output. REGEXP
+ is an extended regexp so choices are separated by pipes
+ ('|'). However, REGEXP should not contain parentheses.
+ See also <citetitle>CTDB_DEBUG_HUNG_SCRIPT</citetitle>.
</para>
<para>
- Default is "exportfs\|rpcinfo".
+ Default is "exportfs|rpcinfo".
</para>
</listitem>
</varlistentry>
diff --git a/ctdb/tests/complex/90_debug_hung_script.sh b/ctdb/tests/complex/90_debug_hung_script.sh
index 5438449..09718c4 100755
--- a/ctdb/tests/complex/90_debug_hung_script.sh
+++ b/ctdb/tests/complex/90_debug_hung_script.sh
@@ -58,7 +58,7 @@ ctdb_test_exit_hook_add "onnode $test_node rm -f $debug_output"
try_command_on_node -i $test_node tee "$rc_local_f" <<<"\
CTDB_RUN_TIMEOUT_MONITOR=yes
CTDB_DEBUG_HUNG_SCRIPT_LOGFILE=\"$debug_output\"
-CTDB_DEBUG_HUNG_SCRIPT_STACKPAT='exportfs\|rpcinfo\|sleep'"
+CTDB_DEBUG_HUNG_SCRIPT_STACKPAT='exportfs|rpcinfo|sleep'"
try_command_on_node $test_node chmod +x "$rc_local_f"
diff --git a/ctdb/tests/eventscripts/60.ganesha.monitor.141.sh b/ctdb/tests/eventscripts/60.ganesha.monitor.141.sh
index 9cd82f8..c9a5ab7 100755
--- a/ctdb/tests/eventscripts/60.ganesha.monitor.141.sh
+++ b/ctdb/tests/eventscripts/60.ganesha.monitor.141.sh
@@ -2,7 +2,7 @@
. "${TEST_SCRIPTS_DIR}/unit.sh"
-define_test "statd down, 6 iterations"
+define_test "statd down, 10 iterations"
# statd fails and attempts to restart it fail.
@@ -37,3 +37,19 @@ rpcinfo: RPC: Program not registered
program status version 1 is not available
EOF
simple_test || exit $?
+simple_test || exit $?
+simple_test || exit $?
+simple_test || exit $?
+
+CTDB_NFS_DUMP_STUCK_THREADS=3
+FAKE_RPC_THREAD_PIDS=1234
+
+required_result 1 <<EOF
+ERROR: status failed RPC check:
+rpcinfo: RPC: Program not registered
+program status version 1 is not available
+Trying to restart statd [rpc.statd]
+Stack trace for rpc.statd[1234]:
+[<ffffffff87654321>] fake_stack_trace_for_pid_1234/stack+0x0/0xff
+EOF
+simple_test || exit $?
diff --git a/ctdb/tests/eventscripts/60.nfs.monitor.143.sh b/ctdb/tests/eventscripts/60.nfs.monitor.143.sh
new file mode 100755
index 0000000..732de9c
--- /dev/null
+++ b/ctdb/tests/eventscripts/60.nfs.monitor.143.sh
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+. "${TEST_SCRIPTS_DIR}/unit.sh"
+
+define_test "statd down, 2 iterations, stuck process"
+
+# statd fails and the first attempt to restart it succeeds.
+
+setup_nfs
+rpc_services_down "status"
+CTDB_NFS_DUMP_STUCK_THREADS=2
+FAKE_RPC_THREAD_PIDS="1001"
+
+iterate_test 2 'ok_null' \
+ 2 'rpc_set_service_failure_response "statd"'
diff --git a/ctdb/tests/eventscripts/60.ganesha.monitor.141.sh b/ctdb/tests/eventscripts/60.nfs.monitor.144.sh
similarity index 57%
copy from ctdb/tests/eventscripts/60.ganesha.monitor.141.sh
copy to ctdb/tests/eventscripts/60.nfs.monitor.144.sh
index 9cd82f8..0a3beb7 100755
--- a/ctdb/tests/eventscripts/60.ganesha.monitor.141.sh
+++ b/ctdb/tests/eventscripts/60.nfs.monitor.144.sh
@@ -2,11 +2,11 @@
. "${TEST_SCRIPTS_DIR}/unit.sh"
-define_test "statd down, 6 iterations"
+define_test "statd down, 10 iterations"
# statd fails and attempts to restart it fail.
-setup_nfs_ganesha
+setup_nfs
rpc_services_down "status"
ok_null
@@ -37,3 +37,19 @@ rpcinfo: RPC: Program not registered
program status version 1 is not available
EOF
simple_test || exit $?
+simple_test || exit $?
+simple_test || exit $?
+simple_test || exit $?
+
+CTDB_NFS_DUMP_STUCK_THREADS=3
+FAKE_RPC_THREAD_PIDS=1234
+
+required_result 1 <<EOF
+ERROR: status failed RPC check:
+rpcinfo: RPC: Program not registered
+program status version 1 is not available
+Trying to restart statd [rpc.statd]
+Stack trace for rpc.statd[1234]:
+[<ffffffff87654321>] fake_stack_trace_for_pid_1234/stack+0x0/0xff
+EOF
+simple_test || exit $?
diff --git a/ctdb/tests/eventscripts/scripts/local.sh b/ctdb/tests/eventscripts/scripts/local.sh
index d14e42e..cea6c4d 100644
--- a/ctdb/tests/eventscripts/scripts/local.sh
+++ b/ctdb/tests/eventscripts/scripts/local.sh
@@ -713,7 +713,7 @@ setup_nfs ()
export CTDB_NFS_SKIP_SHARE_CHECK="no"
export CTDB_MONITOR_NFS_THREAD_COUNT RPCNFSDCOUNT FAKE_NFSD_THREAD_PIDS
- export CTDB_NFS_DUMP_STUCK_THREADS
+ export CTDB_NFS_DUMP_STUCK_THREADS FAKE_RPC_THREAD_PIDS
# Reset the failcounts for nfs services.
eventscript_call eval rm -f '$ctdb_fail_dir/nfs_*'
@@ -861,7 +861,7 @@ Trying to restart NFS service"
for _pid in $FAKE_NFSD_THREAD_PIDS ; do
_t="\
$_t
-${_bg}Stack trace for stuck nfsd thread [${_pid}]:
+${_bg}Stack trace for nfsd[${_pid}]:
${_bg}[<ffffffff87654321>] fake_stack_trace_for_pid_${_pid}/stack+0x0/0xff"
done
fi
@@ -878,6 +878,14 @@ ${_bg}Starting nfslock: OK"
;;
*)
_t="Trying to restart $_progname [${_p}]"
+ if [ -n "$CTDB_NFS_DUMP_STUCK_THREADS" ] ; then
+ for _pid in $FAKE_RPC_THREAD_PIDS ; do
+ _t="\
+$_t
+Stack trace for ${_p}[${_pid}]:
+[<ffffffff87654321>] fake_stack_trace_for_pid_${_pid}/stack+0x0/0xff"
+ done
+ fi
esac
_out="${_out}${_out:+${_nl}}${_t}"
;;
diff --git a/ctdb/tests/eventscripts/stubs/pidof b/ctdb/tests/eventscripts/stubs/pidof
index b6ad6d8..b84aef0 100755
--- a/ctdb/tests/eventscripts/stubs/pidof
+++ b/ctdb/tests/eventscripts/stubs/pidof
@@ -4,6 +4,9 @@ case "$1" in
nfsd)
echo "$FAKE_NFSD_THREAD_PIDS"
;;
+ rpc.statd|rpc.rquotad|rpc.mountd)
+ echo "$FAKE_RPC_THREAD_PIDS"
+ ;;
*)
echo "pidof: \"$1\" not implemented"
exit 1
diff --git a/source4/dns_server/dns_crypto.c b/source4/dns_server/dns_crypto.c
index aba906d..3f199de 100644
--- a/source4/dns_server/dns_crypto.c
+++ b/source4/dns_server/dns_crypto.c
@@ -131,7 +131,7 @@ WERROR dns_verify_tsig(struct dns_server *dns,
/* We got a TSIG, so we need to sign our reply */
state->sign = true;
- state->tsig = talloc_zero(mem_ctx, struct dns_res_rec);
+ state->tsig = talloc_zero(state->mem_ctx, struct dns_res_rec);
if (state->tsig == NULL) {
return WERR_NOMEM;
}
@@ -226,7 +226,7 @@ WERROR dns_verify_tsig(struct dns_server *dns,
}
state->authenticated = true;
- state->key_name = talloc_strdup(mem_ctx, tkey->name);
+ state->key_name = talloc_strdup(state->mem_ctx, tkey->name);
if (state->key_name == NULL) {
return WERR_NOMEM;
}
diff --git a/source4/dns_server/dns_query.c b/source4/dns_server/dns_query.c
index 16e14c1..4e3c6cc 100644
--- a/source4/dns_server/dns_query.c
+++ b/source4/dns_server/dns_query.c
@@ -529,7 +529,7 @@ static WERROR handle_tkey(struct dns_server *dns,
reply.data,
reply.length);
state->sign = true;
- state->key_name = talloc_strdup(mem_ctx, tkey->name);
+ state->key_name = talloc_strdup(state->mem_ctx, tkey->name);
if (state->key_name == NULL) {
return WERR_NOMEM;
}
diff --git a/source4/dns_server/dns_server.c b/source4/dns_server/dns_server.c
index 7ea70db..f1a4c4c 100644
--- a/source4/dns_server/dns_server.c
+++ b/source4/dns_server/dns_server.c
@@ -125,6 +125,7 @@ static struct tevent_req *dns_process_send(TALLOC_CTX *mem_ctx,
if (req == NULL) {
return NULL;
}
+ state->state.mem_ctx = state;
state->in = in;
state->dns = dns;
diff --git a/source4/dns_server/dns_server.h b/source4/dns_server/dns_server.h
index 12ccc9b..3423ee0 100644
--- a/source4/dns_server/dns_server.h
+++ b/source4/dns_server/dns_server.h
@@ -60,6 +60,7 @@ struct dns_server {
};
struct dns_request_state {
+ TALLOC_CTX *mem_ctx;
uint16_t flags;
bool authenticated;
bool sign;
@@ -79,9 +80,9 @@ WERROR dns_server_process_query_recv(
struct dns_res_rec **additional, uint16_t *arcount);
WERROR dns_server_process_update(struct dns_server *dns,
- struct dns_request_state *state,
+ const struct dns_request_state *state,
TALLOC_CTX *mem_ctx,
- struct dns_name_packet *in,
+ const struct dns_name_packet *in,
struct dns_res_rec **prereqs, uint16_t *prereq_count,
struct dns_res_rec **updates, uint16_t *update_count,
struct dns_res_rec **additional, uint16_t *arcount);
diff --git a/source4/dns_server/dns_update.c b/source4/dns_server/dns_update.c
index b18c3c5..c002b4d 100644
--- a/source4/dns_server/dns_update.c
+++ b/source4/dns_server/dns_update.c
@@ -732,7 +732,7 @@ failed:
}
static WERROR dns_update_allowed(struct dns_server *dns,
- struct dns_request_state *state,
+ const struct dns_request_state *state,
struct dns_server_tkey **tkey)
{
if (lpcfg_allow_dns_updates(dns->task->lp_ctx) == DNS_UPDATE_ON) {
@@ -761,9 +761,9 @@ static WERROR dns_update_allowed(struct dns_server *dns,
WERROR dns_server_process_update(struct dns_server *dns,
- struct dns_request_state *state,
+ const struct dns_request_state *state,
TALLOC_CTX *mem_ctx,
- struct dns_name_packet *in,
+ const struct dns_name_packet *in,
struct dns_res_rec **prereqs, uint16_t *prereq_count,
struct dns_res_rec **updates, uint16_t *update_count,
struct dns_res_rec **additional, uint16_t *arcount)
--
Samba Shared Repository
More information about the samba-cvs
mailing list