[SCM] Samba Shared Repository - branch master updated

Amitay Isaacs amitay at samba.org
Fri May 22 08:06:02 UTC 2020


The branch, master has been updated
       via  1e55591bc54 ctdb-tests: Add a new fetch ring test that also checks hot keys
       via  fb38252677d ctdb-tests: Update fetch_ring to take database and key on command line
       via  53b73b9b0f9 ctdb-daemon: Fix sorting of hot keys
       via  5c8dfbbf9be ctdb-daemon: Add extra logging of hot keys
       via  baf058dcf72 ctdb-daemon: Update hot key logging
       via  1ab39b3270e ctdb-daemon: Fix bug in slot 0 comparison optimisation
       via  f9f60c2a604 ctdb-daemon: Switch some variables to unsigned
       via  21b9844bcbb ctdb-daemon: Add separate hot keys array for database statistics
       via  c28914bfa70 ctdb-build: Fix a typo
      from  c68133d41c3 smbd: prepare DH disconnect for AT-names

https://git.samba.org/?p=samba.git;a=shortlog;h=master


- Log -----------------------------------------------------------------
commit 1e55591bc54949848eeb01a8be4378a18bc9e6c3
Author: Martin Schwenke <martin at meltin.net>
Date:   Mon May 4 17:15:19 2020 +1000

    ctdb-tests: Add a new fetch ring test that also checks hot keys
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>
    
    Autobuild-User(master): Amitay Isaacs <amitay at samba.org>
    Autobuild-Date(master): Fri May 22 08:05:54 UTC 2020 on sn-devel-184

commit fb38252677d0dab299f3fab051eb031167b87242
Author: Martin Schwenke <martin at meltin.net>
Date:   Mon May 4 16:58:38 2020 +1000

    ctdb-tests: Update fetch_ring to take database and key on command line
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit 53b73b9b0f99c9683fca043f4fee74be6a488c66
Author: Martin Schwenke <martin at meltin.net>
Date:   Thu Apr 23 18:59:47 2020 +1000

    ctdb-daemon: Fix sorting of hot keys
    
    The current code only ever swaps with slot 0.  This will only ever
    happen with slots 0 and 1, so probably never sorts.
    
    Replace with qsort().
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit 5c8dfbbf9bea05c35fe1ce454ee47dbee5172722
Author: Martin Schwenke <martin at meltin.net>
Date:   Fri May 1 16:44:22 2020 +1000

    ctdb-daemon: Add extra logging of hot keys
    
    ctdbd currently only logs when a new hot key is added.  If a key gets
    hotter then nothing new is logged.
    
    Log hot key updates when the number of migrations has doubled since
    the last time that key was logged.
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit baf058dcf72e4072ecc2f4bf775f04c271a6e10b
Author: Martin Schwenke <martin at meltin.net>
Date:   Fri May 1 16:24:27 2020 +1000

    ctdb-daemon: Update hot key logging
    
    This message indicates that a hot key was added, so say that.  After
    all the hot key slots have been filled the id will always be 0, so
    don't bother logging it.
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit 1ab39b3270ea34a1953e79d63f3f6d9e04983898
Author: Martin Schwenke <martin at meltin.net>
Date:   Thu May 14 20:25:22 2020 +1000

    ctdb-daemon: Fix bug in slot 0 comparison optimisation
    
    This is only valid if all slots are in use.
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit f9f60c2a60464e8773663848572ec9ec08be0ee5
Author: Martin Schwenke <martin at meltin.net>
Date:   Thu Apr 23 18:59:24 2020 +1000

    ctdb-daemon: Switch some variables to unsigned
    
    These should be unsigned but luck is currently on our side.
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit 21b9844bcbb4d8c18d82df063d6c143b435cdfb7
Author: Martin Schwenke <martin at meltin.net>
Date:   Thu Apr 23 18:51:40 2020 +1000

    ctdb-daemon: Add separate hot keys array for database statistics
    
    There are 2 reasons for this.  Sorting of hot keys is broken and will
    be changed to an implementation that needs a named (i.e. not
    anonymous) structure.  Also, at least one non-protocol field will be
    added to facilitate more useful logging.
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

commit c28914bfa7020227066387b637e0e90b8590510e
Author: Martin Schwenke <martin at meltin.net>
Date:   Thu May 21 20:19:17 2020 +1000

    ctdb-build: Fix a typo
    
    Signed-off-by: Martin Schwenke <martin at meltin.net>
    Reviewed-by: Amitay Isaacs <amitay at gmail.com>

-----------------------------------------------------------------------

Summary of changes:
 ctdb/include/ctdb_private.h                        |   7 +
 ctdb/server/ctdb_call.c                            |  83 +++++++----
 ctdb/server/ctdb_ltdb_server.c                     |  22 ++-
 ctdb/tests/INTEGRATION/database/fetch.001.ring.sh  |   3 +-
 .../INTEGRATION/database/fetch.002.ring-hotkeys.sh | 161 +++++++++++++++++++++
 ctdb/tests/src/fetch_ring.c                        |  30 ++--
 ctdb/wscript                                       |   2 +-
 7 files changed, 259 insertions(+), 49 deletions(-)
 create mode 100755 ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh


Changeset truncated at 500 lines:

diff --git a/ctdb/include/ctdb_private.h b/ctdb/include/ctdb_private.h
index e532323bb34..9ca87332d61 100644
--- a/ctdb/include/ctdb_private.h
+++ b/ctdb/include/ctdb_private.h
@@ -342,6 +342,12 @@ struct ctdb_context {
 	struct lock_context *lock_pending;
 };
 
+struct ctdb_db_hot_key {
+	uint32_t count;
+	TDB_DATA key;
+	uint32_t last_logged_count;
+};
+
 struct ctdb_db_context {
 	struct ctdb_db_context *next, *prev;
 	struct ctdb_context *ctdb;
@@ -375,6 +381,7 @@ struct ctdb_db_context {
 	struct trbt_tree *defer_dmaster;
 
 	struct ctdb_db_statistics_old statistics;
+	struct ctdb_db_hot_key hot_keys[MAX_HOT_KEYS];
 
 	struct lock_context *lock_current;
 	struct lock_context *lock_pending;
diff --git a/ctdb/server/ctdb_call.c b/ctdb/server/ctdb_call.c
index f8bf2d8d41e..14baa797bd6 100644
--- a/ctdb/server/ctdb_call.c
+++ b/ctdb/server/ctdb_call.c
@@ -820,31 +820,62 @@ ctdb_defer_pinned_down_request(struct ctdb_context *ctdb, struct ctdb_db_context
 	return 0;
 }
 
+static int hot_key_cmp(const void *a, const void *b)
+{
+	const struct ctdb_db_hot_key *ka = (const struct ctdb_db_hot_key *)a;
+	const struct ctdb_db_hot_key *kb = (const struct ctdb_db_hot_key *)b;
+
+	if (ka->count < kb->count) {
+		return -1;
+	}
+	if (ka->count > kb->count) {
+		return 1;
+	}
+
+	return 0;
+}
+
 static void
 ctdb_update_db_stat_hot_keys(struct ctdb_db_context *ctdb_db, TDB_DATA key,
 			     unsigned int count)
 {
-	int i, id;
+	unsigned int i, id;
 	char *keystr;
 
-	/* smallest value is always at index 0 */
-	if (count <= ctdb_db->statistics.hot_keys[0].count) {
+	/*
+	 * If all slots are being used then only need to compare
+	 * against the count in the 0th slot, since it contains the
+	 * smallest count.
+	 */
+	if (ctdb_db->statistics.num_hot_keys == MAX_HOT_KEYS &&
+	    count <= ctdb_db->hot_keys[0].count) {
 		return;
 	}
 
 	/* see if we already know this key */
 	for (i = 0; i < MAX_HOT_KEYS; i++) {
-		if (key.dsize != ctdb_db->statistics.hot_keys[i].key.dsize) {
+		if (key.dsize != ctdb_db->hot_keys[i].key.dsize) {
 			continue;
 		}
-		if (memcmp(key.dptr, ctdb_db->statistics.hot_keys[i].key.dptr, key.dsize)) {
+		if (memcmp(key.dptr, ctdb_db->hot_keys[i].key.dptr, key.dsize)) {
 			continue;
 		}
 		/* found an entry for this key */
-		if (count <= ctdb_db->statistics.hot_keys[i].count) {
+		if (count <= ctdb_db->hot_keys[i].count) {
 			return;
 		}
-		ctdb_db->statistics.hot_keys[i].count = count;
+		if (count >= (2 * ctdb_db->hot_keys[i].last_logged_count)) {
+			keystr = hex_encode_talloc(ctdb_db,
+						   (unsigned char *)key.dptr,
+						   key.dsize);
+			D_NOTICE("Updated hot key database=%s key=%s count=%d\n",
+				 ctdb_db->db_name,
+				 keystr ? keystr : "" ,
+				 count);
+			TALLOC_FREE(keystr);
+			ctdb_db->hot_keys[i].last_logged_count = count;
+		}
+		ctdb_db->hot_keys[i].count = count;
 		goto sort_keys;
 	}
 
@@ -855,35 +886,29 @@ ctdb_update_db_stat_hot_keys(struct ctdb_db_context *ctdb_db, TDB_DATA key,
 		id = 0;
 	}
 
-	if (ctdb_db->statistics.hot_keys[id].key.dptr != NULL) {
-		talloc_free(ctdb_db->statistics.hot_keys[id].key.dptr);
+	if (ctdb_db->hot_keys[id].key.dptr != NULL) {
+		talloc_free(ctdb_db->hot_keys[id].key.dptr);
 	}
-	ctdb_db->statistics.hot_keys[id].key.dsize = key.dsize;
-	ctdb_db->statistics.hot_keys[id].key.dptr  = talloc_memdup(ctdb_db, key.dptr, key.dsize);
-	ctdb_db->statistics.hot_keys[id].count = count;
+	ctdb_db->hot_keys[id].key.dsize = key.dsize;
+	ctdb_db->hot_keys[id].key.dptr = talloc_memdup(ctdb_db,
+						       key.dptr,
+						       key.dsize);
+	ctdb_db->hot_keys[id].count = count;
 
 	keystr = hex_encode_talloc(ctdb_db,
 				   (unsigned char *)key.dptr, key.dsize);
-	DEBUG(DEBUG_NOTICE,("Updated hot key database=%s key=%s id=%d "
-			    "count=%d\n", ctdb_db->db_name,
-			    keystr ? keystr : "" , id, count));
+	D_NOTICE("Added hot key database=%s key=%s count=%d\n",
+		 ctdb_db->db_name,
+		 keystr ? keystr : "" ,
+		 count);
 	talloc_free(keystr);
+	ctdb_db->hot_keys[id].last_logged_count = count;
 
 sort_keys:
-	for (i = 1; i < MAX_HOT_KEYS; i++) {
-		if (ctdb_db->statistics.hot_keys[i].count == 0) {
-			continue;
-		}
-		if (ctdb_db->statistics.hot_keys[i].count < ctdb_db->statistics.hot_keys[0].count) {
-			count = ctdb_db->statistics.hot_keys[i].count;
-			ctdb_db->statistics.hot_keys[i].count = ctdb_db->statistics.hot_keys[0].count;
-			ctdb_db->statistics.hot_keys[0].count = count;
-
-			key = ctdb_db->statistics.hot_keys[i].key;
-			ctdb_db->statistics.hot_keys[i].key = ctdb_db->statistics.hot_keys[0].key;
-			ctdb_db->statistics.hot_keys[0].key = key;
-		}
-	}
+	qsort(&ctdb_db->hot_keys[0],
+	      ctdb_db->statistics.num_hot_keys,
+	      sizeof(struct ctdb_db_hot_key),
+	      hot_key_cmp);
 }
 
 /*
diff --git a/ctdb/server/ctdb_ltdb_server.c b/ctdb/server/ctdb_ltdb_server.c
index ce3569fe7b1..b6c7630bea3 100644
--- a/ctdb/server/ctdb_ltdb_server.c
+++ b/ctdb/server/ctdb_ltdb_server.c
@@ -1597,13 +1597,15 @@ int ctdb_set_db_sticky(struct ctdb_context *ctdb, struct ctdb_db_context *ctdb_d
 
 void ctdb_db_statistics_reset(struct ctdb_db_context *ctdb_db)
 {
-	struct ctdb_db_statistics_old *s = &ctdb_db->statistics;
-	int i;
+	unsigned int i;
 
 	for (i=0; i<MAX_HOT_KEYS; i++) {
-		if (s->hot_keys[i].key.dsize > 0) {
-			talloc_free(s->hot_keys[i].key.dptr);
+		if (ctdb_db->hot_keys[i].key.dsize > 0) {
+			TALLOC_FREE(ctdb_db->hot_keys[i].key.dptr);
+			ctdb_db->hot_keys[i].key.dsize = 0;
 		}
+		ctdb_db->hot_keys[i].count = 0;
+		ctdb_db->hot_keys[i].last_logged_count = 0;
 	}
 
 	ZERO_STRUCT(ctdb_db->statistics);
@@ -1615,8 +1617,8 @@ int32_t ctdb_control_get_db_statistics(struct ctdb_context *ctdb,
 {
 	struct ctdb_db_context *ctdb_db;
 	struct ctdb_db_statistics_old *stats;
-	int i;
-	int len;
+	unsigned int i;
+	size_t len;
 	char *ptr;
 
 	ctdb_db = find_ctdb_db(ctdb, db_id);
@@ -1627,7 +1629,13 @@ int32_t ctdb_control_get_db_statistics(struct ctdb_context *ctdb,
 
 	len = offsetof(struct ctdb_db_statistics_old, hot_keys_wire);
 	for (i = 0; i < MAX_HOT_KEYS; i++) {
-		len += ctdb_db->statistics.hot_keys[i].key.dsize;
+		struct ctdb_db_statistics_old *s = &ctdb_db->statistics;
+
+		s->hot_keys[i].key.dsize = ctdb_db->hot_keys[i].key.dsize;
+		s->hot_keys[i].key.dptr = ctdb_db->hot_keys[i].key.dptr;
+		s->hot_keys[i].count = ctdb_db->hot_keys[i].count;
+
+		len += s->hot_keys[i].key.dsize;
 	}
 
 	stats = talloc_size(outdata, len);
diff --git a/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh b/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
index 05632024cb5..33df07b7d88 100755
--- a/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
+++ b/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
@@ -23,7 +23,8 @@ try_command_on_node 0 "$CTDB listnodes | wc -l"
 num_nodes="$out"
 
 echo "Running fetch_ring on all $num_nodes nodes."
-try_command_on_node -v -p all $CTDB_TEST_WRAPPER $VALGRIND fetch_ring -n $num_nodes
+testprog_onnode -v -p all \
+		fetch_ring -n "$num_nodes" -D "fetch_ring.tdb" -k "testkey"
 
 pat='^(Waiting for cluster|Fetch\[[[:digit:]]+\]: [[:digit:]]+(\.[[:digit:]]+)? msgs/sec)$'
 sanity_check_output 1 "$pat"
diff --git a/ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh b/ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh
new file mode 100755
index 00000000000..c4e0023e11c
--- /dev/null
+++ b/ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Run the fetch_ring test, sanity check the output and check hot keys
+# statistics
+
+. "${TEST_SCRIPTS_DIR}/integration.bash"
+
+set -e
+
+ctdb_test_init
+
+testdb="fetch_ring.tdb"
+
+ctdb_get_all_pnns
+# $all_pnns is set above
+# shellcheck disable=SC2154
+num_nodes=$(echo "$all_pnns" | wc -w | tr -d '[:space:]')
+first=$(echo "$all_pnns" | sed -n -e '1p')
+
+get_key ()
+{
+	_n="$1"
+
+	echo "testkey${_n}"
+}
+
+run_fetch_ring ()
+{
+	_timelimit="$1"
+	_key_num="$2"
+
+	_key=$(get_key "$_key_num")
+	_base_cmd="fetch_ring -n ${num_nodes} -D ${testdb}"
+	_cmd="${_base_cmd} -t ${_timelimit} -k ${_key}"
+	echo "Running \"${_cmd}\" on all $num_nodes nodes."
+	testprog_onnode -v -p all "$_cmd"
+
+	_pat='^(Waiting for cluster|Fetch\[[[:digit:]]+\]: [[:digit:]]+(\.[[:digit:]]+)? msgs/sec)$'
+	sanity_check_output 1 "$_pat"
+
+	# Get the last line of output.
+	# $outfile is set above by testprog_onnode()
+	# shellcheck disable=SC2154
+	_last=$(tail -n 1 "$outfile")
+
+	# $last should look like this:
+	#    Fetch[1]: 10670.93 msgs/sec
+	_stuff="${_last##*Fetch\[*\]: }"
+	_mps="${_stuff% msgs/sec*}"
+
+	if [ "${_mps%.*}" -ge 10 ] ; then
+		echo "OK: ${_mps} msgs/sec >= 10 msgs/sec"
+	else
+		ctdb_test_fail "BAD: ${_mps} msgs/sec < 10 msgs/sec"
+	fi
+}
+
+check_hot_keys ()
+{
+	_pnn="$1"
+	_first_key="$2"
+	_num_keys="$3"
+
+	echo
+	echo "Checking hot keys on node ${_pnn}"
+
+	ctdb_onnode "$_pnn" dbstatistics "$testdb"
+
+	# Get hot keys with a non-empty key
+	_hotkeys=$(grep -Ex '[[:space:]]+Count:[[:digit:]]+ Key:[[:xdigit:]]+' \
+			"$outfile") || true
+
+	# Check that there are the right number of non-empty slots
+	if [ -z "$_hotkeys" ] ; then
+		_num=0
+	else
+		_num=$(echo "$_hotkeys" | wc -l | tr -d '[:space:]')
+	fi
+	_msg="hot key slots in use = ${_num}"
+	if [ "$_num_keys" -ne "$_num" ] ; then
+		echo
+		cat "$outfile"
+		ctdb_test_fail "BAD: ${_msg} (expected ${_num_keys})"
+	fi
+	echo "GOOD: ${_msg}"
+
+	# No hot keys?  Done...
+	if [ "$_num" = 0 ] ; then
+		return
+	fi
+
+	# Check that hot key counts are correctly sorted
+	#
+	# Try to be as POSIX as possible
+	# shellcheck disable=SC2001
+	_counts=$(echo "$_hotkeys" | \
+			  sed -e 's|.*Count:\([[:digit:]][[:digit:]]*\).*|\1|')
+	_counts_sorted=$(echo "$_counts" | sort -n)
+	if [ "$_counts" != "$_counts_sorted" ] ; then
+		echo
+		cat "$outfile"
+		ctdb_test_fail "BAD: hot keys not sorted"
+	fi
+	echo "GOOD: hot key counts are correctly sorted"
+
+	# Check that all keys are considered hot
+	for _j in $(seq "$_first_key" $((_first_key + _num_keys - 1))) ; do
+		_key=$(get_key "$_j")
+		_key_hex=$(printf '%s' "$_key" | \
+				   od -A n -t x1 | \
+				   tr -d '[:space:]')
+		if ! echo "$_hotkeys" | grep -q "Key:${_key_hex}\$" ; then
+			echo
+			cat "$outfile"
+			ctdb_test_fail "BAD: key \"${_key}\" is not a hot key"
+		fi
+	done
+	echo "GOOD: all keys are listed as hot keys"
+}
+
+# Run fetch_ring for each of 10 keys.  After each run confirm that all
+# keys used so far are considered hot keys (and do other hot key
+# sanity checks) on all nodes.
+for i in $(seq 1 10) ; do
+	run_fetch_ring 5 "$i"
+
+	for pnn in $all_pnns ; do
+		check_hot_keys "$pnn" 1 "$i"
+	done
+
+	echo
+done
+
+echo
+echo "Resetting statistics on node ${first}"
+ctdb_onnode "$first" statisticsreset
+
+# Ensure that only node $first has had statistics reset
+for pnn in $all_pnns ; do
+	if [ "$pnn" = "$first" ] ; then
+		check_hot_keys "$pnn" 1 0
+	else
+		check_hot_keys "$pnn" 1 10
+	fi
+done
+
+echo
+
+# Run fetch_ring for each of 3 new keys.  After each run confirm that
+# the new keys used so far are considered hot keys (and do other hot
+# key sanity checks) on node $first.
+#
+# Note that nothing can be said about hot keys on other nodes, since
+# they may be an arbitrary blend of old and new keys.
+for i in $(seq 1 3) ; do
+	run_fetch_ring 5 $((100 + i))
+
+	check_hot_keys 0 101 "$i"
+
+	echo
+done
diff --git a/ctdb/tests/src/fetch_ring.c b/ctdb/tests/src/fetch_ring.c
index a8d103ed3f2..f1786ef370b 100644
--- a/ctdb/tests/src/fetch_ring.c
+++ b/ctdb/tests/src/fetch_ring.c
@@ -28,9 +28,6 @@
 #include "tests/src/test_options.h"
 #include "tests/src/cluster_wait.h"
 
-#define TESTDB	"fetch_ring.tdb"
-#define TESTKEY	"testkey"
-
 #define MSG_ID_FETCH	0
 
 static uint32_t next_node(struct ctdb_client_context *client, uint32_t num_nodes)
@@ -63,7 +60,9 @@ static struct tevent_req *fetch_ring_send(TALLOC_CTX *mem_ctx,
 					  struct tevent_context *ev,
 					  struct ctdb_client_context *client,
 					  struct ctdb_db_context *ctdb_db,
-					  uint32_t num_nodes, int timelimit,
+					  const char *keystr,
+					  uint32_t num_nodes,
+					  int timelimit,
 					  int interactive)
 {
 	struct tevent_req *req, *subreq;
@@ -80,8 +79,8 @@ static struct tevent_req *fetch_ring_send(TALLOC_CTX *mem_ctx,
 	state->num_nodes = num_nodes;
 	state->timelimit = timelimit;
 	state->interactive = interactive;
-	state->key.dptr = discard_const(TESTKEY);
-	state->key.dsize = strlen(TESTKEY);
+	state->key.dptr = discard_const(keystr);
+	state->key.dsize = strlen(keystr);
 
 	subreq = ctdb_client_set_message_handler_send(
 					state, ev, client, MSG_ID_FETCH,
@@ -334,7 +333,7 @@ int main(int argc, const char *argv[])
 
 	setup_logging("fetch_ring", DEBUG_STDERR);
 
-	status = process_options_basic(argc, argv, &opts);
+	status = process_options_database(argc, argv, &opts);
 	if (! status) {
 		exit(1);
 	}
@@ -362,15 +361,24 @@ int main(int argc, const char *argv[])
 		exit(1);
 	}
 
-	ret = ctdb_attach(ev, client, tevent_timeval_zero(), TESTDB, 0,
+	ret = ctdb_attach(ev,
+			  client,
+			  tevent_timeval_zero(),
+			  opts->dbname,
+			  0,
 			  &ctdb_db);
 	if (ret != 0) {
-		fprintf(stderr, "Failed to attach to DB %s\n", TESTDB);
+		fprintf(stderr, "Failed to attach to DB %s\n", opts->dbname);
 		exit(1);
 	}
 
-	req = fetch_ring_send(mem_ctx, ev, client, ctdb_db,
-			      opts->num_nodes, opts->timelimit,
+	req = fetch_ring_send(mem_ctx,
+			      ev,
+			      client,
+			      ctdb_db,
+			      opts->keystr,
+			      opts->num_nodes,
+			      opts->timelimit,
 			      opts->interactive);
 	if (req == NULL) {
 		fprintf(stderr, "Memory allocation error\n");
diff --git a/ctdb/wscript b/ctdb/wscript
index 2273853c74f..5f93f3a0668 100644
--- a/ctdb/wscript
+++ b/ctdb/wscript
@@ -33,7 +33,7 @@ samba_dist.DIST_DIRS('''ctdb:. lib/replace:lib/replace lib/talloc:lib/talloc
                         lib/util:lib/util lib/tdb_wrap:lib/tdb_wrap
                         lib/ccan:lib/ccan libcli/util:libcli/util
                         lib/async_req:lib/async_req
-                        lib/pthreadpool:lib/pthreadpoool
+                        lib/pthreadpool:lib/pthreadpool
                         lib/messaging:lib/messaging
                         buildtools:buildtools third_party/waf:third_party/waf''')
 


-- 
Samba Shared Repository



More information about the samba-cvs mailing list