[SCM] CTDB repository - branch 1.2.40 updated - ctdb-1.2.54-7-g8b2d844
Amitay Isaacs
amitay at samba.org
Mon Nov 26 21:49:43 MST 2012
The branch, 1.2.40 has been updated
via 8b2d84482bacd3b31db013496ce82c2e7b730e86 (commit)
via e78e37205308e1507a2cf86a655a95893a7cd413 (commit)
via cfb85046e6ed87bf01c3abe8cc908a6d2be741e1 (commit)
via 5c44156d1aea799f1d6655dd0237e01c49027b82 (commit)
via bde1c733fc8c4009202bf185452914f17631c1e7 (commit)
via 6479566a0a104b903f499979db594541ffc00a1f (commit)
via 5205d545e8d8c72d73b9d5fd148df6de30392fc8 (commit)
from b7467294465b6225982c90315df20a8699ccf812 (commit)
http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=1.2.40
- Log -----------------------------------------------------------------
commit 8b2d84482bacd3b31db013496ce82c2e7b730e86
Author: Amitay Isaacs <amitay at gmail.com>
Date: Tue Nov 27 15:50:54 2012 +1100
New version 1.2.55
Signed-off-by: Amitay Isaacs <amitay at gmail.com>
commit e78e37205308e1507a2cf86a655a95893a7cd413
Author: Amitay Isaacs <amitay at gmail.com>
Date: Thu Nov 22 14:37:45 2012 +1100
Revert "when creating/adding a public ip, set the initial interface to be the first interface specified"
This reverts commit 4308935ba48ac7a29e7523315acf580019715f0f.
When IP is added to a node on a new interface for the first time,
vnn->iface gets set to the first interface defined for that IP. This
actually causes problem in ctdb_vnn_assign_iface(). Since vnn->iface
is set it takes an early exit without updating vnn->pnn. This results
in IP being hosted on the node, but CTDB still thinks it's unassigned.
Signed-off-by: Amitay Isaacs <amitay at gmail.com>
commit cfb85046e6ed87bf01c3abe8cc908a6d2be741e1
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 16 20:21:15 2012 +1100
Eventscripts: 10.interface should list configured interfaces
The current code lists available interfaces. If IPs are configured in
some other way than the public addresses file (e.g. ctdb addip) and their
interfaces default to being marked down then, since down interfaces are
not available, these interfaces can never be marked up.
The configured interfaces should be listed instead.
Signed-off-by: Martin Schwenke <martin at meltin.net>
Cherry-pick-from: d8f010355b715e49709836e057a5d0f110919897
Conflicts:
config/events.d/10.interface
commit 5c44156d1aea799f1d6655dd0237e01c49027b82
Author: Martin Schwenke <martin at meltin.net>
Date: Fri Nov 16 19:43:14 2012 +1100
ctdbd: Make the link status of new interfaces more flexible
Neither up nor down is a good default value for the link status of a
new interface. Up means that IPs can be assigned to interfaces before
the true state is known and they can move away quickly if the interface
is actually down. Down means that IPs can't be assigned to an interface
for a variable amount of time - until a monitor cycle occurs - and this
can result in imbalanced IPs.
This is a neat compromise. Before the startup event completes, IPs
can't be assigned to interfaces because all interfaces begin in a down
state. As soon as the startup event completes, IPs can be allocated
to any interface that has been marked up by the eventscript. Later,
during normal operation, newly added IPs can be assigned to new
interfaces immediately. The IPs will still move away if an interface
is noticed to be down in the next monitor cycle, but that is the
exception rather than the rule.
Signed-off-by: Martin Schwenke <martin at meltin.net>
Cherry-pick-from: 9275a69a414482f1053ae14528d5972575b9214e
commit bde1c733fc8c4009202bf185452914f17631c1e7
Author: Amitay Isaacs <amitay at gmail.com>
Date: Tue Nov 6 17:06:54 2012 +1100
tools/ctdb: Do not use function return value as pnn
This fixes the wrong code where same variable 'ret' is used to track the pnn
and the return value of a function call.
Signed-off-by: Amitay Isaacs <amitay at gmail.com>
Cherry-pick-from: 718233c445cd6627ab3962b6565c2655f1f8efd0
commit 6479566a0a104b903f499979db594541ffc00a1f
Author: Amitay Isaacs <amitay at gmail.com>
Date: Tue Oct 23 16:23:12 2012 +1100
recoverd: Track the nodes that fail takeover run and set culprit count
If any of the nodes fail takeover run (either due to timeout or failure
to complete within takeover_timeout interval) from main loop, recovery
master will give up trying takeover run with following message:
"Unable to setup public takeover addresses. Try again later"
And as a side-effect the monitoring is disabled on all the nodes. Before
ctdb_takeover_run() is called from main loop, monitoring get disabled via
startrecovery event. Since ctdb_takeover_run() fails, it never runs
recovered event and monitoring does not get re-enabled.
In main_loop, ctdb_takeover_run() is called with a takeover_fail_callback.
This callback will get called if any of the nodes fail in handling
takeip/releaseip/ipreallocated events in ctdb_takeover_run().
Signed-off-by: Amitay Isaacs <amitay at gmail.com>
Cherry-pick-from: a5c6bb1fffb8dc3960af113957a1fd080cc7c245
Conflicts:
include/ctdb_private.h
server/ctdb_takeover.c
commit 5205d545e8d8c72d73b9d5fd148df6de30392fc8
Author: Amitay Isaacs <amitay at gmail.com>
Date: Tue Oct 23 15:06:33 2012 +1100
daemon: Do not ignore timed out monitor events
If an eventscript times out for monitor event, it is considered successful
and the remaining eventscripts are not run. This can make a node prematurely
healthy, cause healthy node to fail over IPs to this node and this node will
not be able to host those IPs. Thus causing loss of access and in case of NAT-GW
configuration, loss of a default route.
Copy-code-from: 6e68797af67bee36f2bad045f94806e7e98f27e9
Signed-off-by: Amitay Isaacs <amitay at gmail.com>
-----------------------------------------------------------------------
Summary of changes:
config/events.d/10.interface | 6 +++---
include/ctdb_private.h | 6 +++---
packaging/RPM/ctdb.spec.in | 8 +++++++-
server/ctdb_recoverd.c | 34 +++++++++++++++++++++++++++++-----
server/ctdb_takeover.c | 32 +++++++++++++++++++++++++-------
server/ctdb_tunables.c | 2 +-
server/eventscript.c | 1 -
tools/ctdb.c | 8 +++++---
8 files changed, 73 insertions(+), 24 deletions(-)
Changeset truncated at 500 lines:
diff --git a/config/events.d/10.interface b/config/events.d/10.interface
index 8f0c0bc..d407154 100755
--- a/config/events.d/10.interface
+++ b/config/events.d/10.interface
@@ -25,9 +25,9 @@ monitor_interfaces()
[ "$CTDB_NATGW_PUBLIC_IFACE" ] && INTERFACES="$CTDB_NATGW_PUBLIC_IFACE $INTERFACES"
- # For all but the 1st line, get the 2nd last field with commas
- # changes to spaces.
- local IFACES=`ctdb -Y ip -v | sed -e '1d' -e 's/:[^:]*:$//' -e 's/^.*://' -e 's/,/ /g'`
+ # Get the configured interfaces for each IP. That is, for all but
+ # the 1st line, get the last field with commas changed to spaces.
+ local IFACES=$(ctdb -Y ip -v | sed -e '1d' -e 's/:$//' -e 's/^.*://' -e 's/,/ /g')
local IFACE
diff --git a/include/ctdb_private.h b/include/ctdb_private.h
index c54e1b0..3ff5957 100644
--- a/include/ctdb_private.h
+++ b/include/ctdb_private.h
@@ -1132,6 +1132,8 @@ int ctdb_sys_send_tcp(const ctdb_sock_addr *dest,
const ctdb_sock_addr *src,
uint32_t seq, uint32_t ack, int rst);
+typedef void (*client_async_callback)(struct ctdb_context *ctdb, uint32_t node_pnn, int32_t res, TDB_DATA outdata, void *callback_data);
+
int ctdb_set_public_addresses(struct ctdb_context *ctdb, const char *alist);
int ctdb_set_single_public_ip(struct ctdb_context *ctdb,
const char *iface,
@@ -1139,7 +1141,7 @@ int ctdb_set_single_public_ip(struct ctdb_context *ctdb,
int ctdb_set_event_script(struct ctdb_context *ctdb, const char *script);
int ctdb_set_event_script_dir(struct ctdb_context *ctdb, const char *script_dir);
int ctdb_set_notification_script(struct ctdb_context *ctdb, const char *script);
-int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap);
+int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap, client_async_callback fail_callback, void *callback_data);
int32_t ctdb_control_tcp_client(struct ctdb_context *ctdb, uint32_t client_id,
TDB_DATA indata);
@@ -1279,8 +1281,6 @@ bool ctdb_stopped_monitoring(struct ctdb_context *ctdb);
int ctdb_set_child_logging(struct ctdb_context *ctdb);
void ctdb_lockdown_memory(struct ctdb_context *ctdb);
-typedef void (*client_async_callback)(struct ctdb_context *ctdb, uint32_t node_pnn, int32_t res, TDB_DATA outdata, void *callback_data);
-
struct client_async_data {
enum ctdb_controls opcode;
bool dont_log_errors;
diff --git a/packaging/RPM/ctdb.spec.in b/packaging/RPM/ctdb.spec.in
index 3ded827..de8193d 100644
--- a/packaging/RPM/ctdb.spec.in
+++ b/packaging/RPM/ctdb.spec.in
@@ -3,7 +3,7 @@ Name: ctdb
Summary: Clustered TDB
Vendor: Samba Team
Packager: Samba Team <samba at samba.org>
-Version: 1.2.54
+Version: 1.2.55
Release: 1GITHASH
Epoch: 0
License: GNU GPL version 3
@@ -146,6 +146,12 @@ development libraries for ctdb
%changelog
+* Tue Nov 27 2012 : Version 1.2.55
+ - Do not ignore timed out monitor events
+ - Track nodes that fail takeover run and ban them
+ - For new interfaces mark the link up by default after startup
+ - In 10.interface eventscript use configured interaces to monitor
+ - Do not set the initial interface when adding new public address
* Wed Oct 30 2012 : Version 1.2.54
- Protect against double free of active monitor callback state during shutdown
- Remove duplicate code to set tunables from initscript
diff --git a/server/ctdb_recoverd.c b/server/ctdb_recoverd.c
index a6a748e..810112f 100644
--- a/server/ctdb_recoverd.c
+++ b/server/ctdb_recoverd.c
@@ -1347,6 +1347,21 @@ static int sync_recovery_lock_file_across_cluster(struct ctdb_recoverd *rec)
/*
+ * this callback is called for every node that failed to execute ctdb_takeover_run()
+ * and set flag to re-run takeover run.
+ */
+static void takeover_fail_callback(struct ctdb_context *ctdb, uint32_t node_pnn, int32_t res, TDB_DATA outdata, void *callback_data)
+{
+ struct ctdb_recoverd *rec = talloc_get_type(callback_data, struct ctdb_recoverd);
+
+ DEBUG(DEBUG_ERR, (__location__ " Node %u failed the takeover run. Setting it as recovery fail culprit\n", node_pnn));
+
+ ctdb_set_culprit(rec, node_pnn);
+ rec->need_takeover_run = true;
+}
+
+
+/*
we are the recmaster, and recovery is needed - start a recovery run
*/
static int do_recovery(struct ctdb_recoverd *rec,
@@ -1633,7 +1648,7 @@ static int do_recovery(struct ctdb_recoverd *rec,
return -1;
}
rec->need_takeover_run = false;
- ret = ctdb_takeover_run(ctdb, nodemap);
+ ret = ctdb_takeover_run(ctdb, nodemap, NULL, NULL);
if (ret != 0) {
DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. ctdb_takeover_run() failed.\n"));
rec->need_takeover_run = true;
@@ -1952,7 +1967,7 @@ static void ctdb_rebalance_timeout(struct event_context *ev, struct timed_event
DEBUG(DEBUG_NOTICE,("Rebalance all nodes that have had ip assignment changes.\n"));
- ret = ctdb_takeover_run(ctdb, rec->nodemap);
+ ret = ctdb_takeover_run(ctdb, rec->nodemap, NULL, NULL);
if (ret != 0) {
DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. ctdb_takeover_run() failed.\n"));
rec->need_takeover_run = true;
@@ -2099,7 +2114,7 @@ static void process_ipreallocate_requests(struct ctdb_context *ctdb, struct ctdb
rec->need_takeover_run = true;
}
if (ret == 0) {
- ret = ctdb_takeover_run(ctdb, rec->nodemap);
+ ret = ctdb_takeover_run(ctdb, rec->nodemap, NULL, NULL);
if (ret != 0) {
DEBUG(DEBUG_ERR,("Failed to reallocate addresses: ctdb_takeover_run() failed.\n"));
rec->need_takeover_run = true;
@@ -3469,9 +3484,18 @@ static void main_loop(struct ctdb_context *ctdb, struct ctdb_recoverd *rec,
return;
}
- ret = ctdb_takeover_run(ctdb, nodemap);
+ /* If takeover run fails, then the offending nodes are
+ * assigned ban culprit counts. And we re-try takeover.
+ * If takeover run fails repeatedly, the node would get
+ * banned.
+ *
+ * If rec->need_takeover_run is not set to true at this
+ * failure, monitoring is disabled cluster-wide (via
+ * startrecovery eventscript) and will not get enabled.
+ */
+ ret = ctdb_takeover_run(ctdb, nodemap, takeover_fail_callback, rec);
if (ret != 0) {
- DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. Try again later\n"));
+ DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. Trying again\n"));
return;
}
diff --git a/server/ctdb_takeover.c b/server/ctdb_takeover.c
index e04ea6a..83b9347 100644
--- a/server/ctdb_takeover.c
+++ b/server/ctdb_takeover.c
@@ -66,7 +66,20 @@ static int ctdb_add_local_iface(struct ctdb_context *ctdb, const char *iface)
CTDB_NO_MEMORY_FATAL(ctdb, i);
i->name = talloc_strdup(i, iface);
CTDB_NO_MEMORY(ctdb, i->name);
- i->link_up = false;
+ /*
+ * If link_up defaults to true then IPs can be allocated to a
+ * node during the first recovery. However, then an interface
+ * could have its link marked down during the startup event,
+ * causing the IP to move almost immediately. If link_up
+ * defaults to false then, during normal operation, IPs added
+ * to a new interface can't be assigned until a monitor cycle
+ * has occurred and marked the new interfaces up. This makes
+ * IP allocation unpredictable. The following is a neat
+ * compromise: early in startup link_up defaults to false, so
+ * IPs can't be assigned, and after startup IPs can be
+ * assigned immediately.
+ */
+ i->link_up = ctdb->done_startup;
DLIST_ADD(ctdb->ifaces, i);
@@ -922,9 +935,6 @@ static int ctdb_add_public_address(struct ctdb_context *ctdb,
talloc_free(vnn);
return -1;
}
- if (i == 0) {
- vnn->iface = ctdb_find_iface(ctdb, vnn->ifaces[i]);
- }
}
DLIST_ADD(ctdb->vnn, vnn);
@@ -1974,7 +1984,8 @@ finished:
/*
make any IP alias changes for public addresses that are necessary
*/
-int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap)
+int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap,
+ client_async_callback fail_callback, void *callback_data)
{
int i;
struct ctdb_public_ip ip;
@@ -2006,6 +2017,9 @@ int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap)
async_data = talloc_zero(tmp_ctx, struct client_async_data);
CTDB_NO_MEMORY_FATAL(ctdb, async_data);
+ async_data->fail_callback = fail_callback;
+ async_data->callback_data = callback_data;
+
for (i=0;i<nodemap->num;i++) {
/* don't talk to unconnected nodes, but do talk to banned nodes */
if (nodemap->nodes[i].flags & NODE_FLAGS_DISCONNECTED) {
@@ -2063,6 +2077,10 @@ int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap)
/* tell all nodes to get their own IPs */
async_data = talloc_zero(tmp_ctx, struct client_async_data);
CTDB_NO_MEMORY_FATAL(ctdb, async_data);
+
+ async_data->fail_callback = fail_callback;
+ async_data->callback_data = callback_data;
+
for (tmp_ip=all_ips;tmp_ip;tmp_ip=tmp_ip->next) {
if (tmp_ip->pnn == -1) {
/* this IP won't be taken over */
@@ -2115,8 +2133,8 @@ ipreallocated:
if (ctdb_client_async_control(ctdb, CTDB_CONTROL_RUN_EVENTSCRIPTS,
nodes, 0, TAKEOVER_TIMEOUT(),
false, data,
- NULL, NULL,
- NULL) != 0) {
+ NULL, fail_callback,
+ callback_data) != 0) {
DEBUG(DEBUG_ERR, (__location__ " ctdb_control to updatenatgw failed\n"));
}
diff --git a/server/ctdb_tunables.c b/server/ctdb_tunables.c
index 62a4dd4..9c70008 100644
--- a/server/ctdb_tunables.c
+++ b/server/ctdb_tunables.c
@@ -37,7 +37,7 @@ static const struct {
{ "MonitorInterval", 15, offsetof(struct ctdb_tunable, monitor_interval) },
{ "TickleUpdateInterval",20, offsetof(struct ctdb_tunable, tickle_update_interval) },
{ "EventScriptTimeout", 30, offsetof(struct ctdb_tunable, script_timeout) },
- { "EventScriptTimeoutCount", 1, offsetof(struct ctdb_tunable, script_timeout_count) },
+ { "EventScriptTimeoutCount", 20, offsetof(struct ctdb_tunable, script_timeout_count) },
{ "EventScriptUnhealthyOnTimeout", 0, offsetof(struct ctdb_tunable, script_unhealthy_on_timeout) },/* OBSOLETE */
{ "RecoveryGracePeriod", 120, offsetof(struct ctdb_tunable, recovery_grace_period) },
{ "RecoveryBanPeriod", 300, offsetof(struct ctdb_tunable, recovery_ban_period) },
diff --git a/server/eventscript.c b/server/eventscript.c
index 2f86fdd..9ef1f3d 100644
--- a/server/eventscript.c
+++ b/server/eventscript.c
@@ -561,7 +561,6 @@ static void ctdb_event_script_timeout(struct event_context *ev, struct timed_eve
case CTDB_EVENT_TAKE_IP:
case CTDB_EVENT_RELEASE_IP:
case CTDB_EVENT_STOPPED:
- case CTDB_EVENT_MONITOR:
case CTDB_EVENT_STATUS:
state->scripts->scripts[state->current].status = 0;
DEBUG(DEBUG_ERR,("Ignoring hung script for %s call %d\n", state->options, state->call));
diff --git a/tools/ctdb.c b/tools/ctdb.c
index 03373a2..b6daa5a 100644
--- a/tools/ctdb.c
+++ b/tools/ctdb.c
@@ -1954,10 +1954,12 @@ static int control_delip(struct ctdb_context *ctdb, int argc, const char **argv)
}
if (ips->ips[i].pnn == options.pnn) {
- ret = find_other_host_for_public_ip(ctdb, &addr);
- if (ret != -1) {
+ int pnn;
+
+ pnn = find_other_host_for_public_ip(ctdb, &addr);
+ if (pnn != -1) {
do {
- ret = move_ip(ctdb, &addr, ret);
+ ret = move_ip(ctdb, &addr, pnn);
if (ret != 0) {
DEBUG(DEBUG_ERR,("Failed to move ip to node %d. Wait 3 seconds and try again.\n", options.pnn));
sleep(3);
--
CTDB repository
More information about the samba-cvs
mailing list