[SCM] Samba Shared Repository - branch master updated
Michael Adam
obnox at samba.org
Tue Jun 17 03:58:05 MDT 2014
The branch, master has been updated
via a4c50e2 autobuild: fix samba-ctdb samba build to pick up right tdb
via 30f3a86 autobuild: add cflags and ldflags to find locally installed tdb
via 5334881 ctdb:vacuum: always run freelist_size again
via 026d79c ctdb:vacuum: add missing return to ctdb_vacuum_traverse_db() error path.
via b8658b3 ctdb:vacuum: remove now unused talloc ctx argument from ctdb_vacuum_db()
via c3cb8c2 ctdb:vacuum: move init of vdata into init_vdata funcion
via 3cf0189 ctdb:vacuum: remove vacuum limit from vdata - not used
via a99035a ctdb:vacuum: remove a superfluous comment.
via 385e223 ctdb:vacuum: untangle assignmend and check for return of tdb_repack()
via e0a1136 ctdb:vacuum: remove a superfluous and misleading comment
via 413f99f ctdb:vacuum: remove now unused ctdb_repack_tdb().
via 368683d ctdb:vacuum: use plain tdb_repack() instead of ctdb_repack_tdb()
via ec2f1ab ctdb:vacuum: remove a comment in ctdb_vacuum_and_repack_db()
via 67e5746 ctdb:vacuum: adapt debug message for repacking.
from 35dd4de messaging3: Use server_id_str_buf
http://gitweb.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit a4c50e233265ef4fd0d472cbded0bb8346ef0528
Author: Michael Adam <obnox at samba.org>
Date: Wed Jun 11 16:40:02 2014 +0200
autobuild: fix samba-ctdb samba build to pick up right tdb
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
Autobuild-User(master): Michael Adam <obnox at samba.org>
Autobuild-Date(master): Tue Jun 17 11:58:00 CEST 2014 on sn-devel-104
commit 30f3a8661996a47c3237d4e553666f0d8b2be08e
Author: Michael Adam <obnox at samba.org>
Date: Wed Jun 11 13:57:26 2014 +0200
autobuild: add cflags and ldflags to find locally installed tdb
So that we can find our self-installed tdb lib and use new features.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 5334881afab42eae77bb2015ec21cbfe1df87807
Author: Michael Adam <obnox at samba.org>
Date: Sat Feb 15 01:36:06 2014 +0100
ctdb:vacuum: always run freelist_size again
and not only if repack_limit != 0. This partially reverts
commit 48f2d1158820bfb063ba0a0bbfb6f496a8e7522.
With the new tdb code this defragments the
free list by merging adjacent records.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 026d79cb009beba6987da6a6dd5fd98609140136
Author: Michael Adam <obnox at samba.org>
Date: Tue Apr 22 22:09:35 2014 +0200
ctdb:vacuum: add missing return to ctdb_vacuum_traverse_db() error path.
This got lost in commit 19948702992c94553e1a611540ad398de9f9d8b9
("ctdb-vacuum: make ctdb_vacuum_traverse_db() void.")
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit b8658b395921a5400c9f794a07748f5ad18991f8
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 03:36:49 2014 +0200
ctdb:vacuum: remove now unused talloc ctx argument from ctdb_vacuum_db()
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit c3cb8c277a02a8a68c11ef8d341c8116172e989b
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 03:34:05 2014 +0200
ctdb:vacuum: move init of vdata into init_vdata funcion
This is a small code cleanup.
vdata is only used in ctdb_vacuum_db() and not in
ctdb_vacuum_and_repack_db() where it is currently initialized.
This patch moves creation and all previously scattered
initialization of vacuum_data into ctdb_vacuum_init_vacuum_data
which is called from ctdb_vacuum_db.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 3cf018935e057c1748ab44491135c632c023de9f
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 03:08:20 2014 +0200
ctdb:vacuum: remove vacuum limit from vdata - not used
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit a99035a4c52f68a4a4f1862c74c1c71273a47d5b
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 03:02:42 2014 +0200
ctdb:vacuum: remove a superfluous comment.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 385e2236fbc2a4dca97c3b2f83c592f99abaee25
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 02:59:51 2014 +0200
ctdb:vacuum: untangle assignmend and check for return of tdb_repack()
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit e0a11369cf85b111e0966f2677315345ddaaac76
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 02:57:25 2014 +0200
ctdb:vacuum: remove a superfluous and misleading comment
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 413f99f86e26a8b12a807d0c2436c78805983566
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 02:57:00 2014 +0200
ctdb:vacuum: remove now unused ctdb_repack_tdb().
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 368683d7af747e3d67c48a702ff315a2022c6f85
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 02:53:29 2014 +0200
ctdb:vacuum: use plain tdb_repack() instead of ctdb_repack_tdb()
Since we usually have 0 records left for repack-deletion,
repacking is essentially used for the purpose of defragmenting
the freelist, we can use the vanilla tdb_repack function.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit ec2f1abe486c77f8a27ebc731917337efc9cd04f
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 02:47:21 2014 +0200
ctdb:vacuum: remove a comment in ctdb_vacuum_and_repack_db()
The repack operation now mainly defragments the freelist
and does not usually delete any records any more.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
commit 67e5746b236cca860d02f741411b567a37a9cf4e
Author: Michael Adam <obnox at samba.org>
Date: Sat Apr 19 02:51:39 2014 +0200
ctdb:vacuum: adapt debug message for repacking.
Now we usually have records to delete == 0 after the preceding
vacuum run. Anyways, deletion is not a major aspect any more
of the repack run and will vanish soon.
Signed-off-by: Michael Adam <obnox at samba.org>
Reviewed-by: Amitay Isaacs <amitay at gmail.com>
-----------------------------------------------------------------------
Summary of changes:
ctdb/server/ctdb_vacuum.c | 238 ++++++++++----------------------------------
script/autobuild.py | 4 +-
2 files changed, 56 insertions(+), 186 deletions(-)
Changeset truncated at 500 lines:
diff --git a/ctdb/server/ctdb_vacuum.c b/ctdb/server/ctdb_vacuum.c
index 574ad87..ce3c600 100644
--- a/ctdb/server/ctdb_vacuum.c
+++ b/ctdb/server/ctdb_vacuum.c
@@ -53,7 +53,6 @@ struct ctdb_vacuum_handle {
/* a list of records to possibly delete */
struct vacuum_data {
- uint32_t repack_limit;
struct ctdb_context *ctdb;
struct ctdb_db_context *ctdb_db;
struct tdb_context *dest_db;
@@ -745,6 +744,7 @@ static void ctdb_vacuum_traverse_db(struct ctdb_db_context *ctdb_db,
if (ret == -1 || vdata->traverse_error) {
DEBUG(DEBUG_ERR, (__location__ " Traverse error in vacuuming "
"'%s'\n", ctdb_db->db_name));
+ return;
}
if (vdata->count.db_traverse.total > 0) {
@@ -1171,11 +1171,29 @@ done:
/**
* initialize the vacuum_data
*/
-static int ctdb_vacuum_init_vacuum_data(struct ctdb_db_context *ctdb_db,
- struct vacuum_data *vdata)
+static struct vacuum_data *ctdb_vacuum_init_vacuum_data(
+ struct ctdb_db_context *ctdb_db,
+ TALLOC_CTX *mem_ctx)
{
int i;
struct ctdb_context *ctdb = ctdb_db->ctdb;
+ struct vacuum_data *vdata;
+
+ vdata = talloc_zero(mem_ctx, struct vacuum_data);
+ if (vdata == NULL) {
+ DEBUG(DEBUG_ERR,(__location__ " Out of memory\n"));
+ return NULL;
+ }
+
+ vdata->ctdb = ctdb_db->ctdb;
+ vdata->ctdb_db = ctdb_db;
+ vdata->delete_list = trbt_create(vdata, 0);
+ if (vdata->delete_list == NULL) {
+ DEBUG(DEBUG_ERR,(__location__ " Out of memory\n"));
+ goto fail;
+ }
+
+ vdata->start = timeval_current();
vdata->count.delete_queue.added_to_delete_list = 0;
vdata->count.delete_queue.added_to_vacuum_fetch_list = 0;
@@ -1200,7 +1218,7 @@ static int ctdb_vacuum_init_vacuum_data(struct ctdb_db_context *ctdb_db,
ctdb->num_nodes);
if (vdata->vacuum_fetch_list == NULL) {
DEBUG(DEBUG_ERR,(__location__ " Out of memory\n"));
- return -1;
+ goto fail;
}
for (i = 0; i < ctdb->num_nodes; i++) {
vdata->vacuum_fetch_list[i] = (struct ctdb_marshall_buffer *)
@@ -1208,12 +1226,17 @@ static int ctdb_vacuum_init_vacuum_data(struct ctdb_db_context *ctdb_db,
offsetof(struct ctdb_marshall_buffer, data));
if (vdata->vacuum_fetch_list[i] == NULL) {
DEBUG(DEBUG_ERR,(__location__ " Out of memory\n"));
- return -1;
+ talloc_free(vdata);
+ return NULL;
}
vdata->vacuum_fetch_list[i]->db_id = ctdb_db->db_id;
}
- return 0;
+ return vdata;
+
+fail:
+ talloc_free(vdata);
+ return NULL;
}
/**
@@ -1249,11 +1272,12 @@ static int ctdb_vacuum_init_vacuum_data(struct ctdb_db_context *ctdb_db,
* This executes in the child context.
*/
static int ctdb_vacuum_db(struct ctdb_db_context *ctdb_db,
- struct vacuum_data *vdata,
bool full_vacuum_run)
{
struct ctdb_context *ctdb = ctdb_db->ctdb;
int ret, pnn;
+ struct vacuum_data *vdata;
+ TALLOC_CTX *tmp_ctx;
DEBUG(DEBUG_INFO, (__location__ " Entering %s vacuum run for db "
"%s db_id[0x%08x]\n",
@@ -1274,9 +1298,16 @@ static int ctdb_vacuum_db(struct ctdb_db_context *ctdb_db,
ctdb->pnn = pnn;
- ret = ctdb_vacuum_init_vacuum_data(ctdb_db, vdata);
- if (ret != 0) {
- return ret;
+ tmp_ctx = talloc_new(ctdb_db);
+ if (tmp_ctx == NULL) {
+ DEBUG(DEBUG_ERR, ("Out of memory!\n"));
+ return -1;
+ }
+
+ vdata = ctdb_vacuum_init_vacuum_data(ctdb_db, tmp_ctx);
+ if (vdata == NULL) {
+ talloc_free(tmp_ctx);
+ return -1;
}
if (full_vacuum_run) {
@@ -1289,186 +1320,34 @@ static int ctdb_vacuum_db(struct ctdb_db_context *ctdb_db,
ctdb_process_delete_list(ctdb_db, vdata);
+ talloc_free(tmp_ctx);
+
/* this ensures we run our event queue */
ctdb_ctrl_getpnn(ctdb, TIMELIMIT(), CTDB_CURRENT_NODE);
return 0;
}
-
-/*
- * traverse function for repacking
- */
-static int repack_traverse(struct tdb_context *tdb, TDB_DATA key, TDB_DATA data,
- void *private_data)
-{
- struct vacuum_data *vdata = (struct vacuum_data *)private_data;
-
- if (vdata->vacuum) {
- uint32_t hash = ctdb_hash(&key);
- struct delete_record_data *kd;
- /*
- * check if we can ignore this record because it's in the delete_list
- */
- kd = (struct delete_record_data *)trbt_lookup32(vdata->delete_list, hash);
- /*
- * there might be hash collisions so we have to compare the keys here to be sure
- */
- if (kd && kd->key.dsize == key.dsize && memcmp(kd->key.dptr, key.dptr, key.dsize) == 0) {
- struct ctdb_ltdb_header *hdr = (struct ctdb_ltdb_header *)data.dptr;
- /*
- * we have to check if the record hasn't changed in the meantime in order to
- * savely remove it from the database
- */
- if (data.dsize == sizeof(struct ctdb_ltdb_header) &&
- hdr->dmaster == kd->ctdb->pnn &&
- ctdb_lmaster(kd->ctdb, &(kd->key)) == kd->ctdb->pnn &&
- kd->hdr.rsn == hdr->rsn) {
- vdata->count.repack.vacuumed++;
- return 0;
- }
- }
- }
- if (tdb_store(vdata->dest_db, key, data, TDB_INSERT) != 0) {
- vdata->traverse_error = true;
- return -1;
- }
- vdata->count.repack.copied++;
- return 0;
-}
-
-/*
- * repack a tdb
- */
-static int ctdb_repack_tdb(struct tdb_context *tdb, TALLOC_CTX *mem_ctx, struct vacuum_data *vdata)
-{
- struct tdb_context *tmp_db;
-
- if (tdb_transaction_start(tdb) != 0) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to start transaction\n"));
- return -1;
- }
-
- tmp_db = tdb_open("tmpdb", tdb_hash_size(tdb),
- TDB_INTERNAL|TDB_DISALLOW_NESTING,
- O_RDWR|O_CREAT, 0);
- if (tmp_db == NULL) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to create tmp_db\n"));
- tdb_transaction_cancel(tdb);
- return -1;
- }
-
- vdata->traverse_error = false;
- vdata->dest_db = tmp_db;
- vdata->vacuum = true;
- vdata->count.repack.vacuumed = 0;
- vdata->count.repack.copied = 0;
-
- /*
- * repack and vacuum on-the-fly by not writing the records that are
- * no longer needed
- */
- if (tdb_traverse_read(tdb, repack_traverse, vdata) == -1) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to traverse copying out\n"));
- tdb_transaction_cancel(tdb);
- tdb_close(tmp_db);
- return -1;
- }
-
- DEBUG(DEBUG_INFO,(__location__ " %u records vacuumed\n",
- vdata->count.repack.vacuumed));
-
- if (vdata->traverse_error) {
- DEBUG(DEBUG_ERR,(__location__ " Error during traversal\n"));
- tdb_transaction_cancel(tdb);
- tdb_close(tmp_db);
- return -1;
- }
-
- if (tdb_wipe_all(tdb) != 0) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to wipe database\n"));
- tdb_transaction_cancel(tdb);
- tdb_close(tmp_db);
- return -1;
- }
-
- vdata->traverse_error = false;
- vdata->dest_db = tdb;
- vdata->vacuum = false;
- vdata->count.repack.copied = 0;
-
- if (tdb_traverse_read(tmp_db, repack_traverse, vdata) == -1) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to traverse copying back\n"));
- tdb_transaction_cancel(tdb);
- tdb_close(tmp_db);
- return -1;
- }
-
- if (vdata->traverse_error) {
- DEBUG(DEBUG_ERR,(__location__ " Error during second traversal\n"));
- tdb_transaction_cancel(tdb);
- tdb_close(tmp_db);
- return -1;
- }
-
- tdb_close(tmp_db);
-
-
- if (tdb_transaction_commit(tdb) != 0) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to commit\n"));
- return -1;
- }
- DEBUG(DEBUG_INFO,(__location__ " %u records copied\n",
- vdata->count.repack.copied));
-
- return 0;
-}
-
/*
* repack and vaccum a db
* called from the child context
*/
static int ctdb_vacuum_and_repack_db(struct ctdb_db_context *ctdb_db,
- TALLOC_CTX *mem_ctx,
bool full_vacuum_run)
{
uint32_t repack_limit = ctdb_db->ctdb->tunable.repack_limit;
const char *name = ctdb_db->db_name;
int freelist_size = 0;
- struct vacuum_data *vdata;
-
- vdata = talloc_zero(mem_ctx, struct vacuum_data);
- if (vdata == NULL) {
- DEBUG(DEBUG_ERR,(__location__ " Out of memory\n"));
- return -1;
- }
-
- vdata->ctdb = ctdb_db->ctdb;
- vdata->repack_limit = repack_limit;
- vdata->delete_list = trbt_create(vdata, 0);
- vdata->ctdb_db = ctdb_db;
- if (vdata->delete_list == NULL) {
- DEBUG(DEBUG_ERR,(__location__ " Out of memory\n"));
- talloc_free(vdata);
- return -1;
- }
+ int ret;
- vdata->start = timeval_current();
-
- /*
- * gather all records that can be deleted in vdata
- */
- if (ctdb_vacuum_db(ctdb_db, vdata, full_vacuum_run) != 0) {
+ if (ctdb_vacuum_db(ctdb_db, full_vacuum_run) != 0) {
DEBUG(DEBUG_ERR,(__location__ " Failed to vacuum '%s'\n", name));
}
- if (repack_limit != 0) {
- freelist_size = tdb_freelist_size(ctdb_db->ltdb->tdb);
- if (freelist_size == -1) {
- DEBUG(DEBUG_ERR,(__location__ " Failed to get freelist size for '%s'\n", name));
- talloc_free(vdata);
- return -1;
- }
+ freelist_size = tdb_freelist_size(ctdb_db->ltdb->tdb);
+ if (freelist_size == -1) {
+ DEBUG(DEBUG_ERR,(__location__ " Failed to get freelist size for '%s'\n", name));
+ return -1;
}
/*
@@ -1476,22 +1355,17 @@ static int ctdb_vacuum_and_repack_db(struct ctdb_db_context *ctdb_db,
*/
if ((repack_limit == 0 || (uint32_t)freelist_size < repack_limit))
{
- talloc_free(vdata);
return 0;
}
- DEBUG(DEBUG_INFO,("Repacking %s with %u freelist entries and %u records to delete\n",
- name, freelist_size, vdata->count.delete_list.left));
+ DEBUG(DEBUG_INFO, ("Repacking %s with %u freelist entries\n",
+ name, freelist_size));
- /*
- * repack and implicitely get rid of the records we can delete
- */
- if (ctdb_repack_tdb(ctdb_db->ltdb->tdb, mem_ctx, vdata) != 0) {
+ ret = tdb_repack(ctdb_db->ltdb->tdb);
+ if (ret != 0) {
DEBUG(DEBUG_ERR,(__location__ " Failed to repack '%s'\n", name));
- talloc_free(vdata);
return -1;
}
- talloc_free(vdata);
return 0;
}
@@ -1641,16 +1515,12 @@ ctdb_vacuum_event(struct event_context *ev, struct timed_event *te,
_exit(1);
}
- /*
- * repack the db
- */
if ((ctdb->tunable.vacuum_fast_path_count > 0) &&
(vacuum_handle->fast_path_count == 0))
{
full_vacuum_run = true;
}
- cc = ctdb_vacuum_and_repack_db(ctdb_db, child_ctx,
- full_vacuum_run);
+ cc = ctdb_vacuum_and_repack_db(ctdb_db, full_vacuum_run);
write(child_ctx->fd[1], &cc, 1);
_exit(0);
diff --git a/script/autobuild.py b/script/autobuild.py
index 3b634e8..14377c0 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -61,13 +61,13 @@ tasks = {
# build and install ctdb:
("ctdb-autogen", "cd ./ctdb && ./autogen.sh", "text/plain"),
- ("ctdb-configure", "cd ./ctdb && ./configure ${PREFIX} --enable-socket-wrapper --with-included-tdb=no", "text/plain"),
+ ("ctdb-configure", "cd ./ctdb && CFLAGS=-I${PREFIX_DIR}/include LDFLAGS=-L${PREFIX_DIR}/lib ./configure ${PREFIX} --enable-socket-wrapper --with-included-tdb=no", "text/plain"),
("ctdb-make", "cd ./ctdb && make all", "text/plain"),
("ctdb-install", "cd ./ctdb && make install", "text/plain"),
("ctdb-header-ls", "ls ${PREFIX_DIR}/include/ctdb.h", "text/plain"),
# build samba with cluster support against this ctdb:
- ("samba-configure", "PYTHONPATH=${PYTHON_PREFIX}/site-packages:$PYTHONPATH PKG_CONFIG_PATH=$PKG_CONFIG_PATH:${PREFIX_DIR}/lib/pkgconfig ./configure.developer ${PREFIX} ${PERL_VENDOR_LIB} --with-selftest-prefix=./bin/ab --with-cluster-support --with-ctdb-dir=${PREFIX_DIR} --bundled-libraries=!tdb", "text/plain"),
+ ("samba-configure", "PYTHONPATH=${PYTHON_PREFIX}/site-packages:$PYTHONPATH PKG_CONFIG_PATH=${PREFIX_DIR}/lib/pkgconfig:${PKG_CONFIG_PATH} ./configure.developer ${PREFIX} ${PERL_VENDOR_LIB} --with-selftest-prefix=./bin/ab --with-cluster-support --with-ctdb-dir=${PREFIX_DIR} --bundled-libraries=!tdb", "text/plain"),
("samba-make", "make", "text/plain"),
("samba-check", "./bin/smbd -b | grep CLUSTER_SUPPORT", "text/plain"),
("samba-install", "make install", "text/plain"),
--
Samba Shared Repository
More information about the samba-cvs
mailing list