[SCM] Samba Shared Repository - branch master updated
Volker Lendecke
vlendec at samba.org
Fri Jul 15 18:33:03 UTC 2016
The branch, master has been updated
via db020b3 smbd: Remove a reference to dbwrap_watch_db()
via f23b25c smbd: Convert smbXsrv_open_global.tdb to new dbwrap_watch
via 83e4e1a smbd: Convert locking.tdb to new dbwrap_watch
via db22df4 lib: Convert g_lock to new dbwrap_watch
via 54d0dbe dbwrap: Add an alternative implementation of dbwrap_watch_record_send
via a672379 dbwrap: Add overflow protection to dbwrap_record_watchers_key()
via e364e51 g_lock: Use "blocker" argument to dbwrap_record_watch_send
via d4ca284 dbwrap: Add "blocker" to record_watch_send
via 7d84267 lib: Add server_id_watch_send
from 9d1883a renamedc: Make a more targeted dbcheck
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit db020b390342ebe03cfa672bb9d3cffe28a404f7
Author: Volker Lendecke <vl at samba.org>
Date: Thu Jul 14 07:43:20 2016 +0200
smbd: Remove a reference to dbwrap_watch_db()
This has never been watched, so it's an unnecessary overhead on
dbwrap_record_store().
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
Autobuild-User(master): Volker Lendecke <vl at samba.org>
Autobuild-Date(master): Fri Jul 15 20:32:19 CEST 2016 on sn-devel-144
commit f23b25cfbe62e4e3a43ff55fe95680ca33733467
Author: Volker Lendecke <vl at samba.org>
Date: Wed Jul 13 07:41:02 2016 +0200
smbd: Convert smbXsrv_open_global.tdb to new dbwrap_watch
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 83e4e1a78645011f4cf527f0eb1a5b0f7a815f97
Author: Volker Lendecke <vl at samba.org>
Date: Wed Jul 13 07:27:30 2016 +0200
smbd: Convert locking.tdb to new dbwrap_watch
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit db22df41727ce6e9d778307f4137652820b984e1
Author: Volker Lendecke <vl at samba.org>
Date: Wed Jul 13 07:26:52 2016 +0200
lib: Convert g_lock to new dbwrap_watch
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 54d0dbeae6a914f985e0f6dd51fbc290a232c3dd
Author: Volker Lendecke <vl at samba.org>
Date: Tue Jul 12 15:59:56 2016 +0200
dbwrap: Add an alternative implementation of dbwrap_watch_record_send
The existing one with a separate dbwrap_watchers.tdb turns out to
create a performance penalty in a clustered environment. Non-clustered,
dbwrap_parse_record on non-existent records is very cheap, but in a
cluster environment this is very noticable.
This implementation puts the watcher information into the records itself. For
large records, this might be another performance penalty, because we have to
assemble the final record together with talloc and memcpy, but this might be
fixed later with a tdb_storev call.
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit a672379bd9a7934a2cc82ba340723044048676e8
Author: Volker Lendecke <vl at samba.org>
Date: Tue Jul 12 15:57:29 2016 +0200
dbwrap: Add overflow protection to dbwrap_record_watchers_key()
It's highly unlinkely that this will ever kick in, because our current tdb keys
are rather small, but offset calculations without overflow checks are bad.
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit e364e5114a9dadd546b840b533cee13e69281923
Author: Volker Lendecke <vl at samba.org>
Date: Tue Jul 12 16:07:51 2016 +0200
g_lock: Use "blocker" argument to dbwrap_record_watch_send
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit d4ca284333dbef0e1d20b5577f5746514aedfd91
Author: Volker Lendecke <vl at samba.org>
Date: Thu Mar 10 14:37:12 2016 +0100
dbwrap: Add "blocker" to record_watch_send
Typicall, when we watch a record, we wait for a process to give up some
resource. Be it an oplock, a share mode or the g_lock. If everything goes well,
the blocker sends us a message. If the blocker dies hard, we want to also be
informed immediately.
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 7d84267923c6b276ec0b54d07bb0995b69f228a9
Author: Volker Lendecke <vl at samba.org>
Date: Tue Jul 12 15:33:59 2016 +0200
lib: Add server_id_watch_send
This is a brute force variant, trying twice a second. We'll have better
variants with tmsgd in the future.
Signed-off-by: Volker Lendecke <vl at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
-----------------------------------------------------------------------
Summary of changes:
source3/lib/dbwrap/dbwrap_watch.c | 860 ++++++++++++++++++++-
source3/lib/dbwrap/dbwrap_watch.h | 19 +-
source3/lib/g_lock.c | 41 +-
source3/lib/server_id_watch.c | 105 +++
.../lib/{messages_dgm_ref.h => server_id_watch.h} | 29 +-
source3/locking/share_mode_lock.c | 18 +-
source3/smbd/globals.h | 2 +-
source3/smbd/open.c | 9 +-
source3/smbd/server.c | 2 +-
source3/smbd/smb2_setinfo.c | 7 +-
source3/smbd/smbXsrv_client.c | 2 -
source3/smbd/smbXsrv_session.c | 40 +-
source3/torture/test_dbwrap_watch.c | 19 +-
source3/wscript_build | 1 +
14 files changed, 1073 insertions(+), 81 deletions(-)
create mode 100644 source3/lib/server_id_watch.c
copy source3/lib/{messages_dgm_ref.h => server_id_watch.h} (63%)
Changeset truncated at 500 lines:
diff --git a/source3/lib/dbwrap/dbwrap_watch.c b/source3/lib/dbwrap/dbwrap_watch.c
index 09e67fb..b8ffb04 100644
--- a/source3/lib/dbwrap/dbwrap_watch.c
+++ b/source3/lib/dbwrap/dbwrap_watch.c
@@ -24,6 +24,8 @@
#include "dbwrap_open.h"
#include "lib/util/util_tdb.h"
#include "lib/util/tevent_ntstatus.h"
+#include "server_id_watch.h"
+#include "lib/dbwrap/dbwrap_private.h"
static struct db_context *dbwrap_record_watchers_db(void)
{
@@ -45,9 +47,9 @@ static struct db_context *dbwrap_record_watchers_db(void)
return watchers_db;
}
-static size_t dbwrap_record_watchers_key(struct db_context *db,
- struct db_record *rec,
- uint8_t *wkey, size_t wkey_len)
+static ssize_t dbwrap_record_watchers_key(struct db_context *db,
+ struct db_record *rec,
+ uint8_t *wkey, size_t wkey_len)
{
size_t db_id_len = dbwrap_db_id(db, NULL, 0);
uint8_t db_id[db_id_len];
@@ -58,7 +60,15 @@ static size_t dbwrap_record_watchers_key(struct db_context *db,
key = dbwrap_record_get_key(rec);
- needed = sizeof(uint32_t) + db_id_len + key.dsize;
+ needed = sizeof(uint32_t) + db_id_len;
+ if (needed < sizeof(uint32_t)) {
+ return -1;
+ }
+
+ needed += key.dsize;
+ if (needed < key.dsize) {
+ return -1;
+ }
if (wkey_len >= needed) {
SIVAL(wkey, 0, db_id_len);
@@ -199,23 +209,28 @@ struct dbwrap_record_watch_state {
struct tevent_req *req;
struct messaging_context *msg;
TDB_DATA w_key;
+ bool blockerdead;
+ struct server_id blocker;
};
static bool dbwrap_record_watch_filter(struct messaging_rec *rec,
void *private_data);
static void dbwrap_record_watch_done(struct tevent_req *subreq);
+static void dbwrap_record_watch_blocker_died(struct tevent_req *subreq);
static int dbwrap_record_watch_state_destructor(
struct dbwrap_record_watch_state *state);
struct tevent_req *dbwrap_record_watch_send(TALLOC_CTX *mem_ctx,
struct tevent_context *ev,
struct db_record *rec,
- struct messaging_context *msg)
+ struct messaging_context *msg,
+ struct server_id blocker)
{
struct tevent_req *req, *subreq;
struct dbwrap_record_watch_state *state;
struct db_context *watchers_db;
NTSTATUS status;
+ ssize_t needed;
req = tevent_req_create(mem_ctx, &state,
struct dbwrap_record_watch_state);
@@ -226,6 +241,7 @@ struct tevent_req *dbwrap_record_watch_send(TALLOC_CTX *mem_ctx,
state->ev = ev;
state->req = req;
state->msg = msg;
+ state->blocker = blocker;
watchers_db = dbwrap_record_watchers_db();
if (watchers_db == NULL) {
@@ -233,8 +249,12 @@ struct tevent_req *dbwrap_record_watch_send(TALLOC_CTX *mem_ctx,
return tevent_req_post(req, ev);
}
- state->w_key.dsize = dbwrap_record_watchers_key(
- state->db, rec, NULL, 0);
+ needed = dbwrap_record_watchers_key(state->db, rec, NULL, 0);
+ if (needed == -1) {
+ tevent_req_nterror(req, NT_STATUS_INSUFFICIENT_RESOURCES);
+ return tevent_req_post(req, ev);
+ }
+ state->w_key.dsize = needed;
state->w_key.dptr = talloc_array(state, uint8_t, state->w_key.dsize);
if (tevent_req_nomem(state->w_key.dptr, req)) {
@@ -250,6 +270,15 @@ struct tevent_req *dbwrap_record_watch_send(TALLOC_CTX *mem_ctx,
}
tevent_req_set_callback(subreq, dbwrap_record_watch_done, req);
+ if (blocker.pid != 0) {
+ subreq = server_id_watch_send(state, ev, msg, blocker);
+ if (tevent_req_nomem(subreq, req)) {
+ return tevent_req_post(req, ev);
+ }
+ tevent_req_set_callback(
+ subreq, dbwrap_record_watch_blocker_died, req);
+ }
+
status = dbwrap_record_add_watcher(
state->w_key, messaging_server_id(state->msg));
if (tevent_req_nterror(req, status)) {
@@ -371,9 +400,29 @@ static void dbwrap_record_watch_done(struct tevent_req *subreq)
tevent_req_done(req);
}
+static void dbwrap_record_watch_blocker_died(struct tevent_req *subreq)
+{
+ struct tevent_req *req = tevent_req_callback_data(
+ subreq, struct tevent_req);
+ struct dbwrap_record_watch_state *state = tevent_req_data(
+ req, struct dbwrap_record_watch_state);
+ int ret;
+
+ ret = server_id_watch_recv(subreq, NULL);
+ TALLOC_FREE(subreq);
+ if (ret != 0) {
+ tevent_req_nterror(req, map_nt_error_from_unix(ret));
+ return;
+ }
+ state->blockerdead = true;
+ tevent_req_done(req);
+}
+
NTSTATUS dbwrap_record_watch_recv(struct tevent_req *req,
TALLOC_CTX *mem_ctx,
- struct db_record **prec)
+ struct db_record **prec,
+ bool *blockerdead,
+ struct server_id *blocker)
{
struct dbwrap_record_watch_state *state = tevent_req_data(
req, struct dbwrap_record_watch_state);
@@ -385,6 +434,12 @@ NTSTATUS dbwrap_record_watch_recv(struct tevent_req *req,
if (tevent_req_is_nterror(req, &status)) {
return status;
}
+ if (blockerdead != NULL) {
+ *blockerdead = state->blockerdead;
+ }
+ if (blocker != NULL) {
+ *blocker = state->blocker;
+ }
if (prec == NULL) {
return NT_STATUS_OK;
}
@@ -479,3 +534,792 @@ void dbwrap_watchers_wakeall(struct messaging_context *msg)
{
dbwrap_watchers_traverse_read(dbwrap_wakeall_cb, msg);
}
+
+/*
+ * Watched records contain a header of:
+ *
+ * [uint32] num_records | deleted bit
+ * 0 [SERVER_ID_BUF_LENGTH] \
+ * 1 [SERVER_ID_BUF_LENGTH] |
+ * .. |- Array of watchers
+ * (num_records-1)[SERVER_ID_BUF_LENGTH] /
+ *
+ * [Remainder of record....]
+ *
+ * If this header is absent then this is a
+ * fresh record of length zero (no watchers).
+ *
+ * Note that a record can be deleted with
+ * watchers present. If so the deleted bit
+ * is set and the watcher server_id's are
+ * woken to allow them to remove themselves
+ * from the watcher array. The record is left
+ * present marked with the deleted bit until all
+ * watchers are removed, then the record itself
+ * is deleted.
+ */
+
+#define NUM_WATCHERS_DELETED_BIT (1UL<<31)
+#define NUM_WATCHERS_MASK (NUM_WATCHERS_DELETED_BIT-1)
+
+static ssize_t dbwrap_watched_parse(TDB_DATA data, struct server_id *ids,
+ size_t num_ids, bool *pdeleted,
+ TDB_DATA *pdata)
+{
+ size_t i, num_watchers;
+ bool deleted;
+
+ if (data.dsize < sizeof(uint32_t)) {
+ /* Fresh or invalid record */
+ return -1;
+ }
+
+ num_watchers = IVAL(data.dptr, 0);
+
+ deleted = num_watchers & NUM_WATCHERS_DELETED_BIT;
+ num_watchers &= NUM_WATCHERS_MASK;
+
+ data.dptr += sizeof(uint32_t);
+ data.dsize -= sizeof(uint32_t);
+
+ if (num_watchers > data.dsize/SERVER_ID_BUF_LENGTH) {
+ /* Invalid record */
+ return -1;
+ }
+
+ if (num_watchers > num_ids) {
+ /*
+ * Not enough space to store the watchers server_id's.
+ * Just move past all of them to allow the remaining part
+ * of the record to be returned.
+ */
+ data.dptr += num_watchers * SERVER_ID_BUF_LENGTH;
+ data.dsize -= num_watchers * SERVER_ID_BUF_LENGTH;
+ goto done;
+ }
+
+ /*
+ * Note, even if marked deleted we still must
+ * return the id's array to allow awoken
+ * watchers to remove themselves.
+ */
+
+ for (i=0; i<num_watchers; i++) {
+ server_id_get(&ids[i], data.dptr);
+ data.dptr += SERVER_ID_BUF_LENGTH;
+ data.dsize -= SERVER_ID_BUF_LENGTH;
+ }
+
+done:
+ if (deleted) {
+ data = (TDB_DATA) {0};
+ }
+ if (pdata != NULL) {
+ *pdata = data;
+ }
+ if (pdeleted != NULL) {
+ *pdeleted = deleted;
+ }
+
+ return num_watchers;
+}
+
+static ssize_t dbwrap_watched_unparse(const struct server_id *watchers,
+ size_t num_watchers, bool deleted,
+ TDB_DATA data,
+ uint8_t *buf, size_t buflen)
+{
+ size_t i, len, ofs;
+ uint32_t num_watchers_buf;
+
+ if (num_watchers > UINT32_MAX/SERVER_ID_BUF_LENGTH) {
+ return -1;
+ }
+
+ len = num_watchers * SERVER_ID_BUF_LENGTH;
+
+ len += sizeof(uint32_t);
+ if (len < sizeof(uint32_t)) {
+ return -1;
+ }
+
+ len += data.dsize;
+ if (len < data.dsize) {
+ return -1;
+ }
+
+ if (len > buflen) {
+ return len;
+ }
+
+ num_watchers_buf = num_watchers;
+ if (deleted) {
+ num_watchers_buf |= NUM_WATCHERS_DELETED_BIT;
+ }
+
+ ofs = 0;
+ SIVAL(buf, ofs, num_watchers_buf);
+ ofs += 4;
+
+ for (i=0; i<num_watchers; i++) {
+ server_id_put(buf+ofs, watchers[i]);
+ ofs += SERVER_ID_BUF_LENGTH;
+ }
+
+ if ((data.dptr != NULL) && (data.dsize != 0)) {
+ memcpy(buf + ofs, data.dptr, data.dsize);
+ }
+
+ return len;
+}
+
+struct db_watched_ctx {
+ struct db_context *backend;
+ struct messaging_context *msg;
+};
+
+struct db_watched_subrec {
+ struct db_record *subrec;
+ struct server_id *watchers;
+ bool deleted;
+};
+
+static NTSTATUS dbwrap_watched_store(struct db_record *rec, TDB_DATA data,
+ int flag);
+static NTSTATUS dbwrap_watched_delete(struct db_record *rec);
+
+static struct db_record *dbwrap_watched_fetch_locked(
+ struct db_context *db, TALLOC_CTX *mem_ctx, TDB_DATA key)
+{
+ struct db_watched_ctx *ctx = talloc_get_type_abort(
+ db->private_data, struct db_watched_ctx);
+ struct db_record *rec;
+ struct db_watched_subrec *subrec;
+ TDB_DATA subrec_value;
+ ssize_t num_watchers;
+
+ rec = talloc_zero(mem_ctx, struct db_record);
+ if (rec == NULL) {
+ return NULL;
+ }
+ subrec = talloc_zero(rec, struct db_watched_subrec);
+ if (subrec == NULL) {
+ TALLOC_FREE(rec);
+ return NULL;
+ }
+ rec->private_data = subrec;
+
+ subrec->subrec = dbwrap_fetch_locked(ctx->backend, subrec, key);
+ if (subrec->subrec == NULL) {
+ TALLOC_FREE(rec);
+ return NULL;
+ }
+
+ rec->db = db;
+ rec->key = dbwrap_record_get_key(subrec->subrec);
+ rec->store = dbwrap_watched_store;
+ rec->delete_rec = dbwrap_watched_delete;
+
+ subrec_value = dbwrap_record_get_value(subrec->subrec);
+
+ num_watchers = dbwrap_watched_parse(subrec_value, NULL, 0, NULL, NULL);
+ if (num_watchers == -1) {
+ /* Fresh or invalid record */
+ rec->value = (TDB_DATA) {};
+ return rec;
+ }
+
+ subrec->watchers = talloc_array(subrec, struct server_id,
+ num_watchers);
+ if (subrec->watchers == NULL) {
+ TALLOC_FREE(rec);
+ return NULL;
+ }
+
+ dbwrap_watched_parse(subrec_value, subrec->watchers, num_watchers,
+ &subrec->deleted, &rec->value);
+
+ return rec;
+}
+
+static void dbwrap_watched_wakeup(struct db_record *rec,
+ struct db_watched_subrec *subrec)
+{
+ struct db_context *db = dbwrap_record_get_db(rec);
+ struct db_watched_ctx *ctx = talloc_get_type_abort(
+ db->private_data, struct db_watched_ctx);
+ size_t i, num_watchers;
+ size_t db_id_len = dbwrap_db_id(db, NULL, 0);
+ uint8_t db_id[db_id_len];
+ uint8_t len_buf[4];
+ struct iovec iov[3];
+
+ SIVAL(len_buf, 0, db_id_len);
+
+ iov[0] = (struct iovec) { .iov_base = len_buf, .iov_len = 4 };
+ iov[1] = (struct iovec) { .iov_base = db_id, .iov_len = db_id_len };
+ iov[2] = (struct iovec) { .iov_base = rec->key.dptr,
+ .iov_len = rec->key.dsize };
+
+ dbwrap_db_id(db, db_id, db_id_len);
+
+ num_watchers = talloc_array_length(subrec->watchers);
+
+ i = 0;
+
+ while (i < num_watchers) {
+ NTSTATUS status;
+ struct server_id_buf tmp;
+
+ DBG_DEBUG("Alerting %s\n",
+ server_id_str_buf(subrec->watchers[i], &tmp));
+
+ status = messaging_send_iov(ctx->msg, subrec->watchers[i],
+ MSG_DBWRAP_MODIFIED,
+ iov, ARRAY_SIZE(iov), NULL, 0);
+ if (!NT_STATUS_IS_OK(status)) {
+ DBG_DEBUG("messaging_send_iov to %s failed: %s\n",
+ server_id_str_buf(subrec->watchers[i], &tmp),
+ nt_errstr(status));
+ }
+ if (NT_STATUS_EQUAL(status, NT_STATUS_OBJECT_NAME_NOT_FOUND)) {
+ subrec->watchers[i] = subrec->watchers[num_watchers-1];
+ num_watchers -= 1;
+
+ subrec->watchers = talloc_realloc(
+ subrec, subrec->watchers, struct server_id,
+ num_watchers);
+ continue;
+ }
+
+ i += 1;
+ }
+}
+
+static NTSTATUS dbwrap_watched_save(struct db_watched_subrec *subrec,
+ TDB_DATA data, int flag)
+{
+ size_t num_watchers;
+ ssize_t len;
+ uint8_t *buf;
+ NTSTATUS status;
+
+ num_watchers = talloc_array_length(subrec->watchers);
+
+ len = dbwrap_watched_unparse(subrec->watchers, num_watchers,
+ subrec->deleted, data, NULL, 0);
+ if (len == -1) {
+ return NT_STATUS_INSUFFICIENT_RESOURCES;
+ }
+
+ buf = talloc_array(subrec, uint8_t, len);
+ if (buf == NULL) {
+ return NT_STATUS_NO_MEMORY;
+ }
+
+ dbwrap_watched_unparse(subrec->watchers, num_watchers,
+ subrec->deleted, data, buf, len);
+
+ status = dbwrap_record_store(
+ subrec->subrec, (TDB_DATA) { .dptr = buf, .dsize = len },
+ flag);
+
+ TALLOC_FREE(buf);
+
+ return status;
+}
+
+static NTSTATUS dbwrap_watched_store(struct db_record *rec, TDB_DATA data,
+ int flag)
+{
+ struct db_watched_subrec *subrec = talloc_get_type_abort(
+ rec->private_data, struct db_watched_subrec);
+
+ dbwrap_watched_wakeup(rec, subrec);
+
+ subrec->deleted = false;
+
+ return dbwrap_watched_save(subrec, data, flag);
+
+}
+
+static NTSTATUS dbwrap_watched_delete(struct db_record *rec)
+{
+ struct db_watched_subrec *subrec = talloc_get_type_abort(
+ rec->private_data, struct db_watched_subrec);
+ size_t num_watchers;
+
+ dbwrap_watched_wakeup(rec, subrec);
+
+ num_watchers = talloc_array_length(subrec->watchers);
+ if (num_watchers == 0) {
+ return dbwrap_record_delete(subrec->subrec);
+ }
+
+ subrec->deleted = true;
+
+ return dbwrap_watched_save(subrec, (TDB_DATA) {0}, 0);
+}
+
+struct dbwrap_watched_traverse_state {
+ int (*fn)(struct db_record *rec, void *private_data);
+ void *private_data;
+};
+
+static int dbwrap_watched_traverse_fn(struct db_record *rec,
+ void *private_data)
+{
+ struct dbwrap_watched_traverse_state *state = private_data;
+ ssize_t num_watchers;
+ struct db_record prec = *rec;
+ bool deleted;
+
--
Samba Shared Repository
More information about the samba-cvs
mailing list