[SCM] Samba Shared Repository - branch master updated
Jeremy Allison
jra at samba.org
Wed Jul 26 23:43:02 UTC 2023
The branch, master has been updated
via c2e83ebe726 mdssvc: fix returning file modification date for older Mac releases
via 620ca1e68d0 mdssvc: fix date marshalling
via 9dc66fecf7c mdssvc: prepare for returning timestamps with sub-seconds granularity
via 724a0518c90 mdssvc: reduce pagesize to 50
via 7f5e4edf64f tests/mdssvc: match hits:total:value to be the actual amount of entries in hits
via d8fa5c8e2a1 mdssvc: fix enforcement of "elasticsearch:max results"
via 086c2602d07 mdssvc: add and use SL_PAGESIZE
via 925fefae20e mdssvc: fix long running backend queries
via 4149ef97e59 mdssvc: set query state for continued queries to SLQ_STATE_RUNNING
from e86e0da9de6 WHATSNEW: Add TLS cert reload feature
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit c2e83ebe726b7bc42b329198214c784936f19888
Author: Ralph Boehme <slow at samba.org>
Date: Thu May 18 18:12:19 2023 +0200
mdssvc: fix returning file modification date for older Mac releases
Mac 10.10 uses kMDItemContentModificationDate instead of
kMDItemFSContentChangeDate.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
Autobuild-User(master): Jeremy Allison <jra at samba.org>
Autobuild-Date(master): Wed Jul 26 23:42:44 UTC 2023 on atb-devel-224
commit 620ca1e68d02be45a94aa41217a141d211fceb1f
Author: Ralph Boehme <slow at samba.org>
Date: Wed May 17 16:38:39 2023 +0200
mdssvc: fix date marshalling
Did this ever work? Possible just copied over from Netatalk and was always
broken... The Mac client expects the timevalue as seconds relative to
2001-01-01 00:00:00 UTC, packed as IEEE float.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 9dc66fecf7c1743d264c5c4f8978b77bab75ed86
Author: Ralph Boehme <slow at samba.org>
Date: Wed May 17 16:37:36 2023 +0200
mdssvc: prepare for returning timestamps with sub-seconds granularity
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 724a0518c901589fe1171d94648391832e056f4d
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 20 17:27:20 2023 +0200
mdssvc: reduce pagesize to 50
Lastest macOS queries additional file metadata per search result, which causes
the mashalled paged result set including metadata to exceed the 64 KB result
fragment buffer.
Lacking fragementation support in mdssvc (it's supported by the protocol), for
now just reduce the maximum number of results per search page.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 7f5e4edf64f7e4175f652bf8762d4edc110ad6b1
Author: Ralph Boehme <slow at samba.org>
Date: Fri Apr 21 07:07:13 2023 +0200
tests/mdssvc: match hits:total:value to be the actual amount of entries in hits
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit d8fa5c8e2a1794ea8dc663485315ebd9401b2628
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 20 17:58:38 2023 +0200
mdssvc: fix enforcement of "elasticsearch:max results"
This wasn't enforced at all thus a query would return all available matches
without limit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 086c2602d074d4dc0d44f5534857e5f59a8690b2
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 20 17:24:30 2023 +0200
mdssvc: add and use SL_PAGESIZE
SL_PAGESIZE is the number of entries we want to process per paged search result
set. This is different from MAX_SL_RESULTS which ought to be a default maximum
value for total number of results returned for a search query.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 925fefae20e52a3c89a56bdd0cd5b98cc951db5f
Author: Ralph Boehme <slow at samba.org>
Date: Thu Mar 23 16:39:11 2023 +0100
mdssvc: fix long running backend queries
If a query is still running in the backend and we have no results yet, returning
0 triggers a search termination by the client in latest macOS releases. macOS
returns 0x23 in this case.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
commit 4149ef97e5906604be1587622f390f121db183e2
Author: Ralph Boehme <slow at samba.org>
Date: Wed Apr 19 14:38:45 2023 +0200
mdssvc: set query state for continued queries to SLQ_STATE_RUNNING
SLQ_STATE_RESULTS implies that there are already results attached to the slq
which is not the case. Instead the backend will start processing from where it
left off when it hits the maximum result limit and had set the state to
SLQ_STATE_FULL.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
-----------------------------------------------------------------------
Summary of changes:
python/samba/tests/blackbox/mdsearch.py | 2 +-
python/samba/tests/dcerpc/mdssvc.py | 6 +++---
source3/rpc_server/mdssvc/marshalling.c | 29 +++++++++++++++++++++++------
source3/rpc_server/mdssvc/mdssvc.c | 21 +++++++++++++++++----
source3/rpc_server/mdssvc/mdssvc.h | 1 +
source3/rpc_server/mdssvc/mdssvc_es.c | 8 ++++----
6 files changed, 49 insertions(+), 18 deletions(-)
Changeset truncated at 500 lines:
diff --git a/python/samba/tests/blackbox/mdsearch.py b/python/samba/tests/blackbox/mdsearch.py
index c8e75661f15..46af6f94624 100644
--- a/python/samba/tests/blackbox/mdsearch.py
+++ b/python/samba/tests/blackbox/mdsearch.py
@@ -100,7 +100,7 @@ class MdfindBlackboxTests(BlackboxTestCase):
config = os.environ["SMB_CONF_PATH"]
json_in = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\""
diff --git a/python/samba/tests/dcerpc/mdssvc.py b/python/samba/tests/dcerpc/mdssvc.py
index 5002e5d26d6..7d09a555936 100644
--- a/python/samba/tests/dcerpc/mdssvc.py
+++ b/python/samba/tests/dcerpc/mdssvc.py
@@ -125,7 +125,7 @@ class MdssvcTests(RpcInterfaceTestCase):
def test_mdscli_search(self):
exp_json_query = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\""
@@ -157,7 +157,7 @@ class MdssvcTests(RpcInterfaceTestCase):
r'kMDItemFSName=="x\\x"'
)
exp_json_query = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(file.filename:x\\+x OR file.filename:x\\*x OR file.filename:x=x OR file.filename:x'x OR file.filename:x\\?x OR file.filename:x\\ x OR file.filename:x\\(x OR file.filename:x\\\"x OR file.filename:x\\\\x) AND path.real.fulltext:\"%BASEPATH%\""
@@ -166,7 +166,7 @@ class MdssvcTests(RpcInterfaceTestCase):
}'''
fake_json_response = r'''{
"hits" : {
- "total" : {"value" : 2},
+ "total" : {"value" : 9},
"hits" : [
{"_source" : {"path" : {"real" : "%BASEPATH%/x+x"}}},
{"_source" : {"path" : {"real" : "%BASEPATH%/x*x"}}},
diff --git a/source3/rpc_server/mdssvc/marshalling.c b/source3/rpc_server/mdssvc/marshalling.c
index 3bbd5889aaa..0e323fa979e 100644
--- a/source3/rpc_server/mdssvc/marshalling.c
+++ b/source3/rpc_server/mdssvc/marshalling.c
@@ -43,8 +43,8 @@
* RPC data marshalling and unmarshalling
******************************************************************************/
-/* Spotlight epoch is UNIX epoch minus SPOTLIGHT_TIME_DELTA */
-#define SPOTLIGHT_TIME_DELTA 280878921600ULL
+/* Spotlight epoch is 1.1.2001 00:00 UTC */
+#define SPOTLIGHT_TIME_DELTA 978307200 /* Diff from UNIX epoch to Spotlight epoch */
#define SQ_TYPE_NULL 0x0000
#define SQ_TYPE_COMPLEX 0x0200
@@ -253,6 +253,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi
{
uint64_t data;
uint64_t tag;
+ union {
+ double d;
+ uint64_t w;
+ } ieee_fp_union;
tag = sl_pack_tag(SQ_TYPE_DATE, 2, 1);
offset = sl_push_uint64_val(buf, offset, bufsize, tag);
@@ -260,7 +264,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi
return -1;
}
- data = (t.tv_sec + SPOTLIGHT_TIME_DELTA) << 24;
+ ieee_fp_union.d = (double)(t.tv_sec - SPOTLIGHT_TIME_DELTA);
+ ieee_fp_union.d += (double)t.tv_usec / 1000000;
+
+ data = ieee_fp_union.w;
offset = sl_push_uint64_val(buf, offset, bufsize, data);
if (offset == -1) {
return -1;
@@ -723,6 +730,11 @@ static int sl_unpack_date(DALLOC_CTX *query,
int i, result;
struct sl_tag tag;
uint64_t query_data64;
+ union {
+ double d;
+ uint64_t w;
+ } ieee_fp_union;
+ double fraction;
sl_time_t t;
offset = sl_unpack_tag(buf, offset, bufsize, encoding, &tag);
@@ -735,9 +747,14 @@ static int sl_unpack_date(DALLOC_CTX *query,
if (offset == -1) {
return -1;
}
- query_data64 = query_data64 >> 24;
- t.tv_sec = query_data64 - SPOTLIGHT_TIME_DELTA;
- t.tv_usec = 0;
+ ieee_fp_union.w = query_data64;
+ fraction = ieee_fp_union.d - (uint64_t)ieee_fp_union.d;
+
+ t = (sl_time_t) {
+ .tv_sec = ieee_fp_union.d + SPOTLIGHT_TIME_DELTA,
+ .tv_usec = fraction * 1000000
+ };
+
result = dalloc_add_copy(query, &t, sl_time_t);
if (result != 0) {
return -1;
diff --git a/source3/rpc_server/mdssvc/mdssvc.c b/source3/rpc_server/mdssvc/mdssvc.c
index 335b267c0e5..abeaac8263c 100644
--- a/source3/rpc_server/mdssvc/mdssvc.c
+++ b/source3/rpc_server/mdssvc/mdssvc.c
@@ -188,8 +188,10 @@ static bool add_filemeta(struct mds_ctx *mds_ctx,
if (result != 0) {
return false;
}
- } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0) {
- sl_time.tv_sec = sp->st_ex_mtime.tv_sec;
+ } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0 ||
+ strcmp(attribute, "kMDItemContentModificationDate") == 0)
+ {
+ sl_time = convert_timespec_to_timeval(sp->st_ex_mtime);
result = dalloc_add_copy(meta, &sl_time, sl_time_t);
if (result != 0) {
return false;
@@ -306,10 +308,21 @@ static bool create_result_handle(struct sl_query *slq)
static bool add_results(sl_array_t *array, struct sl_query *slq)
{
sl_filemeta_t *fm;
- uint64_t status = 0;
+ uint64_t status;
int result;
bool ok;
+ /*
+ * Taken from a network trace against a macOS SMB Spotlight server. If
+ * the first fetch-query-results has no results yet because the search
+ * is still running, macOS returns 0x23, otherwise 0x0.
+ */
+ if (slq->state >= SLQ_STATE_RESULTS ) {
+ status = 0;
+ } else {
+ status = 0x23;
+ }
+
/* FileMeta */
fm = dalloc_zero(array, sl_filemeta_t);
if (fm == NULL) {
@@ -1126,7 +1139,7 @@ static bool slrpc_fetch_query_results(struct mds_ctx *mds_ctx,
goto error;
}
if (slq->state == SLQ_STATE_FULL) {
- slq->state = SLQ_STATE_RESULTS;
+ slq->state = SLQ_STATE_RUNNING;
slq->mds_ctx->backend->search_cont(slq);
}
break;
diff --git a/source3/rpc_server/mdssvc/mdssvc.h b/source3/rpc_server/mdssvc/mdssvc.h
index 5836caff740..6d4e684ae8d 100644
--- a/source3/rpc_server/mdssvc/mdssvc.h
+++ b/source3/rpc_server/mdssvc/mdssvc.h
@@ -35,6 +35,7 @@
#undef FALSE
#define MAX_SL_RESULTS 100
+#define SL_PAGESIZE 50
#define MAX_SL_RUNTIME 30
#define MDS_TRACKER_ASYNC_TIMEOUT_MS 250
diff --git a/source3/rpc_server/mdssvc/mdssvc_es.c b/source3/rpc_server/mdssvc/mdssvc_es.c
index dafb42610fa..8460b48b80a 100644
--- a/source3/rpc_server/mdssvc/mdssvc_es.c
+++ b/source3/rpc_server/mdssvc/mdssvc_es.c
@@ -398,7 +398,7 @@ static bool mds_es_search(struct sl_query *slq)
.ev = mds_es_ctx->mdssvc_es_ctx->mdssvc_ctx->ev_ctx,
.mds_es_ctx = mds_es_ctx,
.slq = slq,
- .size = MAX_SL_RESULTS,
+ .size = SL_PAGESIZE,
};
/* 0 would mean no limit */
@@ -502,7 +502,7 @@ static void mds_es_search_done(struct tevent_req *subreq)
goto trigger;
}
- if (slq->query_results->num_results >= MAX_SL_RESULTS) {
+ if (slq->query_results->num_results >= SL_PAGESIZE) {
slq->state = SLQ_STATE_FULL;
goto trigger;
}
@@ -693,7 +693,7 @@ static void mds_es_search_http_send_done(struct tevent_req *subreq)
subreq = http_read_response_send(state,
state->ev,
state->s->mds_es_ctx->http_conn,
- MAX_SL_RESULTS * 8192);
+ SL_PAGESIZE * 8192);
if (tevent_req_nomem(subreq, req)) {
return;
}
@@ -800,7 +800,7 @@ static void mds_es_search_http_read_done(struct tevent_req *subreq)
}
DBG_DEBUG("Hits: %zu\n", hits);
- for (i = 0; i < hits; i++) {
+ for (i = 0; i < hits && s->from + i < s->max; i++) {
const char *path = NULL;
match = json_array_get(matches, i);
--
Samba Shared Repository
More information about the samba-cvs
mailing list