[SCM] Samba Shared Repository - branch v4-18-test updated
Jule Anger
janger at samba.org
Mon Jul 31 09:12:02 UTC 2023
The branch, v4-18-test has been updated
via 35573c04465 mdssvc: fix returning file modification date for older Mac releases
via f4a3373585f mdssvc: fix date marshalling
via aa4db8a3cf8 mdssvc: prepare for returning timestamps with sub-seconds granularity
via 590c91640bf mdssvc: reduce pagesize to 50
via 3036f3cb96f tests/mdssvc: match hits:total:value to be the actual amount of entries in hits
via 39c2b556456 mdssvc: fix enforcement of "elasticsearch:max results"
via 60fdb3adbee mdssvc: add and use SL_PAGESIZE
via 7d5e9f5fc23 mdssvc: fix long running backend queries
via 160b7dc05dc mdssvc: set query state for continued queries to SLQ_STATE_RUNNING
from c1c2a0ec80d s3:winbindd: Fix double close(fd)
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-18-test
- Log -----------------------------------------------------------------
commit 35573c04465820f78642a22231a4040ca855e1e1
Author: Ralph Boehme <slow at samba.org>
Date: Thu May 18 18:12:19 2023 +0200
mdssvc: fix returning file modification date for older Mac releases
Mac 10.10 uses kMDItemContentModificationDate instead of
kMDItemFSContentChangeDate.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
Autobuild-User(master): Jeremy Allison <jra at samba.org>
Autobuild-Date(master): Wed Jul 26 23:42:44 UTC 2023 on atb-devel-224
(cherry picked from commit c2e83ebe726b7bc42b329198214c784936f19888)
Autobuild-User(v4-18-test): Jule Anger <janger at samba.org>
Autobuild-Date(v4-18-test): Mon Jul 31 09:11:21 UTC 2023 on atb-devel-224
commit f4a3373585f0ba0b87763d35bb90054a983109c6
Author: Ralph Boehme <slow at samba.org>
Date: Wed May 17 16:38:39 2023 +0200
mdssvc: fix date marshalling
Did this ever work? Possible just copied over from Netatalk and was always
broken... The Mac client expects the timevalue as seconds relative to
2001-01-01 00:00:00 UTC, packed as IEEE float.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 620ca1e68d02be45a94aa41217a141d211fceb1f)
commit aa4db8a3cf82c278380600fa6ce293ac7a7379c3
Author: Ralph Boehme <slow at samba.org>
Date: Wed May 17 16:37:36 2023 +0200
mdssvc: prepare for returning timestamps with sub-seconds granularity
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15427
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 9dc66fecf7c1743d264c5c4f8978b77bab75ed86)
commit 590c91640bfec9909a543eafeef8f52d5a42a89e
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 20 17:27:20 2023 +0200
mdssvc: reduce pagesize to 50
Lastest macOS queries additional file metadata per search result, which causes
the mashalled paged result set including metadata to exceed the 64 KB result
fragment buffer.
Lacking fragementation support in mdssvc (it's supported by the protocol), for
now just reduce the maximum number of results per search page.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 724a0518c901589fe1171d94648391832e056f4d)
commit 3036f3cb96f1923949c7c73d2d3e9273539d0978
Author: Ralph Boehme <slow at samba.org>
Date: Fri Apr 21 07:07:13 2023 +0200
tests/mdssvc: match hits:total:value to be the actual amount of entries in hits
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 7f5e4edf64f7e4175f652bf8762d4edc110ad6b1)
commit 39c2b556456bd0a348148df72cafdecf8e3c4a6f
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 20 17:58:38 2023 +0200
mdssvc: fix enforcement of "elasticsearch:max results"
This wasn't enforced at all thus a query would return all available matches
without limit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit d8fa5c8e2a1794ea8dc663485315ebd9401b2628)
commit 60fdb3adbee0130ecc993741688979d0a1104ee5
Author: Ralph Boehme <slow at samba.org>
Date: Thu Apr 20 17:24:30 2023 +0200
mdssvc: add and use SL_PAGESIZE
SL_PAGESIZE is the number of entries we want to process per paged search result
set. This is different from MAX_SL_RESULTS which ought to be a default maximum
value for total number of results returned for a search query.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 086c2602d074d4dc0d44f5534857e5f59a8690b2)
commit 7d5e9f5fc23d5be2a6b38ee04548378ff8a25ea0
Author: Ralph Boehme <slow at samba.org>
Date: Thu Mar 23 16:39:11 2023 +0100
mdssvc: fix long running backend queries
If a query is still running in the backend and we have no results yet, returning
0 triggers a search termination by the client in latest macOS releases. macOS
returns 0x23 in this case.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 925fefae20e52a3c89a56bdd0cd5b98cc951db5f)
commit 160b7dc05dc666dc642bbb17d7a338db17a49168
Author: Ralph Boehme <slow at samba.org>
Date: Wed Apr 19 14:38:45 2023 +0200
mdssvc: set query state for continued queries to SLQ_STATE_RUNNING
SLQ_STATE_RESULTS implies that there are already results attached to the slq
which is not the case. Instead the backend will start processing from where it
left off when it hits the maximum result limit and had set the state to
SLQ_STATE_FULL.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15342
Signed-off-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit 4149ef97e5906604be1587622f390f121db183e2)
-----------------------------------------------------------------------
Summary of changes:
python/samba/tests/blackbox/mdsearch.py | 2 +-
python/samba/tests/dcerpc/mdssvc.py | 6 +++---
source3/rpc_server/mdssvc/marshalling.c | 29 +++++++++++++++++++++++------
source3/rpc_server/mdssvc/mdssvc.c | 21 +++++++++++++++++----
source3/rpc_server/mdssvc/mdssvc.h | 1 +
source3/rpc_server/mdssvc/mdssvc_es.c | 8 ++++----
6 files changed, 49 insertions(+), 18 deletions(-)
Changeset truncated at 500 lines:
diff --git a/python/samba/tests/blackbox/mdsearch.py b/python/samba/tests/blackbox/mdsearch.py
index c8e75661f15..46af6f94624 100644
--- a/python/samba/tests/blackbox/mdsearch.py
+++ b/python/samba/tests/blackbox/mdsearch.py
@@ -100,7 +100,7 @@ class MdfindBlackboxTests(BlackboxTestCase):
config = os.environ["SMB_CONF_PATH"]
json_in = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\""
diff --git a/python/samba/tests/dcerpc/mdssvc.py b/python/samba/tests/dcerpc/mdssvc.py
index 5002e5d26d6..7d09a555936 100644
--- a/python/samba/tests/dcerpc/mdssvc.py
+++ b/python/samba/tests/dcerpc/mdssvc.py
@@ -125,7 +125,7 @@ class MdssvcTests(RpcInterfaceTestCase):
def test_mdscli_search(self):
exp_json_query = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(samba*) AND path.real.fulltext:\"%BASEPATH%\""
@@ -157,7 +157,7 @@ class MdssvcTests(RpcInterfaceTestCase):
r'kMDItemFSName=="x\\x"'
)
exp_json_query = r'''{
- "from": 0, "size": 100, "_source": ["path.real"],
+ "from": 0, "size": 50, "_source": ["path.real"],
"query": {
"query_string": {
"query": "(file.filename:x\\+x OR file.filename:x\\*x OR file.filename:x=x OR file.filename:x'x OR file.filename:x\\?x OR file.filename:x\\ x OR file.filename:x\\(x OR file.filename:x\\\"x OR file.filename:x\\\\x) AND path.real.fulltext:\"%BASEPATH%\""
@@ -166,7 +166,7 @@ class MdssvcTests(RpcInterfaceTestCase):
}'''
fake_json_response = r'''{
"hits" : {
- "total" : {"value" : 2},
+ "total" : {"value" : 9},
"hits" : [
{"_source" : {"path" : {"real" : "%BASEPATH%/x+x"}}},
{"_source" : {"path" : {"real" : "%BASEPATH%/x*x"}}},
diff --git a/source3/rpc_server/mdssvc/marshalling.c b/source3/rpc_server/mdssvc/marshalling.c
index 5f866d7fb6e..b3e16d9c18f 100644
--- a/source3/rpc_server/mdssvc/marshalling.c
+++ b/source3/rpc_server/mdssvc/marshalling.c
@@ -43,8 +43,8 @@
* RPC data marshalling and unmarshalling
******************************************************************************/
-/* Spotlight epoch is UNIX epoch minus SPOTLIGHT_TIME_DELTA */
-#define SPOTLIGHT_TIME_DELTA 280878921600ULL
+/* Spotlight epoch is 1.1.2001 00:00 UTC */
+#define SPOTLIGHT_TIME_DELTA 978307200 /* Diff from UNIX epoch to Spotlight epoch */
#define SQ_TYPE_NULL 0x0000
#define SQ_TYPE_COMPLEX 0x0200
@@ -253,6 +253,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi
{
uint64_t data;
uint64_t tag;
+ union {
+ double d;
+ uint64_t w;
+ } ieee_fp_union;
tag = sl_pack_tag(SQ_TYPE_DATE, 2, 1);
offset = sl_push_uint64_val(buf, offset, bufsize, tag);
@@ -260,7 +264,10 @@ static ssize_t sl_pack_date(sl_time_t t, char *buf, ssize_t offset, size_t bufsi
return -1;
}
- data = (t.tv_sec + SPOTLIGHT_TIME_DELTA) << 24;
+ ieee_fp_union.d = (double)(t.tv_sec - SPOTLIGHT_TIME_DELTA);
+ ieee_fp_union.d += (double)t.tv_usec / 1000000;
+
+ data = ieee_fp_union.w;
offset = sl_push_uint64_val(buf, offset, bufsize, data);
if (offset == -1) {
return -1;
@@ -723,6 +730,11 @@ static int sl_unpack_date(DALLOC_CTX *query,
int i, result;
struct sl_tag tag;
uint64_t query_data64;
+ union {
+ double d;
+ uint64_t w;
+ } ieee_fp_union;
+ double fraction;
sl_time_t t;
offset = sl_unpack_tag(buf, offset, bufsize, encoding, &tag);
@@ -735,9 +747,14 @@ static int sl_unpack_date(DALLOC_CTX *query,
if (offset == -1) {
return -1;
}
- query_data64 = query_data64 >> 24;
- t.tv_sec = query_data64 - SPOTLIGHT_TIME_DELTA;
- t.tv_usec = 0;
+ ieee_fp_union.w = query_data64;
+ fraction = ieee_fp_union.d - (uint64_t)ieee_fp_union.d;
+
+ t = (sl_time_t) {
+ .tv_sec = ieee_fp_union.d + SPOTLIGHT_TIME_DELTA,
+ .tv_usec = fraction * 1000000
+ };
+
result = dalloc_add_copy(query, &t, sl_time_t);
if (result != 0) {
return -1;
diff --git a/source3/rpc_server/mdssvc/mdssvc.c b/source3/rpc_server/mdssvc/mdssvc.c
index 22cc7abd153..a7d5d7afc02 100644
--- a/source3/rpc_server/mdssvc/mdssvc.c
+++ b/source3/rpc_server/mdssvc/mdssvc.c
@@ -188,8 +188,10 @@ static bool add_filemeta(struct mds_ctx *mds_ctx,
if (result != 0) {
return false;
}
- } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0) {
- sl_time.tv_sec = sp->st_ex_mtime.tv_sec;
+ } else if (strcmp(attribute, "kMDItemFSContentChangeDate") == 0 ||
+ strcmp(attribute, "kMDItemContentModificationDate") == 0)
+ {
+ sl_time = convert_timespec_to_timeval(sp->st_ex_mtime);
result = dalloc_add_copy(meta, &sl_time, sl_time_t);
if (result != 0) {
return false;
@@ -306,10 +308,21 @@ static bool create_result_handle(struct sl_query *slq)
static bool add_results(sl_array_t *array, struct sl_query *slq)
{
sl_filemeta_t *fm;
- uint64_t status = 0;
+ uint64_t status;
int result;
bool ok;
+ /*
+ * Taken from a network trace against a macOS SMB Spotlight server. If
+ * the first fetch-query-results has no results yet because the search
+ * is still running, macOS returns 0x23, otherwise 0x0.
+ */
+ if (slq->state >= SLQ_STATE_RESULTS ) {
+ status = 0;
+ } else {
+ status = 0x23;
+ }
+
/* FileMeta */
fm = dalloc_zero(array, sl_filemeta_t);
if (fm == NULL) {
@@ -1126,7 +1139,7 @@ static bool slrpc_fetch_query_results(struct mds_ctx *mds_ctx,
goto error;
}
if (slq->state == SLQ_STATE_FULL) {
- slq->state = SLQ_STATE_RESULTS;
+ slq->state = SLQ_STATE_RUNNING;
slq->mds_ctx->backend->search_cont(slq);
}
break;
diff --git a/source3/rpc_server/mdssvc/mdssvc.h b/source3/rpc_server/mdssvc/mdssvc.h
index 3b2ce250f1f..8aabf5b86e5 100644
--- a/source3/rpc_server/mdssvc/mdssvc.h
+++ b/source3/rpc_server/mdssvc/mdssvc.h
@@ -36,6 +36,7 @@
#define MAX_SL_FRAGMENT_SIZE 0xFFFFF
#define MAX_SL_RESULTS 100
+#define SL_PAGESIZE 50
#define MAX_SL_RUNTIME 30
#define MDS_TRACKER_ASYNC_TIMEOUT_MS 250
diff --git a/source3/rpc_server/mdssvc/mdssvc_es.c b/source3/rpc_server/mdssvc/mdssvc_es.c
index dafb42610fa..8460b48b80a 100644
--- a/source3/rpc_server/mdssvc/mdssvc_es.c
+++ b/source3/rpc_server/mdssvc/mdssvc_es.c
@@ -398,7 +398,7 @@ static bool mds_es_search(struct sl_query *slq)
.ev = mds_es_ctx->mdssvc_es_ctx->mdssvc_ctx->ev_ctx,
.mds_es_ctx = mds_es_ctx,
.slq = slq,
- .size = MAX_SL_RESULTS,
+ .size = SL_PAGESIZE,
};
/* 0 would mean no limit */
@@ -502,7 +502,7 @@ static void mds_es_search_done(struct tevent_req *subreq)
goto trigger;
}
- if (slq->query_results->num_results >= MAX_SL_RESULTS) {
+ if (slq->query_results->num_results >= SL_PAGESIZE) {
slq->state = SLQ_STATE_FULL;
goto trigger;
}
@@ -693,7 +693,7 @@ static void mds_es_search_http_send_done(struct tevent_req *subreq)
subreq = http_read_response_send(state,
state->ev,
state->s->mds_es_ctx->http_conn,
- MAX_SL_RESULTS * 8192);
+ SL_PAGESIZE * 8192);
if (tevent_req_nomem(subreq, req)) {
return;
}
@@ -800,7 +800,7 @@ static void mds_es_search_http_read_done(struct tevent_req *subreq)
}
DBG_DEBUG("Hits: %zu\n", hits);
- for (i = 0; i < hits; i++) {
+ for (i = 0; i < hits && s->from + i < s->max; i++) {
const char *path = NULL;
match = json_array_get(matches, i);
--
Samba Shared Repository
More information about the samba-cvs
mailing list