WIP WSP patches

Noel Power nopower at suse.com
Fri Jan 6 13:56:29 UTC 2017

Hello List

Another gentle reminder about this stuff (and a little update)

 I finally got around to adding support for 32/64 bit msg negotiation so
that the client/server can operate in either mode. Both the WSP server
and wspsearch cli client now use a version id based on the build time
arch (and thus use either a 32 or 64 bit version id as appropriate)
which they send on connection requests. I cleaned up the client code
quite a bit (and tweaked some of the Row handling code in the server
too) I am sure there is plenty of room for further improvement.

If you want to try it out I've created a repository [1] that you can use
with a suse system (leap or sle)[2]  that you can download rpms from [2]

This repo contains 2 packages of interest, the first obviously is samba
(which has the WSP patches enabled) and secondly a package called
system-tracker, system-tracker sets up all the systemd related foo
(including creating a dedicated user called 'tracker') based on what's
described at https://wiki.samba.org/index.php/Spotlight#Setup. So with a
desktop suse system (e.g. Leap42.2) just a few steps are needed to get
it going

   1. install system-tracker & samba packages from [1]
   2. start tracker as described
   3. configure tracker as described

       NOTE: remember to use the the username 'tracker' instead of 'mdssvc'
   4. configure smb.conf

        rpc_daemon:wspd = fork

        path = /some/path
        wsp = yes
    5. restart smbd and you should be good to go

Once again, please let me know what I can do to move this along......
I've attached patches that correspond to branch WSP-WIP-CLIENT-ONLY &
WSP-WIP-NO_RAWPIPE from my github samba repo


[1] http://download.opensuse.org/repositories/home:/npower:/WSP

[2] although the build service supports creating packages for other
distros like Fedora etc I haven't enabled those because I lack any real
packaging skills, I barely got it working for the suse stuff :-) I'd
happily add any patches to support other distros if anyone cares to
help. Otherwise I will in time try to get around to enabling some other
vendor specific repos

On 25/11/16 14:27, Noel Power wrote:
> Hi,
> I've updated my github repo
> On 23/09/16 09:17, Noel Power wrote:
>> On 23/09/16 05:00, Ralph Böhme wrote: 
> [...]
>>> nevermind, looks like this is because we don't do a pipe wait before
>>> the pipe open. Adding support for this to libsmb atm... 
>> interesting, seems like a nice catch, I never encountered that, I have
>> only run the client against a 2k12 server and the pipe was probably
>> already setup from previous testing.
> I didn't see a patch come through and since I still need all the
> practise I can get to understand this samba/SMB stuff I had a crack at
> it myself (but I'll quite happily replace my patches with whatever is
> else available if needed)
>> Have to say though I only added the client for testing purposes
>> really, it needs some love I think to make it more generally useful.
>> E.g. to make binary queries more flexible, maybe limit results
>> returned and I was thinking maybe some optional shell interface to
>> help create more complex queries.... but that's all just thoughts
>> right now.
> Well thoughts sometimes turn into reality so I've updated the branch
> WSP-WIP-CLIENT-ONLY (and the others) with some new client & server
> functionality
> Firstly the wspsearch tool has some new options
>   o --limit=number which allows you to limit the search results
> returned, the following example will limit the results returned to 500,
> note the default value for this option is 0 (meaning unlimited)
> wspsearch -U$(USER)%$(PASSWD) --kind=Picture
> --limit=500 //$(SERVER)/$(SHARE)
>   o --query=$QUERY where $QUERY is something more or less resembling an
> AQS (advanced query syntax) query. I was looking for a way to specify
> more complex queries and came across
> https://msdn.microsoft.com/en-us/library/windows/desktop/bb266512(v=vs.85).aspx#examples)
> I decided to implement the syntax described there (not slavishly but
> lets say a subset) This at least allows dynamic and expressive queries
> to be built (great for testing the server/protocol and useful for actual
> searches too!!)
> Probably best to describe what is supported by some simple examples;
> The basic syntax is supported e.g. a query is built of a sequence of
> queries connected by AND, OR and NOT where the query elements are
> essentially restrictions defined by a property. There are some
> limitations on the operators supported[1] and additionally some things
> like enumerated ranges are not supported at all, range values and not
> delimited as specified [2]. Some special cases that you see in the UI
> are exceptions [3] which are handled more or less as keywords
> Examples
> --------
> The following are all exactly the same query just expressed using
> different variations of the syntax
>  wspsearch -Ufoo%bar --query='SALL:($<p403 OR $<p404) AND
> System.Kind:picture AND Scope:"FILE://somemachine/someshare" AND
> System.Size:10241-102401' //somemachine
>  wspsearch -Ufoo%bar --query='ALL:$<p403 OR ALL:$<p404 AND
> System.Kind:picture AND Scope:"FILE://somemachine/someshare" AND
> System.Size:>=10241 AND System.Size:<102401' //somemachine
>  wspsearch -Ufoo%bar --query='ALL:$<p403 OR ALL:$<p404 AND
> System.Kind:picture AND Scope:"FILE://somemachine/someshare" AND
> System.Size:small' //somemachine
> The queries above by default select the property System.ItemUrl as the
> one and only column returned, the query parameter however accepts a
> variation to the AQS like syntax to allow arbitrary columns to be
> selected e.g.
> ./bin/wspsearch -Ufoo%bar --query='SELECT System.ItemName,
> System.ItemURL, System.Size  WHERE ALL:$<p403 OR ALL:$<p404 AND
> System.Kind:picture AND Scope:"FILE://somemachine/someshare" AND
> System.Size:small' //somemachine
> Secondly the server implementation has been modified a little too,
> besides some shuffling around of code to share between the cmdline tools
> wsp-to-tracker, wspsearch and the server implementation and some
> cleaning up of the code (more to do there :-)) support for some new
> smb.conf options has been added
>   wsp result limit (G) like the client this limits the number of results
> that the server processes, currently the default value is 0 meaning
> unlimited results returned
>   wsp (S) This allows the wsp searching to be disabled on a per share
> basis. Behaviour here is a little odd (but matches windows) on a machine
> with indexing enabled where a share was previously indexed (but has the
> indexing turned off (but index service still runs)) it takes quite a
> while even up to 30mins or more for the client to switch over to
> searching using SMB commands for that share, in the time there can be 0
> results returned for searches that otherwise would return results
> (strange but true).
> Noel
> ps once again, please let me know what I can do or change to help this along
> [1] supported operators
>     -------------------
>     =    Equals
>     !=   Not Equals
>     >    Greater than
>     <    Less than
>     >=   Greater than or equals
>     <=   Less than or equals
>     $=   equals
>     $<   starts with
> Note: The binary CPMCREATEQUERY message allow many many different
> restriction types to be used, the query syntax described above only
> produces 2 types of restrictions either RTPROPERTY or RTCONTENT. Which
> type of restriction is created is determined by the operators above
> where using the last 2 will result in a RTCONTENT restriction being used
> or RTPROPERTY otherwise.
> [2] ranges are specified as value-value instead of value..value (seems
> my flex/bison skills are not good enough and couldn't get that to work
> yet, I'll keep trying)
> [3] The UI has shortcut ranges (presumably the enumerated ranges
> mentioned earlier) providingdate ranges like 'today', 'tomorrow',
> 'lastweek' etc. and similarly sizes like "empty, tiny, small, large..."
> These are supported (but implemented as keywords)

-------------- next part --------------
A non-text attachment was scrubbed...
Type: text/x-diff
Size: 437391 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170106/d70d91b4/WSP-WIP-CLIENT-ONLY-0001.diff>
-------------- next part --------------
A non-text attachment was scrubbed...
Type: text/x-diff
Size: 757875 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20170106/d70d91b4/WSP-WIP-NO_RAWPIPE-0001.diff>

More information about the samba-technical mailing list