Experimental CONTAINS filter operators

[irevoire]

Feature name	Enable when launching	Enable at runtime	Description
Enable a new CONTAINS filter operators	./meilisearch --experimental-contains-filter	curl \ -X PATCH 'http://localhost:7700/experimental-features/' \ -H 'Content-Type: application/json' \ --data-binary '{ "containsFilter": true }'	Create a new route under `/logs/stream` that streams logs to the client

Meilisearch v1.10.0 (released on August 26, 2024) introduces a new CONTAINS filter operator.

Experimental feature abstract

Enabling the experimental feature will make two new operators available, the CONTAINS and STARTS WITH ones, while filtering on strings.
The CONTAINS one is similar to the SQL LIKE operator used in that way %venture% and the STARTS WITH one is like using vent%.

How to use the feature?

1. Enable the feature through the CLI parameter --experimental-contains-filter or call the PATCH /experimental-features passing "containsFilter": true.
2. Call any route accepting a filter like the /indexes/movies/search route with your parameters:

curl \
	-X POST http://localhost:7700/indexes/movies/search \
	-H 'Content-Type: application/json' \
	--data-binary '{
	    "q": "super hero",
            "filter": "synopsis NOT CONTAINS spider"
    }'

💡 More information about CONTAINS on its dedicated usage page.

Why is this feature not stable yet?

The aspect we are most concerned about with this feature is search performance.

Indeed, it is possible that CONTAINS perform poorly, given your dataset. The time to find the documents matching a query with these operators increases with the total number of bytes of all the facet values of all the documents in an index.

We would like to hear your feedback on this feature, and we would also like your input on the following questions to help us decide on a go-no-go decision for the stabilization of that feature in the future:

• Have you considered using the q parameter to solve your need for a CONTAINS operator, and if not, why?
• Have you considered using the search for facet values feature to solve your need for the CONTAINS operator, and if not, why?

🗣️ You are welcome to give feedback or ask any question on its usage; we are eager to collect feedback on the feature

[honhimW]

Hello,

I’ve been using the CONTAINS filter functionality for some time, as well as the new Federation feature in the current RC version (v1.10.0-rc.1). Personally, I would like to have a feature similar to the following:

Performing a search on the same index for multiple content items and aggregating the returned results (with optional AND or OR logic):

A fake api as follow:

POST /indexes/{indexUid}/multi-search
{
  "queries": [
    {
      "q": "hello",
        "attributesToSearchOn": ["header"]
	"filter": "id < 100"
    },
    {
      "q": "world",
      "attributesToSearchOn": ["footer"]
      "filter": "id > 0"
    },
  ],
  "offset": 0,
  "limit": 20
}

The key here is to simultaneously search on different attributes for different content(q). While CONTAINS is useful in some cases, it can sometimes lead to significant performance issues, and certain search-related features (such as TypoTolerance, tagging(preTag/postTag), and RankingScore) are not working on the filter.

By the way, I’m unsure whether I should ask this question in this discussion. Sorry if i shouldn't.

[honhimW]

Yes, with current 'federation' feature indeed merge result together.
But it dose not deduplicate the result set.

[dureuill]

But it dose not deduplicate the result set.

What do you mean by that?

With federated search, the same document cannot be returned twice for a single index, regardless of how many queries there are targeting this index.

If you see a document twice in the results of federated search queries on the same index, please open a bug report with a reproducer so that we can investigate 🙏

[honhimW]

@dureuill Thanks for your reply.

I apologize for my previous inaccurate expression. What I meant to convey is that the search results they’re working with don’t apply multiple searches simultaneously to a single document. For example, if I want to search for movies in which the overview contains 'tragedy' and the genre contains 'comedy'. So i perform a multi-search request as fallow:

POST /multi-search

{
  "queries": [
    { "q":"tragedy", "attributesToSearchOn": ["overview"] },
    { "q":"comedy", "attributesToSearchOn": ["genres"] }
  ],
  "federation": {}
}

In this seach, I want to get is that both conditions are met, and the result set is now the union of multiple queries. So i thinking may it possible to adding parameters in federation to express whether the results of these queries should be based on “and” or “or” conditions?

[honhimW]

By the way, I think I encountered a bug related to the Issue: Filters returns invalid documents in /multi-search endpoint, which caused the data mentioned above not to be deduplicated.

It's working weird... Totally unexpectable result set are returned

[dureuill]

In this seach, I want to get is that both conditions are met, and the result set is now the union of multiple queries. So i thinking may it possible to adding parameters in federation to express whether the results of these queries should be based on “and” or “or” conditions?

Aaah, that's an interesting feature request. If I may rephrase it, you would like the option to limit the set of results to documents that are matching multiple queries.

I'm not 100% sure what the API should look like, but that's very interesting. A very easy but a bit inflexible way of expressing that could be a query level federationOptions.mustMatch boolean, which, when set to true, predicates that all returned results must belong to the set of results of that query.

Your example then becomes

POST /multi-search

{
  "queries": [
    { "q":"tragedy", "attributesToSearchOn": ["overview"], "federationOptions": { "mustMatch": true } },
    { "q":"comedy", "attributesToSearchOn": ["genres"], "federationOptions": { "mustMatch": true } }
  ],
  "federation": {}
}

we'd probably need a better name, but I like the idea.

By the way, I think I encountered a bug

Please, can I bother you to open a new issue about your bug, detailing the steps you took to get the issue. Whether or not it is indeed related to the bug you linked, it would be very helpful for me to have a way to reproduce 🙏

[sylbru]

This feature turns out to be very useful in our case! We are implementing a search UI with several fields, in order to let the user search for documents that have, say, "chaussier" in the author field and "1802" in the date field. As far as I know, I have to use filters for that. CONTAINS allows the user to provide a word that will often be contained in, not equal to, the value of the field. (This use case might match that of @honhimW above btw)

New operators like STARTS_WITH and ENDS_WITH could be a nice complement to CONTAINS, but CONTAINS would still be relevant and useful. An SQL-like % syntax might be more flexible.

Something that I would love to have would be error tolerance (and maybe other "search" stuff) in filters on text fields. Or a way to do this kind of queries (checking different fields for different keywords) with regular search, which includes error tolerance, synonyms, etc.

[sylbru]

@dureuill Hi! Just circling back to this. I realized after watching the latest webinar that this isn’t going to work in our case, since what we want is the equivalent of an "AND" operator between queries. As it stands currently, this can only emulate an "OR" query. So I guess we’re in the same situation as @honhimW.

I too see this new possibility instinctively as a federation "mode" (merge documents from all queries vs only keep documents that are returned by all queries) rather than an option at the query level (your suggestion, a boolean mustMatch parameter).
I guess the latter is more flexible: I can have one of my queries be required while the other isn’t, and setting the param to true on all of them is equivalent to an AND query. Less flexible but possibly more intuitive would be the former option (only OR like currently, or only AND). More flexible would be to allow any kind of AND/OR combinations and nesting, probably at the expense of clarity of the API?

[sylbru]

Other requests I’ve had for this UI include error tolerance, exact match, etc. so more and more like "just the great search experience we already have on the site, but targeted at specific fields/attributes," which is almost there, just that we’d need an intersection between the results of the different queries, instead of an union.

[dureuill]

Interesting feedback, I guess I have some more work to put on the federated search (AND queries, totalHits by index).

I get your point that the API would be simpler if less flexible with a global "federation mode", we might want to start there so that the simple case is easy.

What's not super clear to me in AND mode however, is what the behavior should be when there are queries against different indexes:

Should we return no results, because a document can't be in two indexes (for now), or should we limit the impact of the AND to the queries belonging to each index, meaning we'd return the intersection of the query results, by index. What do you think?

[sylbru]

So you might have four queries, two (A1 and A2) operating on index A, two (B1 and B2) on index B, and it would essentially do (A1 AND A2) OR (B1 AND B2), am I understanding that right?

I feel like that would be more confusing, rather than just returning no documents in that case. It would be like introducing a way to combine ANDs and ORs, but in a non-flexible way. If you end up adding a way to have any arbitrary combination of AND and OR operators, my feeling is that having these two behaviours would be very confusing. But then in that case having a global federation mode would also be a little confusing I guess.

[dureuill]

I see, so (for now) a federated search in the global AND mode would return no documents when performed across indexes. I think that's the most conservative option, which will allow for a better behavior once/if we add the ability to compare documents across indexes (which we'd like to do someday)

[macraig]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing the STARTS WITH filter!
You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.11.0-rc.0

You are welcome to leave your feedback in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on October 28th, 2024

⚠️ RC (release candidates) are not recommended for production

[Strijdhagen]

Hi, @macraig is CONTAINS available in the Javascript SDK?

[curquiza]

Hello, yes, you can use it with the JS SDK. Don't forget to activate the experimental feature before using it with the SDK

curl \
  -X PATCH 'http://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "containsFilter": true
  }'

[Strijdhagen]

Great, what's the syntax?

[curquiza]

To use it, everything in the description of this discussion

https://github.com/orgs/meilisearch/discussions/763#discussion-6942481

[Strijdhagen]

Got it, I didn't realize JS was exactly the same syntax as the API.
Works like a charm: filter:['locations_derived CONTAINS Amsterdam'],

[Strijdhagen]

Hello everyone, I'm wondering how to handle CONTAINS with two or more words. Is the only way to use two separate filters with 'AND'?

[irevoire]

Hey @Strijdhagen,

Yes currently that’s the only way.

And take care because it’ll match the words that overlap. I.e.: Searching for ferris and risotto in the word ferrisoto will match, even though some of the letter are used by both words.

[macraig]

Hey folks 👋

v1.11 has been released! 🐿️ You can now use the STARTS WITH filter 🎉

Looking forward to your feedback! ✨

[Asparagirl]

Yay!

Next up: ENDS_WITH ?

[macraig]

Hi @Asparagirl , we haven't prioritized ENDS_WITH just yet, but we'll update the discussion to let you know if we do.

Would you mind sharing some details on why ENDS_WITH is useful for you?

[Asparagirl]

Sure. I am thinking about using Meilisearch to replace a modified Apache Solr solution I've had for multiple websites over multiple years, for various genealogy-related databases that search through (mostly) names of people. And being able to search for ENDS_WITH "berg" would get all the names like Greenberg, Steinberg, Mossberg, etc...

But actually, the feature that I have in Solr that I would want even more than ENDS_WITH is Beider Morse Phonetic Matching (BMPM). It is basically a soundalike algorithm, but fine-tuned for surnames and their spellings, so it's better than regular old-fashioned Soundex searches. I got BMPM added to the Apache Commons Codec about a decade ago (which is why I can use it in Solr!), so it's open source and ready to use for you guys too, if you want:

https://stevemorse.org/phonetics/bmpm.htm

https://blog.thedigitalgroup.com/beider-morse-phonetic-matching-in-solr

https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/bm/BeiderMorseEncoder.html

Hope that helps. :-)

[jovanaJacimovic]

Hello people 👋,

Is there a plan to make the CONTAINS operator a stable (non-experimental) feature in any upcoming versions? If so, do you know from which version it might be supported as a standard feature?

[macraig]

Hi @jovanaJacimovic , we do plan to stabilize the feature eventually, but we can’t confirm the exact version yet. There are some performance concerns we want to address first. As a first step, we'll be exploring a proof of concept in our next iteration to better understand how we can resolve the issue.

[Asparagirl]

...or any news on potentially adding Beider Morse Phonetic Matching (BMPM), as detailed above...? Pleeeeeease?

[macraig]

Hi @Asparagirl , we don't have any updates about adding BMPM for now. We're currently starting a PoC to address our remaining concerns about the CONTAINS filter performance to prepare it for stabilization.

[irevoire]

Hey everyone, just a quick message to let you know that we fixed the performance issue with the STARTS WITH filter operator, which means it is now stabilized from now on and since v1.17.0 🎉
This means you don't need to enable the feature flag in order to use the STARTS WITH. Just make your field filterable and it'll work out of the box.

The CONTAINS is still experimental and still suffers from the same performance issues, though 😔