Add 'sort' to /documents/fetch route

[tacman]

Since the new "fetch" route has offset and limit parameters, it seems like it should have a 'sort' parameter as well.

This will be a nice feature when browsing an index by facets without needing to search the text fields.

https://github.com/meilisearch/meilisearch/releases

[daFish]

I was about to propose the same feature. My use case is as follows:

I want to have a paginatable list of documents matching a specific filter. The default now is "Best Match" but a custom sort criteria - as for the search - would be useful here.
There is currently a way to achieve the same with the search but this is rather limited by the maxTotalHits setting (which can be configured to follow my needs).

[macraig]

Hi @tacman @daFish , thank you both for your feedback!

Could you perhaps expand on the reasons behind not using the search route for this? I'd love to understand your use cases better!

[tacman]

Mine is not a particularly sophisticated reason -- when debugging, it's easier to send and replay a GET request than a POST. There's always the extra step of examining the payload, and I can't simply open the URL another other tab and hit refresh to view the data.

[macraig]

Would the /search GET route work for you?

[tacman]

Maybe. I don't need more than 1000 documents in a single call, but I do need to know an accurate number of results, which I can then paginate. Another reason to have a sort is so that pagination is consistent. I've been burned going through an entire dataset with postgres without sorting the query and discovering that fetching by offset / limit may get duplicates and skip some records if there isn't a sort.

…

On Thu, Jun 8, 2023 at 8:22 AM Maria Craig ***@***.***> wrote: Would the /search GET route <https://www.meilisearch.com/docs/reference/api/search#search-in-an-index-with-get> work for you? — Reply to this email directly, view it on GitHub <#657 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEXIQILWD4QDVNRBH7UHQLXKHN4HANCNFSM6AAAAAAYKGHKVE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[daFish]

Hey @macraig, thanks for asking.

in my application the search route is public facing and it makes sense to limit the amount of results for the reasons stated in the documentation.
In order to get more than 1000 results/documents (default) I need to update the setting for the index which is possible but maybe not a good idea.
Using the documents endpoint, my idea was to use this as some kind of search without exposing the search having the same filter/faceting features as the search but without the drawback of the documents limit.

[macraig]

Thank you both for sharing your use cases. It has inspired a potential enhancement, which we've added to our public roadmap. It would be great if you upvote the feature, as your input helps us prioritize our future development efforts.

[tacman]

Thanks! It looks like you have to pay to join productboard. Tac

…

On Thu, Jun 22, 2023 at 10:59 AM Maria Craig ***@***.***> wrote: Thank you both for sharing your use cases. It has inspired a potential enhancement, which we've added to our public roadmap <https://meilisearch.productboard.com/portal/1-meilisearch/tabs/610f17a9-3989-4418-b4ee-16a65b102e41/features/5000020/portal/expanded>. It would be great if you upvote the feature, as your input helps us prioritize our future development efforts. — Reply to this email directly, view it on GitHub <#657 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEXIQIBWXYVFSQBYBHQABTXMR2W7ANCNFSM6AAAAAAYKGHKVE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[macraig]

@tacman apologies, my bad. You should be able to access it for free here: https://roadmap.meilisearch.com/c/196-sort-fetched-documents (I'll edit the link above)

[andrefrcastro]

@tacman apologies, my bad. You should be able to access it for free here: https://roadmap.meilisearch.com/c/196-sort-fetched-documents (I'll edit the link above)

Was this removed from the roadmap ?

[macraig]

Upvote to this feature request from @panzer-punk on #666

[rectifyer]

Has there been further consideration about adding this feature request? I voted on the roadmap, but just curious if it's going to be added anytime soon. I'm evaluating Meilisearch for Trakt and we have pages such as https://trakt.tv/shows/watched/weekly which would return all show documents (no query), but I'd need to sort by the watched_count field.

[daFish]

@curquiza This approach usually works to an extend. I tried to explain it in meilisearch/meilisearch#3887

[curquiza]

Hum, when I see the issue you linked, I can say it's a little bit confusing. Sorry if I miss understand something, but I will do my best to help.

In your message here you write:

I want to show the documents sorted by another field. In my concrete case sorted by the documents title in ascending order. Once one of the documents is updated I would assume that the order of the results is not changed. However, the just updated document is not returned. This made me create this ticket in the first place because it looks like a strange behaviour to me

You should be able to sort by another field. If you did not see the document after updating it, this is another issue (a bug?) but I'm not sure the sort on the fetch documents route will fix this.
Sorting documents by ascending/descending title on title is supposed to work with Meilisearch. But maybe I have not understood your exact usecase.

In your example here, you do not mention any sorting, hence the explanation of Kero: you cannot impact the sorting of documents if you don't apply any sorting or any query, I mean if you don't use the q or sort parameter during the search.
But if you apply the sort, you will be able to see the documents you want

[tacman]

There must already be a sort (internally) in /fetch or pagination wouldn't work. The request is to expose that sort to the API.

In fact, a common mistake in some databases is to paginate through a list and forget to apply the sort, and database engines aren't required to return data in a particular or reliable order if there's no sort applied.

Sure, we can use search, but the part of the utility for fetch is that we don't need to, and fetch is an easier API to use.

[rectifyer]

@curquiza Yes, that is what I was originally going to use but I thought maxTotalHits being 1000 might be an issue. Now that I re-read the docs, it seems like that would only affect the actual result set being returned (which I only need like 100 items max) and not the total number of documents being "searched" and sorted? My largest index will be about 45 million records as well 😅

EDIT: after testing a local index with 20,000 items it does indeed seem like the maxTotalHits setting is indeed an issue. So I think that's why we need sorting support on /documents/fetch since it won't have that upper limit.

[daFish]

@curquiza Thanks for the response. I guess my test was not exact enough. I'll provide another reproducer.

[andrefrcastro]

This feature is highly important. Currently, sorting within search results always considers relevance, but there’s no option for strict sorting when fetching documents directly. If strict sorting were available for fetching, Meilisearch could be used not only for search functionalities but also for retrieving all kinds of structured data, similar to a traditional database.

Strict sorting in document fetching, without relevance adjustments, would greatly broaden Meilisearch’s application. We hope this feature is prioritized for release soon.

[andrefrcastro]

@macraig any updates on this ?

[macraig]

@andrefrcastro unfortunately no, there aren't any updates on this yet. I'll make sure to update the discussion if we decide to prioritize it.

[macraig]

Hi @tacman @andrefrcastro 👋 Good news, we've started working on adding this feature, I'll let you know once we have a concrete ETA

[andrefrcastro]

Hi @macraig,

That’s amazing news , thank you for picking this up. We’re really looking forward to it and would be happy to test or provide feedback once there’s something available.

Thanks again for your great work.

[macraig]

@andrefrcastro great! I'll let you know once we have a testable version

[macraig]

Hey everyone 👋 I'm happy to announce that v1.16 has been released and that you can now sort on the /documents route ✨

[tacman]

Awesome! This takes us yet another step closer to using meilisearch as a datastore. And while the documentation explicitly says not to do that, I'm finding it compelling to try.