Limit length of retrived attribute / Streaming API?

[underflow00]

I have a use case where I need to search and retrieve an attribute that may be a very large text. But this would use too much memory (e.g. 10,000 documents * 1 MB) . Is there way to limit the length of the returned value (e.g. first X bytes/characters), or a way to stream the results so that it can be processed one at a time?

I see there is a cropLength option that does something similar.

[ManyTheFish]

Hey @underflow00,
I see you've created several issues related to your use case.
However, we are missing the big picture. What are you trying to do with Meilisearch globally?
We may find a solution that suits you with the current pool of features.

Thank you!

[ManyTheFish]

Why would the client like to fetch 10k documents from the whole set

Could be anything. Say items for sale in your demo

Ok, but why is your use case different from our demo? We are not requesting 10k documents 🤔

Say items are associated with a seller, and I have a separate SQL database to quickly retrieve all item IDs associated with a seller, and whether or not each item is processed by Meilisearch. And the client wants to get the items of a specific seller.

Why don't you put the seller as a filterable attribute in Meilisearch?

I'm sorry if my questions seem dumb. I really want to understand how specific your use case is and how we can tweak Meilisearch to fit it.

[underflow00]

We are not requesting 10k documents

10k is arbitrary. Could be 100.

Why don't you put the seller as a filterable attribute in Meilisearch

It already is. But that isn't going to retrieve documents that are still in the backlog correct?

---

Basically, Meilisearch's eventual consistency is a barrier and I'm trying to find ways to work around it. The problem isn't really specific to one application.

[sanders41]

The client wants to fetch (search and sort) up to 10K documents from the 200K documents, but cannot wait for the 100K backlog to be processed. Also, the server has quickly identified that only 100 of the 100K backlogged documents are relevant and the rest will definitely not pass the search filter. So one solution is for the server to prioritize these 100 tasks. Another solution is for the server to manually fetch the 100 documents and merge it into the Meilisearch results.

By backlog I think you are referring to the queued tasks. If these documents are queued (and threfore by definition not indexed yet) how would Meilisearch be able to search and prioritize them?

[underflow00]

By backlog I think you are referring to the queued tasks. If these documents are queued (and threfore by definition not indexed yet) how would Meilisearch be able to search and prioritize them?

As mentioned, my server knows which tasks are a priority. If Meilisearch supports task priorities then my server can tell Meilisearch which tasks are a priority. This would be one workaround.

[ManyTheFish]

So if I understand,
The indexing process of Meilisearch is too slow for your use case, forcing you to find a workaround to retrieve non-indexed documents during a search.
Did you try to reduce the indexing time by modifying your settings? If you use the v1.14 Meilisearch version, you should have a /batches route summarizing all the time spent computing each databases, if you send them here we could try to optimize them.

Then, I am not sure to understand why is it so important to retrieve all the documents even the non-indexed ones, isn't it acceptable to wait the end of the indexing process?