Support more than 65K positions per attribute

[underflow00]

Meilisearch can index a maximum of 65535 positions per attribute. Any words exceeding the 65535 position limit will be silently ignored.

This limitation prevents using Meilisearch for searching arbitrary large text with no predefined format, like a 500 page research paper or legal document.

Would be good to have an option to increase this limit.

[dureuill]

Hello, thank you for voicing your request,

Meilisearch does not support this, because large documents are extremely adverse to performance.
Instead, split any large document into smaller chunks that you each index as a document. This also makes document retrieval and highlighting of the results much more accurate. If this doesn't work for you, can you elaborate on why?

Also note that this limit is architectural, so increasing it is a large change.

[underflow00]

If this doesn't work for you, can you elaborate on why?

Well it's already in the post. This limitation prevents using Meilisearch for searching arbitrary large text with no predefined format. Could be uploaded by the user.

[dureuill]

This limitation prevents using Meilisearch for searching arbitrary large text with no predefined format.

Not really, you could split documents after they reach a specific size and index them as multiple documents. As outlined in my previous response, lifting this particular limitation is not going to make arbitrary document support practical in Meilisearch.

Could be uploaded by the user.

Set up a proxy in front of Meilisearch to split the documents, handle authentication, validation, etc.

[underflow00]

you could split documents

I've thought about this, but it's not practical, as it breaks many other features offered by Meilisearch. Among other things, user code will need a reliable way to split the documents via hard separators, sortable fields will have to be duplicated among the split documents for sort to work, search hit limit has to be increased or multiple searches have to be performed to meet the desired limit, search results containing split documents have to be merged and rejoined, etc, etc, etc.

Set up a proxy in front

Of course. I'm saying that the "proxy" does not know what the format is because it is decided by the user.

---

Elasticsearch and some other alternatives do support larger values. Is it Meilisearch's stance that Meilisearch is not suitable for such use cases?

[dureuill]

Of course, chunking will be more efficient if the input is structured.
Is the application expecting raw text, or structured JSON?

• If raw text, one can split after n bytes or n whitespace-separated words.
• If JSON, one can do the same value by value in the JSON object representing the document. Note that if the end-users can decide on the fields of documents, they may also overflow the number of available unique fields in an index, which is also 65,535.

search hit limit has to be increased or multiple searches have to be performed to meet the desired limit

You can keep the identity of the split documents in an field and set up that field as the distinct attribute for your index, so that a search will not match multiple times the same document.

search results containing split documents have to be merged and rejoined

That is true, but also performance-wise it will save a lot of bandwidth and ultimately time, not to serve 20 500-page research papers in response to any search request. Instead providing end-users with the snippets of the documents that matched their query, and letting them access full documents in a separate way.

Is it Meilisearch's stance that Meilisearch is not suitable for such use cases?

Meilisearch is suitable for indexing large documents. However, it does not automatically handle the chunking of such large documents, so integrators have to perform the chunking beforehand. We are considering automatic chunking schemes, we already implemented some in, for example, the document crawlers that we use in our Cloud offering, but these are tailored to HTML documents.