The requirement or advice of chinese word segmentation

[AlphaPigger]

Describe the requirement
i expect the chinese input text will be splited into all possible words
for example:

The behavior of Current Version

The advice of optimization
i notice that you use jieba default constraction and this will cause some highlight errors or search errors of chinese word segmentation.So,can you use the cat_all method from jieba library to get chinese word segmentation?

Additional text or screenshots

I except your reply,thanks @ManyTheFish

[ManyTheFish]

Hello @AlphaPigger,
the issue in cut_all or cut_for_search is that generates words and don't really cut the text, in Meilisearch we want to segment the text only.
The reason for this is that Meilisearch is taking the proximity between words into account, and we need to store words with their position in the database.
for instance:
a text like 在尊嚴和權利上一律平等 would be segmented as ["在", "尊嚴", "和", "權利上", "一律平等"], making:

• position 1: "在"
• position 2: "尊嚴"
• position 3: "和"
• position 4: "權利上"
• position 5: "一律平等"

This allows Meilisearch to rank documents by the proximity between matching words.
Now let's say we use cut_all the result would be ["在", "尊", "和", "權", "利", "上", "一", "一律", "一律平等", "律", "平", "平等", "等"] which would add a lot of mess in word positions. 😣

However, the proximity feature could be an ally for Chinese Languages support, today we've activated the HMM (hidden Markov model) feature that creates longer words, we could try to deactivate it which will create smaller words and proximity could do the rest:
for the text:

人人生而自由﹐在尊嚴和權利上一律平等。他們賦有理性和良心﹐並應以兄弟關係的精神互相對待。

with HMM:

["人人", "生而自由", "﹐", "在", "尊嚴", "和", "權利上", "一律平等", "。", "他們", "賦有", "理性", "和", "良心", "﹐", "並應", "以", "兄弟", "關", "係", "的", "精神", "互相", "對待", "。"]

without HMM:

["人人", "生而自由", "﹐", "在", "尊", "嚴", "和", "權", "利", "上", "一律平等", "。", "他", "們", "賦", "有", "理性", "和", "良心", "﹐", "並", "應", "以", "兄弟", "關", "係", "的", "精神", "互相", "對", "待", "。"]

[ManyTheFish]

closing in favor of meilisearch/product#503