chinese support at 2025,i have some question

[globetour]

I am Chinese. First of all, I would like to thank this great project. I have some questions to ask:

1. I saw a "dict" attribute in the settings. Does it support the custom word dict in Jieba now?
2. When testing, I found that meilisearch v1.15 seems to have automatic word segmentation. The word segmentation result does not look like the effect of Jieba. Is this the default function of automatically discovering new words? Can it be turned off?
3. How to support pinyin search for the same pronunciation in Chinese? Is it supported by default now? For example, "树木(shu mu)" - "数目(shu mu)"
4. Is the jieba synonym dict enabled by default?
5. Is the jieba stop word dict enabled by default?
6. Is the jieba word segmentation dict enabled by default?
7. In addition, what configurations and dict of Jieba are used by default?
8. for example，i search “句子2 句子3”，can find “句子2 句子3 句子4...”，but cannot find “句子1 句子2 句子3”，how can i solve that?

[ManyTheFish]

Hello @globetour,
All questions are welcome, let me answer them one by one,

I saw a "dict" attribute in the settings. Does it support the custom word dict in Jieba now?

No, this feature is not related to Jieba, it will pre-segment all the words contained in the list and let Jieba segment the remaining sections.

This feature is meant to override the default segmentation.

When testing, I found that meilisearch v1.15 seems to have automatic word segmentation. The word segmentation result does not look like the effect of Jieba. Is this the default function of automatically discovering new words?

The Jieba result is over-segmented to increase the recall of Meilisearch, the segmentation result is a deduplicated version of Jieba.cut_for_search

Can it be turned off?

Not really, you could force Meilisearch to consider the document as English, but I am not sure it's what you want.

How to support pinyin search for the same pronunciation in Chinese? Is it supported by default now? For example, "树木(shu mu)" - "数目(shu mu)"

Pinyin support was introduced in an earlier version of Meilisearch, but we rolled back the support due to several complaints about relevancy. Meilisearch doesn't provide a setting to reactivate it.

Is the jieba synonym dict enabled by default?
Is the jieba stop word dict enabled by default?
Is the jieba word segmentation dict enabled by default?
In addition, what configurations and dict of Jieba are used by default?

We use a default configuration of Jieba, below is the link of the segmenter using Jieba:
https://github.com/meilisearch/charabia/blob/main/charabia/src/segmenter/chinese.rs

for example，i search “句子2 句子3”，can find “句子2 句子3 句子4...”，but cannot find “句子1 句子2 句子3”，how can i solve that?

This should work to me, there is maybe a bug here, I'll need a bit of time to investigate 🤔

Thank you for your questions @globetour,

Don't hesitate to ask for more information if you need to!

If you need more context about the Chinese support in Meilisearch an the history around it the following discussion may interest you:
https://github.com/orgs/meilisearch/discussions/503

see you! ☺️

[globetour]

The Chinese new word discovery uses JIEBA's HMM function. When meilisearch uses JIEBA's default function, is HMM=true or false?

[ManyTheFish]

false

[globetour]

false

But I found the new words with meilisearch, i like the HMM function(HMM=true) , and the HMM Configuration Switch is need in meilisearch.

[globetour]

thank you ,i will try again