Improve Arabic Normalizer

[DrAliRagab]

Pull Request

Current normalizer only remove tatweel, this commit add more normalization rules.

• Remove all diacritics.
• Normalize yeh.
• Normalize alef.
• Remove all tatweel.

Currently it's a draft until tests are added.

Related issue

Improve support for Arabic language: meilisearch/product#139

[DrAliRagab]

Arabic alphabet:ا,ب,ت,ث,ج,ح,خ,د,ذ,ر,ز,س,ش,ص,ض,ط,ظ,ع,غ,ف,ق,ك,ل,م,ن,ه,و,ي,ء

Arabic text should be normalized by:

• removing the arabic Tatweel (ـ) characters.
• normalizing the arabic Alef أ,إ,آ,ٱ to ا
• normalizing the arabic Yeh ى to ي
• removing the arabic diacritics: Fatḥah, Damma, Kasrah, Alif Khanjariyah, Maddah, Sukun, Tanwin, Shaddah
• Arabic diacritics: َ, ُ, ِ, ٰ, ٓ, ْ, ۡ, ً, ٍ, ٌ, ّ,
  Arabic alphabet
  Arabic diacritics
  Kashida

For normalizing Arabic char, I used match

fn normalize_arabic_char(c: char) -> Option<CharOrStr> {
    match c {
        'ـ' => None,
        'أ' | 'إ' | 'آ' | 'ٱ' => Some('ا'.into()),
        'ى' => Some('ي'.into()),
        'َ' | 'ُ' | 'ِ' | 'ٰ' | 'ٓ' | 'ْ' | 'ۡ' | 'ً' | 'ٍ' | 'ٌ' | 'ّ' => None,
        _ => Some(c.into()),
    }
}

To detect if char should be normalized, I also used match

fn is_shoud_normalize(c: char) -> bool {
    match c {
        'ـ' | 'أ' | 'إ' | 'آ' | 'ٱ' | 'ى' | 'َ' | 'ُ' | 'ِ' | 'ٰ' | 'ٓ' | 'ْ' | 'ۡ' | 'ً' | 'ٍ' | 'ٌ'
        | 'ّ' => true,
        _ => false,
    }
}

I am not sure how to do test function because I am not fully aware of these char_end, byte_end and char_map.
To test the function:

• This: الْحُرُوف الْعَرَبِيَّة should be normalized to الحروف العربية
• This: آليس should be normalized to اليس
• This: إبرأهيم should be normalized to ابراهيم
• This: يومى should be normalized to يومي

[DrAliRagab]

c0fbe24
Adding support for arabic Taa Marbuta 'ة'
To test proper normalization:

• This: النهاردة should be normalized to النهارده

[ManyTheFish]

Hello @DrAliRagab, I feel that the diacritics are already normalized by the nonspacing-marks normalizer.

[DrAliRagab]

Hello @DrAliRagab, I feel that the diacritics are already normalized by the nonspacing-marks normalizer.

Yes, It is.
Using nonspacing-marks is an awesome idea. I didn't know about it.
I think I should remove the diacritics from this Arabic normalization rules and keep only Alef, Yeh, Taa Marbuta and Tatweel

[ManyTheFish]

Hello @DrAliRagab,
I made a review with only a small change request, the rest is all good and ready to be merged! 😊

[DrAliRagab]

I hope it's ready for merge now

[ManyTheFish]

Yep, thank you!

bors merge

[bors]

Build succeeded:

• tests