[ArthurZucker]

Add reference to normalizer blog post. Could help users as the blog post does not really come out when looking for more detail documentation.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Narsil]

Hi @ArthurZucker

Thanks for this ! Do you mind updating README.tpl too (README.md is generated from it and directly from the docstrings within rust code so updating both README.md and README.tpl is necessary).

Also this is the README of the tokenizers/tokenizers folder, so it has less visibility than the main readme (just fyi)

[ArthurZucker]

Hey, I am not really sure about the README.tpl, it seems pretty empty to me :

<p align="center">
    <br>
    <img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
    <br>
<p>
<p align="center">
    <img alt="Build" src="https://github.com/huggingface/tokenizers/workflows/Rust/badge.svg">
    <a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE">
        <img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue">
    </a>
    <a href="https://docs.rs/tokenizers/">
        <img alt="Doc" src="https://docs.rs/tokenizers/badge.svg">
    </a>
</p>
<br>


{{readme}}

Am I missing something ?

[Narsil]

Oh sorry what you modified seems to be in the Rust docstring

src/lib.rs https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/lib.rs.

The check is in place exactly so that we don't forget to update both (rust docsrtings generate automatically this page https://docs.rs/tokenizers/0.11.3/tokenizers/

[Narsil]

@ArthurZucker still a small issue, I took the liberty to update the PR cause it's harder for you to fix than me (cargo fmt fixes formatting automatically)

[ArthurZucker]

No problem thanks a lot :)