Try to remove pickle - BloomTokenizerFast

[ydshieh]

What does this PR do?

I can see the effect as the value is changed as expected from

(Pdb) self.backend_tokenizer.pre_tokenizer
Sequence(pretokenizers=[Split(pattern=Regex(" ?[^(\s|[.,!?…。，、।۔،])]+"), behavior=Isolated, invert=False), ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=False)])

to

(Pdb) self.backend_tokenizer.pre_tokenizer
Sequence(pretokenizers=[Split(pattern=Regex(" ?[^(\s|[.,!?…。，、।۔،])]+"), behavior=Isolated, invert=False), ByteLevel(add_prefix_space=True, trim_offsets=True, use_regex=False)])

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

There are some similar places like

src/transformers/models/cohere/tokenization_cohere_fast.py

But would like to have a confirmation first.

[ArthurZucker]

Yes and long due, this was not possible before, it is now because of the update to tokenizers ! Can be applied everywhere else!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: bloom, cohere

[ydshieh]

run-slow: bloom, cohere

[ydshieh]

run-slow: bloom, cohere

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/bloom', 'models/cohere']
quantizations: [] ...