Skip to content

Commit 9ec2355

Browse files
authored
fix: tokenizers normalizers sequence api changed (#195)
4 days ago huggingface released `tokenizers@v0.21.2`, which [changed the `get_normalizers` method to be an `impl AsRef<[NormalizerWrapper]> for Sequence`](huggingface/tokenizers@c45aebd#diff-252dff06be81a9c396077c0b02f9f6aef6772058348bd6584031a48b351d623e). Due to the version requirement in `toktrie_hf_tokenizers` (`version = ">=0.20.0, <1.0.0"`), in the absence of a lockfile cargo fetches v0.21.2 which will not compile This was caught in EricLBuehler/mistral.rs#1523
1 parent 6fab794 commit 9ec2355

3 files changed

Lines changed: 382 additions & 432 deletions

File tree

0 commit comments

Comments
 (0)