Commit 9ec2355
authored
fix: tokenizers normalizers sequence api changed (#195)
4 days ago huggingface released `tokenizers@v0.21.2`, which [changed the
`get_normalizers` method to be an `impl AsRef<[NormalizerWrapper]> for
Sequence`](huggingface/tokenizers@c45aebd#diff-252dff06be81a9c396077c0b02f9f6aef6772058348bd6584031a48b351d623e).
Due to the version requirement in `toktrie_hf_tokenizers` (`version =
">=0.20.0, <1.0.0"`), in the absence of a lockfile cargo fetches v0.21.2
which will not compile
This was caught in EricLBuehler/mistral.rs#15231 parent 6fab794 commit 9ec2355
3 files changed
Lines changed: 382 additions & 432 deletions
0 commit comments