Skip to content

Commit 1580f64

Browse files
authored
[smollm3] add tokenizer mapping for smollm3 (#39271)
add tok mapping to smollm3
1 parent db05e4f commit 1580f64

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

src/transformers/models/auto/tokenization_auto.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -580,6 +580,7 @@
580580
"GemmaTokenizerFast" if is_tokenizers_available() else None,
581581
),
582582
),
583+
("smollm3", (None, "PreTrainedTokenizerFast" if is_tokenizers_available() else None)),
583584
("speech_to_text", ("Speech2TextTokenizer" if is_sentencepiece_available() else None, None)),
584585
("speech_to_text_2", ("Speech2Text2Tokenizer", None)),
585586
("speecht5", ("SpeechT5Tokenizer" if is_sentencepiece_available() else None, None)),

0 commit comments

Comments
 (0)