Skip to content
Open
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
d2eceb9
Improve AutoTokenizer error message for Voxtral models
Khansa435 Oct 14, 2025
7f7cfcb
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 14, 2025
a79a7ba
Passing ruff checks
Khansa435 Oct 14, 2025
12ee3b0
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 14, 2025
9b33bb4
Improved formatting
Khansa435 Oct 14, 2025
a3bc18b
fix_bad_error_voxtral_tokenizer
Khansa435 Oct 14, 2025
066bee6
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
3c28280
fix_bad_error
Khansa435 Oct 15, 2025
be6cf68
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 15, 2025
0afd559
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
3c9732c
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
7ee01a5
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
c08826c
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
37b9adb
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
7685851
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 15, 2025
5314035
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 23, 2025
0e52f37
Raise an error for missing mistral
Khansa435 Oct 23, 2025
4bdfc51
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 23, 2025
5de1f23
Improve formatting and code_quality
Khansa435 Oct 23, 2025
92946da
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 27, 2025
918b93f
Simplifying the changes
Khansa435 Oct 29, 2025
a827397
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 29, 2025
5966f8a
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 29, 2025
abe0bda
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 29, 2025
2294157
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 29, 2025
653d2ab
Improve Auto Tokenizer error message
Khansa435 Oct 30, 2025
6da632a
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 30, 2025
aed9a74
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Oct 30, 2025
1d086f6
Merge branch 'fix-bad-error-voxtral-tokenizer' of https://github.com/…
Khansa435 Oct 30, 2025
58a5025
Improve error message for voxtral
Khansa435 Oct 30, 2025
e97185d
Merge branch 'main' into fix-bad-error-voxtral-tokenizer
Khansa435 Nov 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion src/transformers/models/auto/tokenization_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"""Auto Tokenizer class."""

import importlib
import importlib.util
import json
import os
from collections import OrderedDict
Expand Down Expand Up @@ -1156,13 +1157,19 @@ def from_pretrained(
config = config.encoder

model_type = config_class_to_model_type(type(config).__name__)

if model_type is not None:
tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]

if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
else:
if tokenizer_class_py is not None:
# Check for Mistral/Voxtral models before loading
if "Mistral" in tokenizer_class_py.__name__ and importlib.util.find_spec("mistral_common") is None:
raise ImportError(
"The tokenizer for Voxtral or Mistral models requires the `mistral-common` package.\n"
"Please install it with:\n\n pip install mistral-common\n"
)
return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
else:
raise ValueError(
Expand Down