Repository navigation
AutoTokenizer ignores config when model_type is None - #44127
Merged
Merged
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
2 of 5 tasks
Collaborator
Author
|
run-slow: auto |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, auto |
Contributor
|
This comment contains models: ["models/auto"] |
4 tasks done
5 of 8 tasks
3 tasks done
Stanley00
pushed a commit
to stanley-fork/openvino.genai
that referenced
this pull request
Oct 1, 2026
…th transformers 5.5 (openvinotoolkit#4566) <!-- Keep your pull requests (PRs) as atomic as possible. That increases the likelihood that an individual PR won't be stuck because of adjacent problems, merge conflicts, or code review. Your merged PR is going to appear in the automatically generated release notes on GitHub. So the clearer the title the better. --> ## Description <!-- Please include a summary of the change. Also include relevant motivation and context. --> <!-- Jira ticket number (e.g., 123). Delete if there's no ticket. --> [CVS-195654](https://jira.devtools.intel.com/browse/CVS-195654) Transformers 5.5 have changed tokenizers for TinyLLama (huggingface/transformers#44127). That caused a slight change in logits, and the output transaction id changed from 1234 to 1234567 for Python and from 1234 to 123456789 for JS. So now python integerer generation differs from JS's number generation. And we have a couple tests that compare the exact output for them, and I think we probably want exact output for Python/JS. Adding integer typing for JS fixes the issue. I've also relaxed the assert for the second issue, as with the new tokenizer tinyllama started ouputtting different answer. Anyway, the model seems too weak to solve the equations right (see the test and answers), so I think a more general assert might be better here. ## Checklist: - [x] This PR follows [GenAI Contributing guidelines](https://github.com/openvinotoolkit/openvino.genai?tab=contributing-ov-file#contributing). <!-- Always follow them. If there are deviations, explain what and why. --> - [x] <!-- (or N/A) --> Tests have been updated or added to cover the new code. <!-- Specify exactly which tests were added or updated. If the change isn't maintenance related, update the tests at https://github.com/openvinotoolkit/openvino.genai/tree/master/tests or explain in the description why the tests don't need an update. --> - [x] <!-- (or N/A) --> This PR fully addresses the ticket. <!--- If not, explain clearly what is covered and what is not. If follow-up pull requests are needed, specify in the description. -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
when the model_type isn't in
TOKENIZER_MAPPING_NAMES(ex. "llama"),TOKENIZER_MAPPING_NAMES.get("llama", "")--> "". Then we compare"" != "LlamaTokenizer"(thetokenizer_classintokenizer_config.json). Since that's true we early exit and fallback toTokenizerBackend