Skip to content

qwen3 models map to wrong tokenizer class on the hub - #49116

Merged
itazap merged 2 commits into
mainfrom
qwen3_tok
Sep 28, 2026
Merged

qwen3 models map to wrong tokenizer class on the hub#49116
itazap merged 2 commits into
mainfrom
qwen3_tok

Conversation

@itazap

@itazap itazap commented Sep 25, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

CPU CI GPU run-slow

fixed: #49066

Some qwen3 model variants map incorrectly to Qwen2Tokenizer (in their hf hub checkpoints tokenizer_config.json), when they should map to Qwen3_5Tokenizer. Ideal fix is for all of these models to update their tokenizer_config.json on the hub to map 'tokenizer_class: Qwen2Tokenizer, but we can also add this override in tokenization_auto.py` (this PR).

@itazap

itazap commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

run_slow: qwen3_5, qwen3_5_moe, qwen3_vl_moe, qwen4_exp, qwen3_next, qwen3_moe, qwen3_asr

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Sep 25, 2026

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Rocketknight1
Rocketknight1 approved these changes Sep 28, 2026
View reviewed changes

@Rocketknight1 Rocketknight1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, LGTM!

@github-actions

github-actions Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, qwen3_5

@itazap

itazap commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

run_slow: qwen3_vl_moe

@itazap itazap mentioned this pull request Sep 28, 2026
Open
2 of 4 tasks
@itazap
itazap enabled auto-merge September 28, 2026 12:07
@github-actions

github-actions Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

AMD CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on AMD:

models: ["models/qwen3_vl_moe"]

@github-actions

github-actions Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Nvidia CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on Nvidia:

models: ["models/qwen3_vl_moe"]
quantizations: []

@itazap
itazap added this pull request to the merge queue Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 36419739552:1
Result: success | Jobs: 1 | Tests: 86 | Failures: 0 | Duration: 7m 43s

@github-actions

github-actions Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

CI Results (Nvidia)

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN fa776660 workflow commit (merge commit)
PR 462bf5a5 branch commit (from PR)
main 07338b6c base commit (on main)

⚠️ Model CI failed to report results

The test failure analysis could not be completed. Please check the workflow run for details.

@github-actions

github-actions Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

CI Results (AMD)

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN fa776660 workflow commit (merge commit)
PR 462bf5a5 branch commit (from PR)
main 07338b6c base commit (on main)

Model CI Report

❌ 3 new failed tests from this PR 😭

  • qwen3_vl_moe:
    tests/models/qwen3_vl_moe/test_modeling_qwen3_vl_moe.py::Qwen3VLMoeIntegrationTest::test_small_model_integration_test_batch (❌ ⟹ ❌)
    tests/models/qwen3_vl_moe/test_modeling_qwen3_vl_moe.py::Qwen3VLMoeIntegrationTest::test_small_model_integration_test_batch_different_resolutions (❌ ⟹ ❌)
    tests/models/qwen3_vl_moe/test_modeling_qwen3_vl_moe.py::Qwen3VLMoeIntegrationTest::test_small_model_integration_test_batch_flashatt2 (❌ ⟹ ❌)

Merged via the queue into main with commit a4fe1d5 Sep 28, 2026
114 of 116 checks passed
@itazap
itazap deleted the qwen3_tok branch September 28, 2026 12:35
@itazap

itazap commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

verified the failing qwn3_vl_moe test were already failing 👀

@Yinhan-Lu Yinhan-Lu mentioned this pull request Sep 29, 2026
Closed
2 of 4 tasks
This was referenced Oct 12, 2026
Open
Closed
@Irfanwani Irfanwani mentioned this pull request Oct 13, 2026
Closed
4 tasks
@hello-alexmcc hello-alexmcc mentioned this pull request Oct 16, 2026
Merged
@magnuslundstedt magnuslundstedt mentioned this pull request Oct 19, 2026
Closed
2 of 4 tasks
@ArthurZucker ArthurZucker mentioned this pull request Oct 21, 2026
Closed
@stevhliu stevhliu mentioned this pull request Oct 23, 2026
Closed
@mitukx mitukx mentioned this pull request Oct 23, 2026
Closed
@sergereview sergereview Bot mentioned this pull request Nov 2, 2026
Open
@qgallouedec qgallouedec mentioned this pull request Nov 14, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Rocketknight1 Rocketknight1 Rocketknight1 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Qwen2Tokenizer overwrites pre_tokenizer from tokenizer.json. Wrong token IDs for all Qwen3.6 / Qwen3.8 models (v4→v5 regression)

3 participants

@itazap @HuggingFaceDocBuilderDev @Rocketknight1