[tanvir-ux]

Fixes #49066

What was wrong

Qwen2Tokenizer.__init__ always built the Split pre-tokenizer from the module-level PRETOKENIZE_REGEX, even when tokenizer_config.json carried a different pretokenize_regex.

That was invisible for Qwen2 / Qwen3 because the constant matched the published file. Qwen3.6 and Qwen3.8 refine the pattern with \p{M} (combining marks). Loading those checkpoints with AutoTokenizer dropped the marks into the punctuation branch, so Hindi / Thai / Arabic text got 50–75% more tokens than the published tokenizer.json, and save_pretrained baked the wrong pattern into derived repos.

I reproduced the issue's Hindi sample on current main: published IDs [58069, 84237, 150104, 153348, 184642, 235886], loaded IDs differed. After this change they match.

What I changed

• Read pretokenize_regex from kwargs (comes from tokenizer_config.json via from_pretrained) and use that for the Split pattern when present.
• Fall back to PRETOKENIZE_REGEX when the key is missing, so older Qwen2 / Qwen3 checkpoints without it keep the same tokenization.
• Added a small regression test that checks the kwarg path vs the default constant.

I deliberately did not update the module constant itself — that would change default tokenization for older checkpoints that omit the config key.

How I checked

• New unit test: test_honors_pretokenize_regex_from_kwargs — passed.
• Hub repro from Qwen2Tokenizer overwrites pre_tokenizer from tokenizer.json. Wrong token IDs for all Qwen3.6 / Qwen3.8 models (v4→v5 regression) #49066 against Qwen/Qwen3.8-Flash-Next (Hindi / Thai / Arabic / Latin) — IDs match published tokenizer.json.
• save_pretrained round-trip keeps \p{M} in both tokenizer_config.json and tokenizer.json.
• Spot-checked Qwen/Qwen2.5-0.5B-Instruct (no pretokenize_regex in config) still loads.

Happy to adjust if you'd rather pull the pre-tokenizer straight off tokenizer.json in TokenizersBackend instead — this scoped fix matched what the issue suggested for Qwen2Tokenizer.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: qwen2

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 35976981835
Result: failure | Grafana metrics are not available yet.

Code quality check failed: test jobs were skipped. Fix the code quality issues and push again to run tests.

[ArthurZucker]

Thanks @tanvir-ux! As @itazap explained in #49066, the hardcoded pre-tokenizer regex in tokenization_qwen2.py is intentional, and the wrong IDs were fixed by mapping those checkpoints to the right tokenizer class in #49116. Closing as superseded. If you still hit wrong token IDs on current main (e.g. the case @magnuslundstedt described, where the checkpoint declares Qwen2Tokenizer itself), please follow up on #49066 with the checkpoint name.