Repository navigation
fix: harden detokenizer against custom tokenizers with stale/edge-case bugs - #40904
Closed
vshekhawat-hlab wants to merge 2 commits into
Closed
vshekhawat-hlab wants to merge 2 commits into
vshekhawat-hlab wants to merge 2 commits into
Conversation
…e bugs Two related fixes found while diagnosing a real OOM-fix validation crash for internlm/internlm3-8b-instruct and zai-org/chatglm2-6b on Intel XPU: 1. DetokenizerManager._decode_batch_token_id_output(): both the batch (_grouped_batch_decode) and per-request decode paths called tokenizer.decode() directly with no guard against a custom convert_tokens_to_string() that assumes a non-empty token list. skip_special_tokens=True can legitimately filter every id out of a non-empty span (e.g. a span that's entirely an EOS/pad token) -- InternLM3's remote-code tokenizer (`tokens[0].startswith(...)` with no length check) then raises IndexError and crashes the whole detokenizer process, taking down every in-flight request. The existing --disable-tokenizer-batch-decode workaround for this exact tokenizer (added for a different symptom) only moves the crash to the per-request path -- confirmed reproducing there too. Added _safe_decode(), used by both paths, that returns "" and logs a warning instead of propagating. 2. patch_tokenizer.py: some custom remote-code tokenizers (e.g. zai-org/chatglm2-6b's ChatGLMTokenizer, unchanged for years) declare a fixed _pad() signature with no padding_side parameter and no **kwargs catch-all, predating a newer transformers version's PreTrainedTokenizerBase.pad() always forwarding padding_side down to _pad(). Every tokenizer.encode()/pad() call then raises "TypeError: <Tokenizer>._pad() got an unexpected keyword argument 'padding_side'" -- breaking every single generation request for that model regardless of memory/backend settings. Added a shim (following the same pattern as this file's existing Kimi tiktoken special-tokens-cache patch) that detects this structurally via inspect (not a name/class check, so any other tokenizer with the same stale signature is covered too) and drops the padding_side kwarg before delegating to the original _pad(), which already reads self.padding_side directly. Both confirmed via direct reproduction against the real installed tokenizers on real Intel XPU hardware, not just unit-level.
vshekhawat-hlab
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
September 23, 2026 11:47
Contributor
|
Thanks @vshekhawat-hlab. Closing this because you have 8 PRs open and none updated in 7 days, over our soft cap of 5 for idle PRs. Reopen the two or three you most want reviewed - a short active set moves faster than a long idle one. Some directories moved recently, so an older branch may need retargeting: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Diagnosing a real OOM-fix validation failure for
internlm/internlm3-8b-instructandzai-org/chatglm2-6bon Intel XPU surfaced two separate crashes in tokenizer handling that take down the whole detokenizer/scheduler process instead of failing gracefully for the one affected request.Modifications
DetokenizerManager: both the batch (_grouped_batch_decode) and per-request decode paths calledtokenizer.decode()with no guard against a customconvert_tokens_to_string()that assumes a non-empty token list.skip_special_tokens=Truecan legitimately filter every id out of a non-empty span, and some custom remote-code tokenizers (e.g. InternLM3's) raiseIndexErroron that instead of handling it. Added_safe_decode(), used by both paths, which returns""for that span instead of propagating.patch_tokenizer.py: some custom remote-code tokenizers (e.g. zai-org/chatglm2-6b'sChatGLMTokenizer) declare a fixed_pad()signature with nopadding_sideparam and no**kwargs, predating newertransformersversions always forwardingpadding_sidedown to_pad(). Everytokenizer.encode()/pad()call then raisesTypeError: ..._pad() got an unexpected keyword argument 'padding_side'. Added a shim, following the existing Kimi tiktoken patch's pattern, that detects this structurally viainspectand drops the kwarg before delegating to the original_pad().Accuracy Tests
Not applicable -- these are crash-recovery/compatibility fixes, not changes to model outputs.
Speed Tests and Profiling
Not applicable -- no impact on inference speed; the added logic only runs on an exception/signature-mismatch path.
Checklist
Both crashes reproduced and confirmed fixed directly against the real installed tokenizers (InternLM3 and ChatGLM2) on real Intel XPU hardware via Docker.