Skip to content

fix: harden detokenizer against custom tokenizers with stale/edge-case bugs - #40904

Closed
vshekhawat-hlab wants to merge 2 commits into
sgl-project:mainfrom
vshekhawat-hlab:fix/detokenizer-empty-token-crash-and-pad-shim
Closed

vshekhawat-hlab wants to merge 2 commits into
sgl-project:mainfrom
vshekhawat-hlab:fix/detokenizer-empty-token-crash-and-pad-shim

Conversation

@vshekhawat-hlab

@vshekhawat-hlab vshekhawat-hlab commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Diagnosing a real OOM-fix validation failure for internlm/internlm3-8b-instruct and zai-org/chatglm2-6b on Intel XPU surfaced two separate crashes in tokenizer handling that take down the whole detokenizer/scheduler process instead of failing gracefully for the one affected request.

Modifications

  • DetokenizerManager: both the batch (_grouped_batch_decode) and per-request decode paths called tokenizer.decode() with no guard against a custom convert_tokens_to_string() that assumes a non-empty token list. skip_special_tokens=True can legitimately filter every id out of a non-empty span, and some custom remote-code tokenizers (e.g. InternLM3's) raise IndexError on that instead of handling it. Added _safe_decode(), used by both paths, which returns "" for that span instead of propagating.
  • patch_tokenizer.py: some custom remote-code tokenizers (e.g. zai-org/chatglm2-6b's ChatGLMTokenizer) declare a fixed _pad() signature with no padding_side param and no **kwargs, predating newer transformers versions always forwarding padding_side down to _pad(). Every tokenizer.encode()/pad() call then raises TypeError: ..._pad() got an unexpected keyword argument 'padding_side'. Added a shim, following the existing Kimi tiktoken patch's pattern, that detects this structurally via inspect and drops the kwarg before delegating to the original _pad().

Accuracy Tests

Not applicable -- these are crash-recovery/compatibility fixes, not changes to model outputs.

Speed Tests and Profiling

Not applicable -- no impact on inference speed; the added logic only runs on an exception/signature-mismatch path.

Checklist

Both crashes reproduced and confirmed fixed directly against the real installed tokenizers (InternLM3 and ChatGLM2) on real Intel XPU hardware via Docker.

…e bugs

Two related fixes found while diagnosing a real OOM-fix validation crash
for internlm/internlm3-8b-instruct and zai-org/chatglm2-6b on Intel XPU:

1. DetokenizerManager._decode_batch_token_id_output(): both the batch
   (_grouped_batch_decode) and per-request decode paths called
   tokenizer.decode() directly with no guard against a custom
   convert_tokens_to_string() that assumes a non-empty token list.
   skip_special_tokens=True can legitimately filter every id out of a
   non-empty span (e.g. a span that's entirely an EOS/pad token) --
   InternLM3's remote-code tokenizer (`tokens[0].startswith(...)` with no
   length check) then raises IndexError and crashes the whole detokenizer
   process, taking down every in-flight request. The existing
   --disable-tokenizer-batch-decode workaround for this exact tokenizer
   (added for a different symptom) only moves the crash to the per-request
   path -- confirmed reproducing there too. Added _safe_decode(), used by
   both paths, that returns "" and logs a warning instead of propagating.

2. patch_tokenizer.py: some custom remote-code tokenizers (e.g.
   zai-org/chatglm2-6b's ChatGLMTokenizer, unchanged for years) declare a
   fixed _pad() signature with no padding_side parameter and no **kwargs
   catch-all, predating a newer transformers version's
   PreTrainedTokenizerBase.pad() always forwarding padding_side down to
   _pad(). Every tokenizer.encode()/pad() call then raises "TypeError:
   <Tokenizer>._pad() got an unexpected keyword argument 'padding_side'" --
   breaking every single generation request for that model regardless of
   memory/backend settings. Added a shim (following the same pattern as
   this file's existing Kimi tiktoken special-tokens-cache patch) that
   detects this structurally via inspect (not a name/class check, so any
   other tokenizer with the same stale signature is covered too) and drops
   the padding_side kwarg before delegating to the original _pad(), which
   already reads self.padding_side directly.

Both confirmed via direct reproduction against the real installed
tokenizers on real Intel XPU hardware, not just unit-level.
@github-actions

Copy link
Copy Markdown
Contributor

Thanks @vshekhawat-hlab. Closing this because you have 8 PRs open and none updated in 7 days, over our soft cap of 5 for idle PRs.

Reopen the two or three you most want reviewed - a short active set moves faster than a long idle one.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant