Skip to content

[Fix] grammar: Give grammar accept failures an error status - #46049

Merged
ShangmingCai merged 1 commit into
sgl-project:mainfrom
weireweire:fix/pd-prefill-abort-notify-decode
Oct 26, 2026
Merged

ShangmingCai merged 1 commit into
sgl-project:mainfrom
weireweire:fix/pd-prefill-abort-notify-decode

Conversation

@weireweire

@weireweire weireweire commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

A grammar accept_token failure in the shared result processor aborts the request with a bare FINISH_ABORT(), which has no status code (L730, L872). TokenizerManager._handle_abort_finish_reason turns an abort into an HTTP error only when its status is 4xx/5xx, so the client receives a 200 whose finish reason is abort. The OpenAI streaming path likewise treats it as a graceful abort rather than an error.

The rejected token was sampled under the grammar's own bitmask, so the failure is a server-side fault, such as an FSM/bitmask desync or an xgrammar issue. Other grammar handlers already report it that way: the PD prefill handler (process_batch_result_disagg_prefill) and the PD decode prebuilt path both abort with 500. The sampling-mask INVALID failure in the same file does too.

Scope: these two sites run on aggregated servers and on PD decode workers. A PD prefill server never reaches them, because it processes extend results in process_batch_result_disagg_prefill. This change therefore does not affect how PD frontends cancel a paired decode request.

This PR originally made prefill send KVPoll.Failed to decode on prefill-side aborts. Following the review discussion, that is left to the frontend, and this PR only makes this abort report a status.

Modifications

Both grammar accept_token failure sites abort with HTTPStatus.INTERNAL_SERVER_ERROR and InternalServerError, with the accept_token error as the message. This matches the PD grammar handlers and the sampling-mask failures.

Behavior for an affected request:

  • Non-streaming requests get a 500.
  • OpenAI streaming requests get an error chunk.
  • Native /generate streaming carries status_code 500 in the final finish_reason.

Accuracy Tests

No numerical change. Only the abort's status and message change.

Checklist

  • Format your code according to the Format code with pre-commit. (ruff with the repo's hook selection and ruff-format pass on the changed file.)

CI States

Latest PR Test (Base): 🚫 Run #40283550107
Latest PR Test (Extra): ❌ Run #40283550192
Latest PR Test (AMD ROCm 10): ⏳ Run #40283551156

@ShangmingCai ShangmingCai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this notification be done by the router already?

@weireweire

Copy link
Copy Markdown
Contributor Author

@ShangmingCai good point. it's not in current dynamo, but I find ai-dynamo/dynamo#16766 will help when it include in new release.
one minor change needed is that abort like HERE abort without err code and may be ignored by sgl-router. (not by dynamo)
I guess we only need to fix that and don't need this anymore.

@ShangmingCai

Copy link
Copy Markdown
Collaborator

@ShangmingCai good point. it's not in current dynamo, but I find ai-dynamo/dynamo#16766 will help when it include in new release. one minor change needed is that abort like HERE abort without err code and may be ignored by sgl-router. (not by dynamo) I guess we only need to fix that and don't need this anymore.

Sure, you could force-push this or make a new PR for that minor change, we should be able to merge it today.

@weireweire
weireweire force-pushed the fix/pd-prefill-abort-notify-decode branch from f6b5942 to 0118934 Compare October 26, 2026 06:30
@weireweire weireweire changed the title [Fix] disaggregation: Notify decode when prefill aborts a bootstrapped request [Fix] grammar: Give grammar accept failures an error status Oct 26, 2026
@weireweire

Copy link
Copy Markdown
Contributor Author

thanks, changed this pr to the minor fix

A grammar accept_token failure in the shared result processor aborted the
request with a bare FINISH_ABORT(), which carries no status code. The
tokenizer manager only turns aborts with a 4xx/5xx status into an HTTP
error, so the client received a 200 whose finish reason was an abort.

A rejected token was sampled under the grammar's own bitmask, so this is a
server-side fault. Abort with 500 InternalServerError and the accept_token
error, as the sampling-mask INVALID failure and the PD prefill and decode
grammar handlers already do.
@weireweire
weireweire force-pushed the fix/pd-prefill-abort-notify-decode branch from 0118934 to 0fb1c48 Compare October 26, 2026 07:04

@ShangmingCai ShangmingCai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just not sure about some corner use-case.

@ShangmingCai

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 26, 2026
@ShangmingCai

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/managers/test_batch_result_processor_spec_grammar.py test/registered/unit/entrypoints/openai/test_serving_chat.py test/registered/unit/entrypoints/openai/test_serving_completions.py test/registered/constrained_decoding/test_constrained_decoding.py test/registered/spec/test_constrained_decoding_spec_reasoning.py test/registered/e2e/radix_cache/test_mamba_extra_buffer_spec_grammar.py test/registered/disaggregation/test_disaggregation_basic.py

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (3 tests): ✅ View workflow run

cd test/ && python3 registered/unit/managers/test_batch_result_processor_spec_grammar.py
cd test/ && python3 registered/unit/entrypoints/openai/test_serving_chat.py
cd test/ && python3 registered/unit/entrypoints/openai/test_serving_completions.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/constrained_decoding/test_constrained_decoding.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/spec/test_constrained_decoding_spec_reasoning.py
cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/radix_cache/test_mamba_extra_buffer_spec_grammar.py

@ShangmingCai
ShangmingCai merged commit 5ef9901 into sgl-project:main Oct 26, 2026
242 of 283 checks passed
detain pushed a commit to detain/sglang that referenced this pull request Oct 29, 2026
…ect#46049)

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
(cherry picked from commit 5ef9901)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants