Repository navigation
[Fix] grammar: Give grammar accept failures an error status - #46049
Conversation
ShangmingCai
left a comment
There was a problem hiding this comment.
Shouldn't this notification be done by the router already?
|
@ShangmingCai good point. it's not in current dynamo, but I find ai-dynamo/dynamo#16766 will help when it include in new release. |
Sure, you could force-push this or make a new PR for that minor change, we should be able to merge it today. |
f6b5942 to
0118934
Compare
|
thanks, changed this pr to the minor fix |
A grammar accept_token failure in the shared result processor aborted the request with a bare FINISH_ABORT(), which carries no status code. The tokenizer manager only turns aborts with a 4xx/5xx status into an HTTP error, so the client received a 200 whose finish reason was an abort. A rejected token was sampled under the grammar's own bitmask, so this is a server-side fault. Abort with 500 InternalServerError and the accept_token error, as the sampling-mask INVALID failure and the PD prefill and decode grammar handlers already do.
0118934 to
0fb1c48
Compare
ShangmingCai
left a comment
There was a problem hiding this comment.
LGTM, just not sure about some corner use-case.
|
/tag-and-rerun-ci |
|
/rerun-test test/registered/unit/managers/test_batch_result_processor_spec_grammar.py test/registered/unit/entrypoints/openai/test_serving_chat.py test/registered/unit/entrypoints/openai/test_serving_completions.py test/registered/constrained_decoding/test_constrained_decoding.py test/registered/spec/test_constrained_decoding_spec_reasoning.py test/registered/e2e/radix_cache/test_mamba_extra_buffer_spec_grammar.py test/registered/disaggregation/test_disaggregation_basic.py |
|
🚀 🚀 🚀 🚀 |
…ect#46049) Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> (cherry picked from commit 5ef9901)
Motivation
A grammar
accept_tokenfailure in the shared result processor aborts the request with a bareFINISH_ABORT(), which has no status code (L730, L872).TokenizerManager._handle_abort_finish_reasonturns an abort into an HTTP error only when its status is 4xx/5xx, so the client receives a 200 whose finish reason isabort. The OpenAI streaming path likewise treats it as a graceful abort rather than an error.The rejected token was sampled under the grammar's own bitmask, so the failure is a server-side fault, such as an FSM/bitmask desync or an xgrammar issue. Other grammar handlers already report it that way: the PD prefill handler (
process_batch_result_disagg_prefill) and the PD decode prebuilt path both abort with 500. The sampling-mask INVALID failure in the same file does too.Scope: these two sites run on aggregated servers and on PD decode workers. A PD prefill server never reaches them, because it processes extend results in
process_batch_result_disagg_prefill. This change therefore does not affect how PD frontends cancel a paired decode request.This PR originally made prefill send
KVPoll.Failedto decode on prefill-side aborts. Following the review discussion, that is left to the frontend, and this PR only makes this abort report a status.Modifications
Both grammar
accept_tokenfailure sites abort withHTTPStatus.INTERNAL_SERVER_ERRORandInternalServerError, with theaccept_tokenerror as the message. This matches the PD grammar handlers and the sampling-mask failures.Behavior for an affected request:
/generatestreaming carriesstatus_code500 in the finalfinish_reason.Accuracy Tests
No numerical change. Only the abort's status and message change.
Checklist
CI States
Latest PR Test (Base): 🚫 Run #40283550107
Latest PR Test (Extra): ❌ Run #40283550192
Latest PR Test (AMD ROCm 10): ⏳ Run #40283551156