Repository navigation
fix(frontend): reject over-context requests before streaming - #12092
Merged
Merged
Conversation
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Contributor
WalkthroughChangesStreaming error preflight
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
|
🎯 Code Coverage (details) 🔗 Commit SHA: 34d6e9a | Docs | Datadog PR Page | Give us feedback! |
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Contributor
|
Related: #11570 |
1 task done
Apply the engine-declared strict budget to legacy Completions and cover the Responses API token-budget handoff. Centralize admission so chat and completion paths share exact-length and deferral rules.\n\nCo-authored-by: Tianhe Zhang <tianhez@nvidia.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
…-context-overflow-status Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com> # Conflicts: # components/src/dynamo/vllm/handlers.py # components/src/dynamo/vllm/tests/test_vllm_legacy_lora.py # lib/llm/src/http/service/openai.rs # lib/llm/src/preprocessor.rs # lib/llm/tests/http-service.rs
Contributor
Author
|
/ok to test f4fe58a |
biswapanda
approved these changes
Aug 3, 2026
jh-nv
approved these changes
Aug 5, 2026
nnshah1
approved these changes
Aug 5, 2026
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
dmitry-tokarev-nv
approved these changes
Aug 5, 2026
dmitry-tokarev-nv
left a comment
Contributor
There was a problem hiding this comment.
LGTM from DevOps side
connorcarpenter15
approved these changes
Aug 5, 2026
galletas1712
approved these changes
Aug 5, 2026
rmccorm4
reviewed
Aug 6, 2026
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
…erflow-status' into warnold/fix-streaming-context-overflow-status Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Contributor
Author
|
/ok to test a63d228 |
1 similar comment
Contributor
Author
|
/ok to test a63d228 |
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Contributor
Author
|
/ok to test 34d6e9a |
jokerhaos
pushed a commit
to jokerhaos/dynamo
that referenced
this pull request
Sep 11, 2026
… settable) The SGLang chat processor dropped the reason on backend rejections on the disaggregated SGLang path: an over-context ValueError from SGLang and a LocalRouter queue-overflow rejection were both swallowed by the generic `except Exception` and re-raised as `Unknown`, while the `is_error` and `handle_engine_error` yield paths emitted bare dicts the binding misread as completion chunks -- the client got an opaque 500 (or a bare SSE frame) with no reason. ai-dynamo#14063 added the equivalent handling to the vLLM chat processor (classification + `as_error_envelope` + `backend_invalid_argument_to_http_error`) but did not touch `sglang_processor.py`. The preflight guard from ai-dynamo#12092 also does not gate this path: the worker's `token_budget` does not reach the frontend MDC under disaggregation, so `validate_requested_token_budget` returns early and the request still reaches the worker, which rejects it. This mirrors ai-dynamo#14063 onto the SGLang path so the reason always reaches the client, and sets the HTTP status where the HTTP layer can still set it: - classify over-context messages to a 400-shaped error and LocalRouter queue-overflow rejections to a 429-shaped one (regex classifier `_classify_sglang_error`, which reuses `backend_invalid_argument_to_http_error` for the `BackendInvalidArgument: {json}` envelope); - a classified error arriving before the first byte `raise`s `HttpError`. The resulting 400/429 status is set only where the HTTP layer can still set it: non-streaming requests, and streaming with `DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS` (the opt-in pre-commit error peek). Default streaming commits 200 up front, so a raise there surfaces as 200 + SSE error event carrying the reason (still better than an opaque 500); - a classified error arriving mid-stream, after the 200 is committed, is `yield`ed as an `as_error_envelope` SSE error event so the reason reaches the client instead of truncating the stream (`yielded_any` flag, set by the two flush `yield envelope` sites); - add `except HttpError: raise` so a raised `HttpError` propagates to the HTTP boundary instead of being re-caught by `except Exception` and re-wrapped (mirrors ai-dynamo#14063's `except VLLMClientError: raise`); - unclassified errors keep their existing behavior (`as_error_envelope` for `is_error` / missing `token_ids`; `raise Unknown` for genuine engine faults), so the common cases are unchanged. Known limitation: making the 4xx status reach the default (peek-off) streaming path needs the pre-commit error peek enabled for these validation errors -- a Rust HTTP-service change, out of scope here and in the scope of DEP ai-dynamo#14354. Updates the two affected `test_sglang_processor_unit.py` tests to the tagged-envelope shape the binding now expects, and adds a `TestClassifySglangError` unit suite covering over-context / queue-overflow / `BackendInvalidArgument` / unclassified. Validation: py_compile + ruff 0.5.2 check/format pass; runtime checks (21 assertions, all pass) via stub-loaded real `sglang_processor.py` + real `frontend/utils.py` (PyO3/CUDA deps stubbed) -- pre-stream classified still raise (429/400); mid-stream classified yield an error event with the reason instead of truncating; unclassified paths unchanged. Related to GitHub issue ai-dynamo#14354 (DEP: unified semantic error propagation), which would subsume this once it lands. Signed-off-by: joker <a1214096242@gamil.com>
This was referenced Sep 12, 2026
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Dynamo currently commits HTTP 200 for streaming requests before a backend can reject an
over-context token budget. That turns a client error into an inline server error, triggers
meaningless retries, and distorts monitoring.
This change makes each engine publish the request overflows that the frontend may reject
early. The shared OpenAI preprocessor applies only those advertised rejections, before
backend dispatch and before streaming headers are sent. Backend-owned clamping and
truncation remain in the backend.
Summary
add a minimal engine/frontend capability:
A
falseflag delegates that overflow dimension to the backend. Missing or malformedmetadata also delegates both dimensions, so unofficial backends and version-skewed
deployments stay online.
publish the resolved backend behavior:
--allow-auto-truncatemax_seq_lenis explicit; otherwise delegatedpublish matching metadata for base and dynamic LoRA model cards; embedding-only and
engine-less encode cards remain excluded where generation policy does not apply
reject exact over-budget requests in the canonical preprocessor for Chat Completions,
Responses, and legacy Completions
defer prompt embeddings and multimodal lengths the frontend cannot prove
preserve explicit requests on delegated paths; the frontend does not reproduce backend
clamping or truncation
preserve immediate streaming headers, SSE keep-alives, backend inactivity timeouts, and
client cancellation for admitted requests
propagate typed
InvalidArgumentas JSON HTTP 400 and classify it as a validation failurein frontend metrics without changing the backend's wire message
document the capability and fail-open behavior for custom Python workers
Why frontend admission is needed
The engine remains the source of policy. Standalone SGLang rejects an over-budget request in
its tokenizer manager before starting a stream. Dynamo's tokenized path bypasses that layer
and sends directly to the scheduler, where the same condition can surface only after HTTP 200
has been committed. Engine-published
TokenBudgetrestores the backend's public admissionsemantics without hard-coding one policy for every engine.
TensorRT-LLM inferred-limit caveat
When
max_seq_lenis explicit, Dynamo publishes that limit and can reject prompt overflowbefore backend dispatch. When it is omitted, TensorRT-LLM 1.3.0rc23 infers and may clamp
the realized limit inside its initialized executor worker, but does not expose that value
through the parent
LLMAPI in the normal IPC/RPC execution path. Dynamo therefore omitsthe token-budget capability and delegates overflow validation to TensorRT-LLM for this case.
The streaming preflight still propagates the backend rejection as HTTP 400 before SSE headers,
but default inferred-limit deployments do not receive frontend token-count admission or a
populated
context_length. Dynamo intentionally does not duplicate TensorRT-LLM model-specificinference; a future TRT-LLM realized-limit getter would let this path publish the same metadata.
Test matrix
InvalidArgumentreturns JSON HTTP 400 before SSE headersvLLM comparison
Using standalone vLLM and vLLM behind Dynamo with Qwen3-0.6B and
max_model_len=64:Standalone vLLM's Responses
truncation=autobehavior is a separate prompt-transformationcapability. This PR does not emulate it in the frontend.
Related work
This reconciles #11570. It includes that PR's Responses
max_output_tokenshandoff andlegacy Completions coverage, while adding:
--allow-auto-truncateCo-authored coverage from Tianhe Zhang is credited in commit
8628dadd2b.Where should the reviewer start?
components/src/dynamo/common/token_budget.pyandlib/llm/src/local_model/runtime_config.rsdefine the cross-language contractlib/llm/src/preprocessor.rsapplies frontend rejection policylib/llm/tests/preprocessor.rsand backend runtime-metadata tests capture the matrixRelated Issues
This PR is NOT linked to an issue:
Related pull request: #11570.
Validation
cargo fmt --all -- --checkcargo clippy -p dynamo-llm --lib --tests -- -D warningsdynamo-llmTokenBudget unit testscargo test -p dynamo-llm --test preprocessor context_length_validation(9 passed)cargo test -p dynamo-llm --test http-service test_http_service(1 passed)Live backend parity validation
A pre-merge GPU differential test starts the native OpenAI server and Dynamo with the same
backend, model, and context limit. It pins upstream behavior and Dynamo behavior for a
within-budget request, streaming output overflow, and streaming prompt overflow. Accepted
clamp/truncate cases also compare token usage, so HTTP 200 alone cannot mask policy drift.
The TensorRT-LLM transport difference is intentional: native TensorRT-LLM currently commits
streaming headers before prompt-length validation. The prompt-rejection capability lets
Dynamo preserve the engine decision while returning the actionable client status before
headers.