Skip to content

fix(frontend): reject over-context requests before streaming - #12092

Merged
Aphoh merged 19 commits into
mainfrom
warnold/fix-streaming-context-overflow-status
Aug 7, 2026
Merged

Aphoh merged 19 commits into
mainfrom
warnold/fix-streaming-context-overflow-status

Conversation

@Aphoh

@Aphoh Aphoh commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Dynamo currently commits HTTP 200 for streaming requests before a backend can reject an
over-context token budget. That turns a client error into an inline server error, triggers
meaningless retries, and distorts monitoring.

This change makes each engine publish the request overflows that the frontend may reject
early. The shared OpenAI preprocessor applies only those advertised rejections, before
backend dispatch and before streaming headers are sent. Backend-owned clamping and
truncation remain in the backend.

Summary

  • add a minimal engine/frontend capability:

    TokenBudget {
        combined_limit: N,
        reject_prompt_overflow: bool,
        reject_total_overflow: bool,
    }
    

    A false flag delegates that overflow dimension to the backend. Missing or malformed
    metadata also delegates both dimensions, so unofficial backends and version-skewed
    deployments stay online.

  • publish the resolved backend behavior:

    Backend mode Reject prompt overflow early Reject total overflow early
    vLLM generation Yes Yes
    SGLang default Yes Yes
    SGLang --allow-auto-truncate No No
    SGLang total-token validation disabled Yes No
    TensorRT-LLM Yes when max_seq_len is explicit; otherwise delegated No
  • publish matching metadata for base and dynamic LoRA model cards; embedding-only and
    engine-less encode cards remain excluded where generation policy does not apply

  • reject exact over-budget requests in the canonical preprocessor for Chat Completions,
    Responses, and legacy Completions

  • defer prompt embeddings and multimodal lengths the frontend cannot prove

  • preserve explicit requests on delegated paths; the frontend does not reproduce backend
    clamping or truncation

  • preserve immediate streaming headers, SSE keep-alives, backend inactivity timeouts, and
    client cancellation for admitted requests

  • propagate typed InvalidArgument as JSON HTTP 400 and classify it as a validation failure
    in frontend metrics without changing the backend's wire message

  • document the capability and fail-open behavior for custom Python workers

Why frontend admission is needed

The engine remains the source of policy. Standalone SGLang rejects an over-budget request in
its tokenizer manager before starting a stream. Dynamo's tokenized path bypasses that layer
and sends directly to the scheduler, where the same condition can surface only after HTTP 200
has been committed. Engine-published TokenBudget restores the backend's public admission
semantics without hard-coding one policy for every engine.

TensorRT-LLM inferred-limit caveat

When max_seq_len is explicit, Dynamo publishes that limit and can reject prompt overflow
before backend dispatch. When it is omitted, TensorRT-LLM 1.3.0rc23 infers and may clamp
the realized limit inside its initialized executor worker, but does not expose that value
through the parent LLM API in the normal IPC/RPC execution path. Dynamo therefore omits
the token-budget capability and delegates overflow validation to TensorRT-LLM for this case.
The streaming preflight still propagates the backend rejection as HTTP 400 before SSE headers,
but default inferred-limit deployments do not receive frontend token-count admission or a
populated context_length. Dynamo intentionally does not duplicate TensorRT-LLM model-specific
inference; a future TRT-LLM realized-limit getter would let this path publish the same metadata.

Test matrix

  • publisher mappings for vLLM, the three distinct SGLang modes, and TensorRT-LLM
  • exact combined-limit acceptance and one-token overflow rejection
  • independent prompt-overflow and total-overflow delegation
  • prompt-at-limit rejection and omitted-output handling
  • missing and malformed metadata fail open without mutating the request
  • prompt embeddings and inexact multimodal requests defer to the backend
  • Chat, Responses, and legacy Completions wiring
  • streaming typed InvalidArgument returns JSON HTTP 400 before SSE headers

vLLM comparison

Using standalone vLLM and vLLM behind Dynamo with Qwen3-0.6B and max_model_len=64:

Case Standalone vLLM Released Dynamo + vLLM Patched Dynamo + vLLM
Chat: 48 input + 17 output 400 200 SSE, output clamped 400 JSON
Chat: 48 input + 16 output 200 SSE 200 SSE 200 SSE
Chat: 64 input, output omitted 400 400 400
Completions: 48 input + 17 output 400 200 SSE, output clamped 400 JSON
Responses disabled: explicit combined overflow 400 accepted when prompt itself fits 400 JSON

Standalone vLLM's Responses truncation=auto behavior is a separate prompt-transformation
capability. This PR does not emulate it in the frontend.

Related work

This reconciles #11570. It includes that PR's Responses max_output_tokens handoff and
legacy Completions coverage, while adding:

  • backend-published admission capability instead of a frontend-global context rule
  • independent prompt and combined-total rejection/delegation
  • vLLM parity for explicit combined overflow
  • TensorRT-LLM's prompt-reject/total-defer contract
  • reserved-token-aware SGLang limits and --allow-auto-truncate
  • base and dynamic LoRA registration coverage
  • inexact multimodal and prompt-embedding deferral
  • the HTTP 400/metrics contract without delaying valid streaming headers

Co-authored coverage from Tianhe Zhang is credited in commit 8628dadd2b.

Where should the reviewer start?

  • components/src/dynamo/common/token_budget.py and
    lib/llm/src/local_model/runtime_config.rs define the cross-language contract
  • lib/llm/src/preprocessor.rs applies frontend rejection policy
  • backend registration code publishes vLLM, SGLang, and TensorRT-LLM behavior
  • lib/llm/tests/preprocessor.rs and backend runtime-metadata tests capture the matrix

Related Issues

This PR is NOT linked to an issue:

  • Confirmed — no related issue

Related pull request: #11570.

Validation

  • cargo fmt --all -- --check
  • cargo clippy -p dynamo-llm --lib --tests -- -D warnings
  • focused dynamo-llm TokenBudget unit tests
  • cargo test -p dynamo-llm --test preprocessor context_length_validation (9 passed)
  • cargo test -p dynamo-llm --test http-service test_http_service (1 passed)
  • repository-pinned Black and Ruff on changed Python files
  • SGLang 1.3.0 container policy and LoRA matrix
  • TensorRT-LLM runtime-metadata tests
  • focused vLLM publisher validation
  • original SGLang/Qwen3-8B streaming overflow reproduction
  • standalone vLLM vs released/patched Dynamo comparison
  • thermonuclear maintainability review, with findings incorporated

Live backend parity validation

A pre-merge GPU differential test starts the native OpenAI server and Dynamo with the same
backend, model, and context limit. It pins upstream behavior and Dynamo behavior for a
within-budget request, streaming output overflow, and streaming prompt overflow. Accepted
clamp/truncate cases also compare token usage, so HTTP 200 alone cannot mask policy drift.

Backend mode Native behavior Dynamo behavior Focused CI
vLLM default Reject / Reject at request time Same HTTP 400 behavior passed
SGLang default Reject / Reject at request time Same HTTP 400 behavior passed
SGLang auto-truncate Clamp output / truncate prompt Same accepted result and token usage passed
TensorRT-LLM default Clamp output; prompt rejection arrives after HTTP 200 as an SSE error Same rejection semantics, moved to HTTP 400 preflight passed

The TensorRT-LLM transport difference is intentional: native TensorRT-LLM currently commits
streaming headers before prompt-length validation. The prompt-rejection capability lets
Dynamo preserve the engine decision while returning the actionable client status before
headers.

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
@Aphoh
Aphoh requested review from a team as code owners July 24, 2026 00:35
@github-actions github-actions Bot added fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

Streaming error preflight

Layer / File(s) Summary
Pre-check backend stream errors
lib/llm/src/http/service/openai.rs
chat_completions checks the first meaningful backend event before initializing streaming responses, classifies backend HTTP 400 errors as validation failures, and reuses the validated stream for non-streaming responses.
Streaming validation regression coverage
lib/llm/tests/http-service.rs
Adds an invalid-argument streaming engine and verifies HTTP 400 responses, structured error fields, validation metrics, and updated inflight expectations.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes the primary change: rejecting over-context requests before streaming begins.
Description check ✅ Passed The description explains the change, reviewer entry points, related-issue status, implementation details, and validation results.

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Comment thread lib/llm/src/http/service/openai.rs Resolved
@datadog-official

datadog-official Bot commented Jul 24, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 84.13%
• Overall Coverage: 48.11% (-2.36%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 34d6e9a | Docs | Datadog PR Page | Give us feedback!

Comment thread lib/llm/src/http/service/openai.rs Resolved
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
@Aphoh
Aphoh requested a review from a team as a code owner July 24, 2026 01:41
@github-actions github-actions Bot added the backend::sglang Relates to the sglang backend label Jul 24, 2026
@Aphoh Aphoh changed the title fix(frontend): return backend validation errors before streaming fix(frontend): reject over-context requests before streaming Jul 24, 2026
@rmccorm4

Copy link
Copy Markdown
Contributor

Related: #11570

Comment thread lib/llm/src/preprocessor.rs Resolved
Apply the engine-declared strict budget to legacy Completions and cover the Responses API token-budget handoff. Centralize admission so chat and completion paths share exact-length and deferral rules.\n\nCo-authored-by: Tianhe Zhang <tianhez@nvidia.com>

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Comment thread lib/llm/src/preprocessor.rs Resolved
Aphoh added 3 commits July 24, 2026 03:16
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
@Aphoh
Aphoh requested review from a team as code owners July 24, 2026 09:17
@github-actions github-actions Bot added the backend::vllm Relates to the vllm backend label Jul 24, 2026
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
…-context-overflow-status

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

# Conflicts:
#	components/src/dynamo/vllm/handlers.py
#	components/src/dynamo/vllm/tests/test_vllm_legacy_lora.py
#	lib/llm/src/http/service/openai.rs
#	lib/llm/src/preprocessor.rs
#	lib/llm/tests/http-service.rs
@Aphoh

Aphoh commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f4fe58a

Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from DevOps side

Aphoh added 2 commits August 6, 2026 19:42
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
…erflow-status' into warnold/fix-streaming-context-overflow-status

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
@Aphoh

Aphoh commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test a63d228

1 similar comment
@Aphoh

Aphoh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test a63d228

@Aphoh

Aphoh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 34d6e9a

@Aphoh
Aphoh merged commit 8957ec3 into main Aug 7, 2026
200 of 203 checks passed
@Aphoh
Aphoh deleted the warnold/fix-streaming-context-overflow-status branch August 7, 2026 07:37
jokerhaos pushed a commit to jokerhaos/dynamo that referenced this pull request Sep 11, 2026
… settable)

The SGLang chat processor dropped the reason on backend rejections on the
disaggregated SGLang path: an over-context ValueError from SGLang and a
LocalRouter queue-overflow rejection were both swallowed by the generic
`except Exception` and re-raised as `Unknown`, while the `is_error` and
`handle_engine_error` yield paths emitted bare dicts the binding misread as
completion chunks -- the client got an opaque 500 (or a bare SSE frame) with
no reason.

ai-dynamo#14063 added the equivalent handling to the vLLM chat processor
(classification + `as_error_envelope` + `backend_invalid_argument_to_http_error`)
but did not touch `sglang_processor.py`. The preflight guard from ai-dynamo#12092 also
does not gate this path: the worker's `token_budget` does not reach the
frontend MDC under disaggregation, so `validate_requested_token_budget` returns
early and the request still reaches the worker, which rejects it.

This mirrors ai-dynamo#14063 onto the SGLang path so the reason always reaches the
client, and sets the HTTP status where the HTTP layer can still set it:

- classify over-context messages to a 400-shaped error and LocalRouter
  queue-overflow rejections to a 429-shaped one (regex classifier
  `_classify_sglang_error`, which reuses `backend_invalid_argument_to_http_error`
  for the `BackendInvalidArgument: {json}` envelope);
- a classified error arriving before the first byte `raise`s `HttpError`. The
  resulting 400/429 status is set only where the HTTP layer can still set it:
  non-streaming requests, and streaming with
  `DYN_HTTP_PRE_COMMIT_ERROR_PEEK_MS` (the opt-in pre-commit error peek).
  Default streaming commits 200 up front, so a raise there surfaces as
  200 + SSE error event carrying the reason (still better than an opaque 500);
- a classified error arriving mid-stream, after the 200 is committed, is
  `yield`ed as an `as_error_envelope` SSE error event so the reason reaches the
  client instead of truncating the stream (`yielded_any` flag, set by the two
  flush `yield envelope` sites);
- add `except HttpError: raise` so a raised `HttpError` propagates to the HTTP
  boundary instead of being re-caught by `except Exception` and re-wrapped
  (mirrors ai-dynamo#14063's `except VLLMClientError: raise`);
- unclassified errors keep their existing behavior (`as_error_envelope` for
  `is_error` / missing `token_ids`; `raise Unknown` for genuine engine faults),
  so the common cases are unchanged.

Known limitation: making the 4xx status reach the default (peek-off) streaming
path needs the pre-commit error peek enabled for these validation errors -- a
Rust HTTP-service change, out of scope here and in the scope of DEP ai-dynamo#14354.

Updates the two affected `test_sglang_processor_unit.py` tests to the
tagged-envelope shape the binding now expects, and adds a `TestClassifySglangError`
unit suite covering over-context / queue-overflow / `BackendInvalidArgument` /
unclassified.

Validation: py_compile + ruff 0.5.2 check/format pass; runtime checks (21
assertions, all pass) via stub-loaded real `sglang_processor.py` + real
`frontend/utils.py` (PyO3/CUDA deps stubbed) -- pre-stream classified still
raise (429/400); mid-stream classified yield an error event with the reason
instead of truncating; unclassified paths unchanged.

Related to GitHub issue ai-dynamo#14354 (DEP: unified semantic error propagation), which
would subsume this once it lands.

Signed-off-by: joker <a1214096242@gamil.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants