Skip to content

fix(trtllm): return 400 for guided decoding grammar rejections - #18311

Merged
indrajit96 merged 6 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5470-trtllm-async-grammar-error-client-4xx-9d69b7416839
Nov 7, 2026
Merged

indrajit96 merged 6 commits into
ai-dynamo:mainfrom
glamr-agent:dyn-5470-trtllm-async-grammar-error-client-4xx-9d69b7416839

Conversation

@glamr-agent

@glamr-agent glamr-agent commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Failure behavior

With TensorRT-LLM guided decoding on (--guided-decoding-backend xgrammar), a /v1/chat/completions request whose response_format holds an invalid JSON Schema type ({"type":"invalid_type"}) or a malformed regex ([unclosed[) gets HTTP 500. QA reported it on the release/1.7.0 RC1 TensorRT-LLM runtime image. The frontend logs backend asserted status 500: Guided decoding error: ... Unsupported type "invalid_type".

TensorRT-LLM compiles the grammar after the request is submitted. When compilation fails, the engine fails only that request and raises an error while the worker reads its results. TRT-LLM flattens the original exception into a message prefixed with Guided decoding error:. That prefix also covers matcher and mask failures, so it does not by itself identify bad input. The Dynamo TensorRT-LLM worker turned every such error into a generic error result, and the frontend reports those as 500. The defect is in the Dynamo worker; the engine rejects the grammar correctly.

Reproduction Without the fix With the fix Note
Exact failure on the reported setup Not run — Needs one H100 with the 1.7.0-rc.1 TensorRT-LLM runtime image and the QA suite; the coding run had one A100 in a development image built from main.
Exact failure on a similar setup ❌ Exact failure ✅ Passed A live frontend and TensorRT-LLM 1.4.0rc2 worker with xgrammar on one A100, serving Qwen/Qwen3-0.6B. Without the fix, the invalid schema type and the malformed regex both returned HTTP 500 with the ticket's body; with it, both returned 400, and valid-schema requests returned 200 before and after each, with the worker still serving.

Proposed fix

Before output begins, the TRT-LLM worker now asks guided_decoding_error_to_invalid_argument in errors.py to recognize xgrammar's compiler diagnostics for an unsupported schema type or an unclosed regex character class. Only these known rejections become InvalidArgument, which the frontend reports as 400 with the engine's message. Ambiguous guided-decoding errors remain 500. All errors after output begins retain the existing error chunk.

This is a temporary string-based workaround until TRT-LLM preserves a typed error or code across the executor boundary. It deliberately leaves unrecognized compilation errors as 500. The adapter matches the compiler file, diagnostic text, and location; the guided-decoding prefix alone is insufficient.

Case Before After
Invalid schema type or malformed regex 500 Internal server error 400 Bad Request with the grammar error
Same, streamed SSE error event, code 500 SSE error event, code 400
Ambiguous guided decoding failure before output Error result, 500 Unchanged
Guided decoding error after output started Error result, 500 Unchanged
Other engine errors while streaming results Error result, 500 Unchanged
Valid schema 200 Unchanged

The bug was reported on release/1.7.0, which has the same code, so a cherry-pick there is wanted after this merges.

Core change: about 18 lines. The logic is confined to the TRT-LLM backend:

  • components/src/dynamo/trtllm/errors.py recognizes the two verified xgrammar compilation diagnostics and preserves the engine message.
  • HandlerBase.generate_locally calls the adapter only before output begins. Other iteration errors keep their existing handling.

The remaining changes extend the existing handler regression tests and document the workaround.

Validation

The existing unit tests in components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py now cover both reported inputs, compiler paths and appended traces, prefixed internal failures before output, and compiler diagnostics after output. They run in the existing pre_merge, gpu_1, unit, trtllm stage with a mocked result stream; they do not load a model. Runtime for the expanded cases has not been measured locally.

The recorded results below are historical validation of 32c5b2e, before the conservative adapter was added. They establish the reported reproduction, not validation of the new classification rules. Those rules are validated through remote CI.

Behavior checked Checked by Without the fix With the fix
A grammar rejection raised while results stream becomes a client error, not a shutdown New unit test ❌ Failed: did not raise InvalidArgument ✅ Passed
A guided decoding error after a token was sent keeps the error chunk and does not shut down New unit test ✅ Passed ✅ Passed
Other errors while results stream keep their old handling: an error chunk, or shutdown for a fatal error Changed unit test (docstring only); it expected the same before this PR ✅ Passed ✅ Passed
An invalid schema type and a malformed regex return 400, and valid schemas still return 200 before and after them Live run on one A100, also the reproduction above ❌ Exact failure ✅ Passed; both returned 400, three valid-schema requests returned 200
A streamed request with an invalid schema type ends with a client-error event Live run on one A100 ❌ Failed: SSE error event with code 500 ✅ Passed; SSE error event with code 400

Relates to #8342, which reworks the same error branch in the worker for a different purpose; the two pull requests will conflict textually.

Refs DYN-5470.

Summary by CodeRabbit

  • Bug Fixes
    • Recognized schema and regex compilation rejections before any output now return an invalid-argument response. Ambiguous guided-decoding errors remain server errors.
    • If a guided-decoding error occurs after output has started, the generated output is preserved and the error is reported in a subsequent response chunk.

…ration

TensorRT-LLM reports a request's grammar compilation failure (invalid JSON
Schema type, malformed regex) as a RequestError while results are iterated.
The handler turned it into an error chunk, so the frontend answered 500.
Raise InvalidArgument for RequestErrors with TRT-LLM's "Guided decoding
error:" prefix so the client gets 400. Other iteration-time RequestErrors
keep the current error chunk, because the engine also uses RequestError for
failures that affect every active request.

Refs DYN-5470

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
…ut token

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent requested a review from a team as a code owner November 6, 2026 10:25
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 6, 2026 10:25 — with GitHub Actions Inactive
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 6, 2026 10:25 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor backend::trtllm Relates to the trtllm backend labels Nov 6, 2026
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: Initial validation snapshot.

Validation status: complete

Check Result Scope or reason
Starts Dynamo with TensorRT-LLM on one GPU and sends a real request Passed (exit 0) Commit 32c5b2e, one A100, TensorRT-LLM 1.4.0rc2 with xgrammar, Qwen/Qwen3-0.6B. Without the fix the invalid schema type and malformed regex returned 500; with it they returned 400, and valid-schema requests returned 200 before and after them.

Main gap: the exact reported setup (H100, 1.7.0-rc.1 runtime image, QA suite) was not run. Checks ran on 32c5b2e; the head ae8862c only shortens two code comments in handler_base.py.

Detailed evidence — validation complete

Left out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1, passed), lint and formatting (Check 2, passed), and the pre_merge unit tests in test_trtllm_handler_base.py (Check 3, passed: 99 passed; the new before-output test failed without the fix).

Evidence tally: [4/4 validated]

Advisory AI assessment: sound

Validation result: pass — on commit 32c5b2e, on one A100 with TensorRT-LLM 1.4.0rc2 and xgrammar, the invalid schema type and malformed regex returned HTTP 500 without the fix and 400 with it, valid-schema requests returned 200 before and after them with the worker still serving; the new before-output test failed without the fix and passed with it, the new after-output test failed on the first-round handler and passed with the fix, and the build, lint, and all 99 tests in the file passed

Evidence audit: complete [4/4 validated] — the command report comes from recorded runs.

Commands and results [4/4 validated]

Generated from the commands recorded during this run.

Check 1

Builds the changed Dynamo source and confirms that Python can import its compiled extension.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 2

Checks the changed files with the repository's fast lint and formatting commands.

Result: Passed (exit 0)

Command:

bash -c 'set -e; python3 -m py_compile components/src/dynamo/trtllm/request_handlers/handler_base.py components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py && echo py_compile OK; pre-commit run --files components/src/dynamo/trtllm/request_handlers/handler_base.py components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py --hook-stage manual'

Check 3

Runs the relevant GPU-dependent Python tests against the changed source.

Result: Passed (exit 0)

Command:

/opt/dynamo/venv/bin/python -m pytest components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py --timeout=120 -v

Check 4

Starts Dynamo with TensorRT-LLM on one GPU and sends a real request.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

HandlerBase now handles guided-decoding RequestErrors based on whether output tokens have been recorded. Tests cover errors before output and errors after output.

Changes

Guided-decoding error handling

Layer / File(s) Summary
Classify guided-decoding errors
components/src/dynamo/trtllm/request_handlers/handler_base.py, components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py
HandlerBase raises InvalidArgument for a guided-decoding RequestError before output begins. After output begins, it retains the output chunk and emits an error chunk. Tests cover both cases and clarify handling of other iteration errors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to ae886

The documented error-handling paths are covered, and no actionable issue remains before merge.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning The description misstates the key change. It says the worker recognizes only two compiler diagnostics through an adapter in errors.py. The change summary instead describes a prefix-based check in hand… Update the proposed fix and validation sections to match the implemented prefix-based classification and the tests. Remove or correct claims about errors.py and conservative diagnostic matching.
Passed Passed checks: 4 passed

Check name Status Explanation
Title check Passed The title clearly describes the main change: returning HTTP 400 for guided decoding grammar rejections.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 2 files.

Full details: Description check

Explanation

The description misstates the key change. It says the worker recognizes only two compiler diagnostics through an adapter in errors.py. The change summary instead describes a prefix-based check in handler_base.py.

Resolution

Update the proposed fix and validation sections to match the implemented prefix-based classification and the tests. Remove or correct claims about errors.py and conservative diagnostic matching.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between fa98520 and ae8862c.

Files selected for processing: 2

  • components/src/dynamo/trtllm/request_handlers/handler_base.py
  • components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

factory: @dynamo-ops please run full CI for ae8862c

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test ae8862c

@devin-ai-integration

devin-ai-integration Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 42274387407 (attempt 1) on 75e012d9bb

Gate checks: ❌ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework Build
TRT-LLM ❌ 1 ✅ 5
Other Jobs
frontend ❌ 1 ✅ 1
Operator ❌ 1

Failure details

3 jobs failed, all for the same reason: the Azure Container Registry (ACR) rejected every image push with unknown: The operation is disallowed on this registry, repository or image. All builds and tests that ran passed. This is a registry/infrastructure problem: once the ACR push permission or lock is fixed, rerunning the failed jobs should be enough.

❌ Operator: ACR push disallowed during Build and push operator

Job: Operator · Failed step: Build and push operator · Logs: gh run view --job 128087043770 -R ai-dynamo/dynamo --log-failed

ERROR: failed to build: failed to solve: failed to push ***/ai-dynamo/dynamo: unknown: The operation is disallowed on this registry, repository or image. View troubleshooting steps at https://aka.ms/acr/faq/#why-does-my-pull-or-push-request-fail-with-disallowed-operation-
##[error]Process completed with exit code 1.

The operator image built, but the final push to ACR was rejected as a disallowed operation.

❌ frontend / Copy to ACR cuda, amd64: ACR blob upload disallowed

Job: frontend / Copy to ACR cuda, amd64 · Failed step: Copy image to target registry · Logs: gh run view --job 128092033121 -R ai-dynamo/dynamo --log-failed

level=fatal msg="copying system image from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. ..."
⚠️ Attempt 1 failed, retrying in 10s...
⚠️ Attempt 2 failed, retrying in 20s...
❌ All 3 attempts failed

All 3 copy attempts failed because ACR refused the layer upload.

❌ trtllm-runtime / Copy to ACR cuda13.1: ACR blob upload disallowed

Job: trtllm-runtime / Copy to ACR cuda13.1 · Failed step: Copy image to target registry · Logs: gh run view --job 128093440001 -R ai-dynamo/dynamo --log-failed

level=fatal msg="copying image 1/4 from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. ..."
⚠️ Attempt 2 failed, retrying in 20s...
❌ All 3 attempts failed
##[error]Process completed with exit code 1.

The copy failed the same way as the frontend job: ACR refused the layer upload on all 3 attempts.

For agents
{"pr": 18311, "run_id": 42274387407, "run_attempt": 1, "head_sha": "75e012d9bbec11ba42f819e815fa71ccd0909174",
 "failures": [
  {"job": "Operator", "job_id": 128087043770, "failed_step": "Build and push operator", "signature": "ACR push: The operation is disallowed on this registry, repository or image", "tests": [], "log_cmd": "gh run view --job 128087043770 -R ai-dynamo/dynamo --log-failed"},
  {"job": "frontend / Copy to ACR cuda, amd64", "job_id": 128092033121, "failed_step": "Copy image to target registry", "signature": "ACR blob upload: The operation is disallowed on this registry, repository or image", "tests": [], "log_cmd": "gh run view --job 128092033121 -R ai-dynamo/dynamo --log-failed"},
  {"job": "trtllm-runtime / Copy to ACR cuda13.1", "job_id": 128093440001, "failed_step": "Copy image to target registry", "signature": "ACR blob upload: The operation is disallowed on this registry, repository or image", "tests": [], "log_cmd": "gh run view --job 128093440001 -R ai-dynamo/dynamo --log-failed"}
 ]}

Posted automatically by Devin for run 42274387407. Updated on every full-CI run of this PR.

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent
glamr-agent temporarily deployed to external_collaborator November 6, 2026 17:30 — with GitHub Actions Inactive
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 97bb75a

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 97bb75a

Signed-off-by: GLAMR <svc-glamr@nvidia.com>
@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: @dynamo-ops please run full CI for 75e012d

@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 75e012d

@glamr-agent

Copy link
Copy Markdown
Collaborator Author

nursery: I changed nothing for CodeRabbit's description check. The description matches commit 75e012d9bbec11ba42f819e815fa71ccd0909174. errors.py contains guided_decoding_error_to_invalid_argument, which recognizes the unsupported schema type and unclosed regex character class diagnostics. HandlerBase.generate_locally calls that adapter before output begins, and the handler tests cover both diagnostics and ambiguous failures. CodeRabbit's change summary names ae8862cb69359098a25d3b1b08b9db4227b8d9c4, which predates this adapter. Changing the description back to a broad prefix check would describe the older code.

@indrajit96
indrajit96 self-requested a review November 7, 2026 02:23

@indrajit96 indrajit96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Thanks @glamr-agent

@indrajit96
indrajit96 temporarily deployed to external_collaborator November 7, 2026 02:26 — with GitHub Actions Inactive
@indrajit96

Copy link
Copy Markdown
Contributor

/ok to test 664e325

@copy-pr-bot

copy-pr-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown

/ok to test 664e325

@indrajit96, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@indrajit96

Copy link
Copy Markdown
Contributor

/ok to test 664e325

@indrajit96
indrajit96 enabled auto-merge (squash) November 7, 2026 02:28
@indrajit96
indrajit96 merged commit 20a9321 into ai-dynamo:main Nov 7, 2026
110 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 7, 2026
Absorbs two upstream commits, neither on a router-owned path, no conflicts:
- ai-dynamo#18311 fix(trtllm): return 400 for guided decoding grammar rejections: new components/src/dynamo/trtllm/errors.py, handler_base.py raises InvalidArgument before any output, two new tests in test_trtllm_handler_base.py (Python only, merged in place).
- ai-dynamo#18324 fix(frontend): echo null description on Responses function tools: echoed_tools_json in lib/llm/src/protocols/openai/responses/template_tools.rs used by NvResponse serialization and the stream converter's lifecycle snapshots, one new unit test in stream_converter.rs and one in tests/native_tools.rs (lib/llm Responses protocol code, not kv_router; merged in place).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devivasudevan pushed a commit to devivasudevan/dynamo that referenced this pull request Nov 7, 2026
…namo#18311)

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
Co-authored-by: Indrajit Bhosale <iamindrajitb@gmail.com>
nv-nmailhot pushed a commit that referenced this pull request Nov 9, 2026
…ar rejections (#18311) (#18356)

Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
Co-authored-by: Indrajit Bhosale <iamindrajitb@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 664e3257 Deployed Nov 7, 2026 by indrajit96 via ok-to-test #47933
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::trtllm Relates to the trtllm backend external-contribution Pull request is from an external contributor fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants