Repository navigation
fix(trtllm): return 400 for guided decoding grammar rejections - #18311
Conversation
…ration TensorRT-LLM reports a request's grammar compilation failure (invalid JSON Schema type, malformed regex) as a RequestError while results are iterated. The handler turned it into an error chunk, so the frontend answered 500. Raise InvalidArgument for RequestErrors with TRT-LLM's "Guided decoding error:" prefix so the client gets 400. Other iteration-time RequestErrors keep the current error chunk, because the engine also uses RequestError for failures that affect every active request. Refs DYN-5470 Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
…ut token Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
|
👋 Hi glamr-agent! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
factory: Initial validation snapshot. Validation status: complete
Main gap: the exact reported setup (H100, Detailed evidence — validation completeLeft out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1, passed), lint and formatting (Check 2, passed), and the Evidence tally: [4/4 validated] Advisory AI assessment: sound Validation result: pass — on commit Evidence audit: complete [4/4 validated] — the command report comes from recorded runs. Commands and results [4/4 validated]Generated from the commands recorded during this run. Check 1Builds the changed Dynamo source and confirms that Python can import its compiled extension. Result: Passed ( Command: Not shown because the exact command contained private run data. Check 2Checks the changed files with the repository's fast lint and formatting commands. Result: Passed ( Command: bash -c 'set -e; python3 -m py_compile components/src/dynamo/trtllm/request_handlers/handler_base.py components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py && echo py_compile OK; pre-commit run --files components/src/dynamo/trtllm/request_handlers/handler_base.py components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py --hook-stage manual'Check 3Runs the relevant GPU-dependent Python tests against the changed source. Result: Passed ( Command:
Check 4Starts Dynamo with TensorRT-LLM on one GPU and sends a real request. Result: Passed ( Command: Not shown because the exact command contained private run data. |
|
No actionable comments were generated in the recent review. WalkthroughHandlerBase now handles guided-decoding RequestErrors based on whether output tokens have been recorded. Tests cover errors before output and errors after output. ChangesGuided-decoding error handling
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk:
|
|
factory: @dynamo-ops please run full CI for ae8862c |
|
/ok to test ae8862c |
❌ Dynamo PR CI failed — run 42274387407 (attempt 1) on
|
| Framework | Build |
|---|---|
| TRT-LLM | ❌ 1 ✅ 5 |
| Other | Jobs |
|---|---|
| frontend | ❌ 1 ✅ 1 |
| Operator | ❌ 1 |
Failure details
3 jobs failed, all for the same reason: the Azure Container Registry (ACR) rejected every image push with unknown: The operation is disallowed on this registry, repository or image. All builds and tests that ran passed. This is a registry/infrastructure problem: once the ACR push permission or lock is fixed, rerunning the failed jobs should be enough.
❌ Operator: ACR push disallowed during Build and push operator
Job: Operator · Failed step: Build and push operator · Logs: gh run view --job 128087043770 -R ai-dynamo/dynamo --log-failed
ERROR: failed to build: failed to solve: failed to push ***/ai-dynamo/dynamo: unknown: The operation is disallowed on this registry, repository or image. View troubleshooting steps at https://aka.ms/acr/faq/#why-does-my-pull-or-push-request-fail-with-disallowed-operation-
##[error]Process completed with exit code 1.
The operator image built, but the final push to ACR was rejected as a disallowed operation.
❌ frontend / Copy to ACR cuda, amd64: ACR blob upload disallowed
Job: frontend / Copy to ACR cuda, amd64 · Failed step: Copy image to target registry · Logs: gh run view --job 128092033121 -R ai-dynamo/dynamo --log-failed
level=fatal msg="copying system image from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. ..."
⚠️ Attempt 1 failed, retrying in 10s...
⚠️ Attempt 2 failed, retrying in 20s...
❌ All 3 attempts failed
All 3 copy attempts failed because ACR refused the layer upload.
❌ trtllm-runtime / Copy to ACR cuda13.1: ACR blob upload disallowed
Job: trtllm-runtime / Copy to ACR cuda13.1 · Failed step: Copy image to target registry · Logs: gh run view --job 128093440001 -R ai-dynamo/dynamo --log-failed
level=fatal msg="copying image 1/4 from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. ..."
⚠️ Attempt 2 failed, retrying in 20s...
❌ All 3 attempts failed
##[error]Process completed with exit code 1.
The copy failed the same way as the frontend job: ACR refused the layer upload on all 3 attempts.
For agents
{"pr": 18311, "run_id": 42274387407, "run_attempt": 1, "head_sha": "75e012d9bbec11ba42f819e815fa71ccd0909174",
"failures": [
{"job": "Operator", "job_id": 128087043770, "failed_step": "Build and push operator", "signature": "ACR push: The operation is disallowed on this registry, repository or image", "tests": [], "log_cmd": "gh run view --job 128087043770 -R ai-dynamo/dynamo --log-failed"},
{"job": "frontend / Copy to ACR cuda, amd64", "job_id": 128092033121, "failed_step": "Copy image to target registry", "signature": "ACR blob upload: The operation is disallowed on this registry, repository or image", "tests": [], "log_cmd": "gh run view --job 128092033121 -R ai-dynamo/dynamo --log-failed"},
{"job": "trtllm-runtime / Copy to ACR cuda13.1", "job_id": 128093440001, "failed_step": "Copy image to target registry", "signature": "ACR blob upload: The operation is disallowed on this registry, repository or image", "tests": [], "log_cmd": "gh run view --job 128093440001 -R ai-dynamo/dynamo --log-failed"}
]}Posted automatically by Devin for run 42274387407. Updated on every full-CI run of this PR.
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for 97bb75a |
|
/ok to test 97bb75a |
Signed-off-by: GLAMR <svc-glamr@nvidia.com>
|
nursery: @dynamo-ops please run full CI for 75e012d |
|
/ok to test 75e012d |
|
nursery: I changed nothing for CodeRabbit's description check. The description matches commit |
indrajit96
left a comment
There was a problem hiding this comment.
LGTM!
Thanks @glamr-agent
|
/ok to test 664e325 |
@indrajit96, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 664e325 |
Absorbs two upstream commits, neither on a router-owned path, no conflicts: - ai-dynamo#18311 fix(trtllm): return 400 for guided decoding grammar rejections: new components/src/dynamo/trtllm/errors.py, handler_base.py raises InvalidArgument before any output, two new tests in test_trtllm_handler_base.py (Python only, merged in place). - ai-dynamo#18324 fix(frontend): echo null description on Responses function tools: echoed_tools_json in lib/llm/src/protocols/openai/responses/template_tools.rs used by NvResponse serialization and the stream converter's lifecycle snapshots, one new unit test in stream_converter.rs and one in tests/native_tools.rs (lib/llm Responses protocol code, not kv_router; merged in place). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…namo#18311) Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com> Signed-off-by: GLAMR <svc-glamr@nvidia.com> Co-authored-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Failure behavior
With TensorRT-LLM guided decoding on (
--guided-decoding-backend xgrammar), a/v1/chat/completionsrequest whoseresponse_formatholds an invalid JSON Schema type ({"type":"invalid_type"}) or a malformed regex ([unclosed[) gets HTTP 500. QA reported it on therelease/1.7.0RC1 TensorRT-LLM runtime image. The frontend logsbackend asserted status 500: Guided decoding error: ... Unsupported type "invalid_type".TensorRT-LLM compiles the grammar after the request is submitted. When compilation fails, the engine fails only that request and raises an error while the worker reads its results. TRT-LLM flattens the original exception into a message prefixed with
Guided decoding error:. That prefix also covers matcher and mask failures, so it does not by itself identify bad input. The Dynamo TensorRT-LLM worker turned every such error into a generic error result, and the frontend reports those as 500. The defect is in the Dynamo worker; the engine rejects the grammar correctly.1.7.0-rc.1TensorRT-LLM runtime image and the QA suite; the coding run had one A100 in a development image built frommain.Qwen/Qwen3-0.6B. Without the fix, the invalid schema type and the malformed regex both returned HTTP 500 with the ticket's body; with it, both returned 400, and valid-schema requests returned 200 before and after each, with the worker still serving.Proposed fix
Before output begins, the TRT-LLM worker now asks
guided_decoding_error_to_invalid_argumentinerrors.pyto recognize xgrammar's compiler diagnostics for an unsupported schema type or an unclosed regex character class. Only these known rejections becomeInvalidArgument, which the frontend reports as 400 with the engine's message. Ambiguous guided-decoding errors remain 500. All errors after output begins retain the existing error chunk.This is a temporary string-based workaround until TRT-LLM preserves a typed error or code across the executor boundary. It deliberately leaves unrecognized compilation errors as 500. The adapter matches the compiler file, diagnostic text, and location; the guided-decoding prefix alone is insufficient.
The bug was reported on
release/1.7.0, which has the same code, so a cherry-pick there is wanted after this merges.Validation
The existing unit tests in
components/src/dynamo/trtllm/tests/test_trtllm_handler_base.pynow cover both reported inputs, compiler paths and appended traces, prefixed internal failures before output, and compiler diagnostics after output. They run in the existingpre_merge,gpu_1,unit,trtllmstage with a mocked result stream; they do not load a model. Runtime for the expanded cases has not been measured locally.The recorded results below are historical validation of
32c5b2e, before the conservative adapter was added. They establish the reported reproduction, not validation of the new classification rules. Those rules are validated through remote CI.InvalidArgumentRelates to #8342, which reworks the same error branch in the worker for a different purpose; the two pull requests will conflict textually.
Refs DYN-5470.
Summary by CodeRabbit