Skip to content

fix(trtllm): escalate fatal CUDA errors from RequestError to worker shutdown - #8342

Closed
nvyutwu wants to merge 3 commits into
ai-dynamo:mainfrom
nvyutwu:yutwu/fix-trtllm-fatal-cuda-errors
Closed

nvyutwu wants to merge 3 commits into
ai-dynamo:mainfrom
nvyutwu:yutwu/fix-trtllm-fatal-cuda-errors

Conversation

@nvyutwu

@nvyutwu nvyutwu commented Apr 18, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

TRT-LLM raises unrecoverable CUDA errors (cudaErrorNvlinkUncorrectable, uncorrectable ECC, GPU-off-bus, device assert, illegal address, etc.) wrapped inside RequestError. The trtllm handler was downgrading these to a per-request WARN and yielding an error chunk, leaving the worker alive and accepting new traffic in a broken engine state. This PR escalates matching errors to a worker restart so Kubernetes can replace the pod.

Details:

  • New FATAL_CUDA_ERROR_PATTERNS regex at module level listing unrecoverable CUDA error strings (NVLink, ECC, no-device, illegal/misaligned address, device assert, launch timeout, hardware stack, cudaErrorUnknown). re.IGNORECASE covers casing variants.
  • cudaErrorMemoryAllocation (OOM) is intentionally NOT in the list — TRT-LLM aborts the offending request and frees KV cache, so the engine stays healthy. A test pins this behavior.
  • Extracted the except RequestError branch body into HandlerBase._handle_per_request_error(error, request_id) so the escalation path is directly unit-testable. On a fatal match it logs ERROR with traceback, best-effort yields the error chunk, then awaits _initiate_shutdown() (which calls os._exit(1) after engine cleanup). Benign errors keep the existing WARN-and-yield behavior.
  • Tests:
    • TestFatalCudaErrorPatterns — 9 fatal strings match, 5 benign (including OOM and empty string) do not.
    • TestHandlePerRequestError — fatal triggers _initiate_shutdown, benign doesn't, OOM doesn't.

Temporary workaround: once TRT-LLM exposes a first-class health-check API in LLM-API (NVIDIA/TensorRT-LLM#4513), the worker should query that instead of scraping error strings and this block can be reverted.

Where should the reviewer start?

  • `components/src/dynamo/trtllm/request_handlers/handler_base.py`
    • `FATAL_CUDA_ERROR_PATTERNS` (near the top)
    • `HandlerBase._handle_per_request_error` (right after `_initiate_shutdown`)
    • `except RequestError` branch in `_generate_locally_impl` (now a one-liner delegating to the helper)
  • `components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py`
    • `TestFatalCudaErrorPatterns`, `TestHandlePerRequestError`

Related Issues:

Testing

Validated locally in the trtllm-runtime dev container:

  • Unit tests: 49/49 pass (no regression from the `_generate_locally_impl` refactor).
  • Direct shutdown-body test: fatal → `os._exit(1)`, benign → clean yield, OOM → clean yield.
  • Real `_generate_locally_impl` path with mocked engine: confirmed the full `generate_async` raises → `except RequestError` → `_handle_per_request_error` → `_initiate_shutdown` → `engine.cleanup()` → `os._exit(1)` chain fires on fatal messages only.

Real NVLink-fault reproduction is not feasible on available hardware.


Open in Devin Review

Summary by CodeRabbit

  • Bug Fixes

    • Enhanced error handling for CUDA failures with intelligent classification of fatal versus recoverable errors, enabling proper system recovery and shutdown for unrecoverable conditions.
  • Tests

    • Added comprehensive test coverage for fatal error detection and handler behavior across various error scenarios.

@nvyutwu
nvyutwu requested review from a team as code owners April 18, 2026 21:43
@copy-pr-bot

copy-pr-bot Bot commented Apr 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi nvyutwu! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor backend::trtllm Relates to the trtllm backend labels Apr 18, 2026
…hutdown

TRT-LLM raises cudaErrorNvlinkUncorrectable and friends as RequestError,
which handler_base.py was downgrading to a per-request WARN. That left the
worker accepting traffic in a broken state after an unrecoverable GPU
fault (NVLink failure, uncorrectable ECC, GPU-off-bus, device assert,
illegal address, etc.).

Add a module-level FATAL_CUDA_ERROR_PATTERNS regex covering unrecoverable
CUDA error strings. Extract the except-RequestError branch into
_handle_per_request_error so the escalation path is directly unit-testable;
when the message matches, the helper logs ERROR, best-effort yields the
error chunk, then awaits _initiate_shutdown() so Kubernetes restarts the
pod. Benign RequestErrors keep the existing WARN-and-yield behavior.
cudaErrorMemoryAllocation (OOM) is intentionally excluded — TRT-LLM
aborts the offending request and frees KV cache, so the engine stays
healthy.

This is a temporary workaround. Once TRT-LLM exposes a first-class health
check API in LLM-API (NVIDIA/TensorRT-LLM#4513), the worker should query
that instead of scraping error messages, and this block can be reverted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuting Wu (DLAlgo) <yutwu@nvidia.com>
@nvyutwu
nvyutwu force-pushed the yutwu/fix-trtllm-fatal-cuda-errors branch from 108484b to 375327d Compare April 18, 2026 21:44
@coderabbitai

coderabbitai Bot commented Apr 18, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

This change introduces fatal CUDA error classification and handling in request handlers. A compiled regex pattern identifies fatal CUDA errors, and a new async helper method logs errors appropriately and escalates fatal errors to trigger worker shutdown instead of handling them as recoverable request failures.

Changes

Cohort / File(s) Summary
Error Classification & Handling
components/src/dynamo/trtllm/request_handlers/handler_base.py
Added FATAL_CUDA_ERROR_PATTERNS regex constant to classify fatal CUDA errors. Introduced _handle_per_request_error async method that logs errors at error/warning level based on fatality classification, emits error completion chunks, and escalates fatal errors to _initiate_shutdown. Modified _generate_locally_impl exception handler to delegate to the new method.
Test Coverage
components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py
Added TestFatalCudaErrorPatterns to validate regex matching of fatal vs. non-fatal CUDA error strings. Added TestHandlePerRequestError async tests verifying error handling behavior for fatal errors (triggers shutdown), recoverable errors (no shutdown), and specific cases like cudaErrorMemoryAllocation (treated as recoverable).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: escalating fatal CUDA errors from request-level to worker-level handling via shutdown.
Description check ✅ Passed The description covers all required template sections with substantial detail: Overview explains the problem, Details describes the implementation, Where to start directs reviewers, and Related Issues links the GitHub reference.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py (1)

711-748: Consider asserting emit-before-shutdown ordering.

The tests correctly verify that _initiate_shutdown is awaited for fatal errors and that a chunk is emitted, but they do not verify the ordering contract — i.e., the client's error chunk must be yielded before the process-exiting shutdown is invoked. Since _initiate_shutdown in production calls os._exit(1), any future refactor that reorders these two steps would silently drop the client notification, and the existing tests would still pass.

🧪 Optional ordering assertion
     async def test_fatal_cuda_error_triggers_shutdown(self):
         from tensorrt_llm.executor.utils import RequestError
 
         handler = self._make_handler()
+        events: list[str] = []
+        original_shutdown = handler._initiate_shutdown
+
+        async def _record_shutdown(e):
+            events.append("shutdown")
+            return await original_shutdown(e)
+
+        handler._initiate_shutdown.side_effect = _record_shutdown
         err = RequestError("CUDA error: cudaErrorNvlinkUncorrectable")
-        chunks = [c async for c in handler._handle_per_request_error(err, "req-1")]
+        chunks = []
+        async for c in handler._handle_per_request_error(err, "req-1"):
+            events.append("yield")
+            chunks.append(c)
 
         assert len(chunks) == 1
         assert "cudaErrorNvlinkUncorrectable" in chunks[0]["finish_reason"]["error"]
         assert chunks[0]["token_ids"] == []
         handler._initiate_shutdown.assert_awaited_once_with(err)
+        assert events == ["yield", "shutdown"]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py` around lines
711 - 748, Update the tests (e.g., test_fatal_cuda_error_triggers_shutdown and
similar) to assert that the client's error chunk is yielded before
_initiate_shutdown is awaited: instrument handler._initiate_shutdown (AsyncMock
or side_effect) to append a marker/timestamp when invoked, collect the emitted
chunks from _handle_per_request_error into a list first, then assert the
emission happened before the shutdown marker (or compare timestamps/sequence
indices). Reference the test helper _make_handler(), the method under test
_handle_per_request_error, and the shutdown method _initiate_shutdown to find
where to add the ordering assertion.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@components/src/dynamo/trtllm/request_handlers/handler_base.py`:
- Around line 940-961: In _handle_per_request_error, don't silently swallow
exceptions raised by the yield; replace the bare except block with logging that
records the exception (include exc_info=True) so downstream failures (e.g.,
serialization errors) are visible; use logging.warning (or logging.debug for
non-fatal) with context about request_id and error_msg and then continue
best-effort behavior, keeping the fatal-path call to _initiate_shutdown and the
FATAL_CUDA_ERROR_PATTERNS check intact.

---

Nitpick comments:
In `@components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py`:
- Around line 711-748: Update the tests (e.g.,
test_fatal_cuda_error_triggers_shutdown and similar) to assert that the client's
error chunk is yielded before _initiate_shutdown is awaited: instrument
handler._initiate_shutdown (AsyncMock or side_effect) to append a
marker/timestamp when invoked, collect the emitted chunks from
_handle_per_request_error into a list first, then assert the emission happened
before the shutdown marker (or compare timestamps/sequence indices). Reference
the test helper _make_handler(), the method under test
_handle_per_request_error, and the shutdown method _initiate_shutdown to find
where to add the ordering assertion.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 75e3d37e-e3b4-48a5-8f0a-7d677affb41c

📥 Commits

Reviewing files that changed from the base of the PR and between d94b350 and 375327d.

📒 Files selected for processing (2)
  • components/src/dynamo/trtllm/request_handlers/handler_base.py
  • components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py

@nvyutwu
nvyutwu force-pushed the yutwu/fix-trtllm-fatal-cuda-errors branch from a26eb71 to 225b277 Compare April 18, 2026 23:49
- handler_base.py: log (at debug) instead of silently swallowing an
  Exception raised while emitting the RequestError chunk, so a
  serialization failure on the last client-visible signal before
  os._exit(1) remains observable.
- test_trtllm_handler_base.py: instrument _initiate_shutdown's side_effect
  and record each yield to assert the error chunk is emitted before
  shutdown is awaited (guards against reordering the production code).
  Add docstrings to new test methods for the 80% coverage gate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Yuting Wu (DLAlgo) <yutwu@nvidia.com>
@nvyutwu
nvyutwu force-pushed the yutwu/fix-trtllm-fatal-cuda-errors branch from f4e8041 to b8fa311 Compare April 18, 2026 23:55
@nvyutwu

nvyutwu commented Apr 21, 2026

Copy link
Copy Markdown
Contributor Author

@nv-yna would you mind taking a look when you have a chance? This escalates fatal CUDA errors (e.g. cudaErrorNvlinkUncorrectable) surfaced as RequestError from the TRT-LLM executor into a worker shutdown, so the orchestrator restarts the worker rather than letting it continue in a broken state. Tip is 4a4e9f2cdf. Thanks!

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information.

@github-actions github-actions Bot closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::trtllm Relates to the trtllm backend external-contribution Pull request is from an external contributor fix size/L Stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant