Repository navigation
Conversation
|
👋 Hi nvyutwu! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
…hutdown TRT-LLM raises cudaErrorNvlinkUncorrectable and friends as RequestError, which handler_base.py was downgrading to a per-request WARN. That left the worker accepting traffic in a broken state after an unrecoverable GPU fault (NVLink failure, uncorrectable ECC, GPU-off-bus, device assert, illegal address, etc.). Add a module-level FATAL_CUDA_ERROR_PATTERNS regex covering unrecoverable CUDA error strings. Extract the except-RequestError branch into _handle_per_request_error so the escalation path is directly unit-testable; when the message matches, the helper logs ERROR, best-effort yields the error chunk, then awaits _initiate_shutdown() so Kubernetes restarts the pod. Benign RequestErrors keep the existing WARN-and-yield behavior. cudaErrorMemoryAllocation (OOM) is intentionally excluded — TRT-LLM aborts the offending request and frees KV cache, so the engine stays healthy. This is a temporary workaround. Once TRT-LLM exposes a first-class health check API in LLM-API (NVIDIA/TensorRT-LLM#4513), the worker should query that instead of scraping error messages, and this block can be reverted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Yuting Wu (DLAlgo) <yutwu@nvidia.com>
108484b to
375327d
Compare
WalkthroughThis change introduces fatal CUDA error classification and handling in request handlers. A compiled regex pattern identifies fatal CUDA errors, and a new async helper method logs errors appropriately and escalates fatal errors to trigger worker shutdown instead of handling them as recoverable request failures. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py (1)
711-748: Consider asserting emit-before-shutdown ordering.The tests correctly verify that
_initiate_shutdownis awaited for fatal errors and that a chunk is emitted, but they do not verify the ordering contract — i.e., the client's error chunk must be yielded before the process-exiting shutdown is invoked. Since_initiate_shutdownin production callsos._exit(1), any future refactor that reorders these two steps would silently drop the client notification, and the existing tests would still pass.🧪 Optional ordering assertion
async def test_fatal_cuda_error_triggers_shutdown(self): from tensorrt_llm.executor.utils import RequestError handler = self._make_handler() + events: list[str] = [] + original_shutdown = handler._initiate_shutdown + + async def _record_shutdown(e): + events.append("shutdown") + return await original_shutdown(e) + + handler._initiate_shutdown.side_effect = _record_shutdown err = RequestError("CUDA error: cudaErrorNvlinkUncorrectable") - chunks = [c async for c in handler._handle_per_request_error(err, "req-1")] + chunks = [] + async for c in handler._handle_per_request_error(err, "req-1"): + events.append("yield") + chunks.append(c) assert len(chunks) == 1 assert "cudaErrorNvlinkUncorrectable" in chunks[0]["finish_reason"]["error"] assert chunks[0]["token_ids"] == [] handler._initiate_shutdown.assert_awaited_once_with(err) + assert events == ["yield", "shutdown"]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py` around lines 711 - 748, Update the tests (e.g., test_fatal_cuda_error_triggers_shutdown and similar) to assert that the client's error chunk is yielded before _initiate_shutdown is awaited: instrument handler._initiate_shutdown (AsyncMock or side_effect) to append a marker/timestamp when invoked, collect the emitted chunks from _handle_per_request_error into a list first, then assert the emission happened before the shutdown marker (or compare timestamps/sequence indices). Reference the test helper _make_handler(), the method under test _handle_per_request_error, and the shutdown method _initiate_shutdown to find where to add the ordering assertion.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@components/src/dynamo/trtllm/request_handlers/handler_base.py`:
- Around line 940-961: In _handle_per_request_error, don't silently swallow
exceptions raised by the yield; replace the bare except block with logging that
records the exception (include exc_info=True) so downstream failures (e.g.,
serialization errors) are visible; use logging.warning (or logging.debug for
non-fatal) with context about request_id and error_msg and then continue
best-effort behavior, keeping the fatal-path call to _initiate_shutdown and the
FATAL_CUDA_ERROR_PATTERNS check intact.
---
Nitpick comments:
In `@components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py`:
- Around line 711-748: Update the tests (e.g.,
test_fatal_cuda_error_triggers_shutdown and similar) to assert that the client's
error chunk is yielded before _initiate_shutdown is awaited: instrument
handler._initiate_shutdown (AsyncMock or side_effect) to append a
marker/timestamp when invoked, collect the emitted chunks from
_handle_per_request_error into a list first, then assert the emission happened
before the shutdown marker (or compare timestamps/sequence indices). Reference
the test helper _make_handler(), the method under test
_handle_per_request_error, and the shutdown method _initiate_shutdown to find
where to add the ordering assertion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 75e3d37e-e3b4-48a5-8f0a-7d677affb41c
📒 Files selected for processing (2)
components/src/dynamo/trtllm/request_handlers/handler_base.pycomponents/src/dynamo/trtllm/tests/test_trtllm_handler_base.py
a26eb71 to
225b277
Compare
- handler_base.py: log (at debug) instead of silently swallowing an Exception raised while emitting the RequestError chunk, so a serialization failure on the last client-visible signal before os._exit(1) remains observable. - test_trtllm_handler_base.py: instrument _initiate_shutdown's side_effect and record each yield to assert the error chunk is emitted before shutdown is awaited (guards against reordering the production code). Add docstrings to new test methods for the 80% coverage gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Yuting Wu (DLAlgo) <yutwu@nvidia.com>
f4e8041 to
b8fa311
Compare
…/fix-trtllm-fatal-cuda-errors
|
@nv-yna would you mind taking a look when you have a chance? This escalates fatal CUDA errors (e.g. |
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information. |
Overview:
TRT-LLM raises unrecoverable CUDA errors (
cudaErrorNvlinkUncorrectable, uncorrectable ECC, GPU-off-bus, device assert, illegal address, etc.) wrapped insideRequestError. The trtllm handler was downgrading these to a per-request WARN and yielding an error chunk, leaving the worker alive and accepting new traffic in a broken engine state. This PR escalates matching errors to a worker restart so Kubernetes can replace the pod.Details:
FATAL_CUDA_ERROR_PATTERNSregex at module level listing unrecoverable CUDA error strings (NVLink, ECC, no-device, illegal/misaligned address, device assert, launch timeout, hardware stack, cudaErrorUnknown).re.IGNORECASEcovers casing variants.cudaErrorMemoryAllocation(OOM) is intentionally NOT in the list — TRT-LLM aborts the offending request and frees KV cache, so the engine stays healthy. A test pins this behavior.except RequestErrorbranch body intoHandlerBase._handle_per_request_error(error, request_id)so the escalation path is directly unit-testable. On a fatal match it logs ERROR with traceback, best-effort yields the error chunk, then awaits_initiate_shutdown()(which callsos._exit(1)after engine cleanup). Benign errors keep the existing WARN-and-yield behavior.TestFatalCudaErrorPatterns— 9 fatal strings match, 5 benign (including OOM and empty string) do not.TestHandlePerRequestError— fatal triggers_initiate_shutdown, benign doesn't, OOM doesn't.Temporary workaround: once TRT-LLM exposes a first-class health-check API in LLM-API (NVIDIA/TensorRT-LLM#4513), the worker should query that instead of scraping error strings and this block can be reverted.
Where should the reviewer start?
Related Issues:
Testing
Validated locally in the trtllm-runtime dev container:
Real NVLink-fault reproduction is not feasible on available hardware.
Summary by CodeRabbit
Bug Fixes
Tests