Skip to content

fix(trtllm): escalate poisoned KV cache transfer to worker shutdown - #9481

Closed
yifjiang wants to merge 2 commits into
ai-dynamo:mainfrom
yifjiang:v11-poison-restart
Closed

yifjiang wants to merge 2 commits into
ai-dynamo:mainfrom
yifjiang:v11-poison-restart

Conversation

@yifjiang

Copy link
Copy Markdown
Contributor

Summary

When TRT-LLM raises a RequestError for an unrecoverable KV cache transfer failure (poisoned NIXL/UCX buffer pool), the dynamo TRT-LLM handler currently swallows it as a per-request warning and keeps serving with a corrupted buffer pool. This results in a wedge where every subsequent generation request gets Cannot assign cache transfer buffer kind=kv and clients see HTTP 500.

This PR extends the existing FATAL_CUDA_ERROR_PATTERNS design (#8342) to also escalate poisoned-KV-cache-transfer-buffer errors. On a match, the worker logs the fatal, emits the error chunk, and calls _initiate_shutdown() so k8s/docker can restart the container.

Validation

Validated against the 1P/1D production-like canary reproducer that wedges with current main:

Patterns

The regex matches both the C++ log strings (kept for backward compat) and the Python wrapper that surfaces to the RequestError:

  • C++ strings from baseTransBuffer.cpp: Cannot assign cache transfer buffer.*poisoned, previous transfer left the buffer pool poisoned, Poisoned .* cache transfer buffer, process must restart before these memory ranges
  • Python wrapper from py_executor.py: Error in kv cache transfer for (context|generation) requests (matches both pre-PR-13713-v10 plain wrappers and post-v10 poison-aware wrappers since they share the prefix)

Test plan

  • Reproducer triggers decode_restarts=1 on first poison hit (was 0)
  • 15-min soak: 30/30 probes succeed after self-recovery
  • Final sanity probes 3/3 OK
  • Side-by-side burst-sweep test against unpatched dynamo main to confirm no regression on non-poison KV transfer errors

🤖 Generated with Claude Code

yifjiang added 2 commits May 13, 2026 06:43
…down

Modeled on ai-dynamo/dynamo PR ai-dynamo#8342. Adds FATAL_KV_TRANSFER_ERROR_PATTERNS and extends the except RequestError branch to also restart the worker on TRT-LLM poisoned-cache-transfer-buffer errors.
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi yifjiang! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor backend::trtllm Relates to the trtllm backend labels May 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions Bot added the Stale label Jun 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information.

@github-actions github-actions Bot closed this Jun 19, 2026

This branch was previously deployed

1 inactive deployment
GITLAB — 5bc90e85 Deployed May 13, 2026 by copy-pr-bot[bot] via Trigger CI Pipeline #27399
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::trtllm Relates to the trtllm backend external-contribution Pull request is from an external contributor fix size/M Stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant