Skip to content

fix(trtllm): make sidecar bootstrap cancellable - #15946

Closed
tanmayv25 wants to merge 2 commits into
mainfrom
fix/trtllm-cancellable-bootstrap
Closed

tanmayv25 wants to merge 2 commits into
mainfrom
fix/trtllm-cancellable-bootstrap

Conversation

@tanmayv25

@tanmayv25 tanmayv25 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by #15686, which now targets main and includes this fix together with the CPU sidecar tests.

Summary

When SIGTERM arrives while the TensorRT-LLM sidecar is waiting for an engine connection, startup continues until the worker forces an exit. Move connection and model-limit discovery into the shared runner's cancellable bootstrap so shutdown can finish cleanly.

Worker startup reuses the discovered connection; generation remains unavailable until the worker starts.

Validation

  • The CPU process test trtllm_failed_and_interrupted_startup_leave_no_registration failed on main 1c021a47b9 with forced exit code 911 and passed with this fix.
  • 52 TensorRT-LLM unit tests passed, including the bootstrap connection-reuse regression. Two existing GPU tests were ignored.
  • The combined sidecar/testkit run passed all 372 CPU tests with normal parallelism and GPUs hidden.
  • Rust formatting, focused Clippy with warnings denied, and repository hooks passed.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested review from a team as code owners October 8, 2026 23:36
@github-actions github-actions Bot added the fix label Oct 8, 2026
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 5128bfc1-27de-4e7e-95cf-df27e5a6ca00
📥 Commits

Reviewing files that changed from the base of the PR and between 3bdbb2e and 999d982.

📒 Files selected for processing (4)
  • lib/bindings/python/rust/backend.rs
  • lib/sidecar/trtllm/src/client.rs
  • lib/sidecar/trtllm/src/engine.rs
  • lib/sidecar/trtllm/src/tests/engine.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The TensorRT-LLM sidecar launcher now maps runtime errors directly to Python RuntimeError. Async engine entry points resolve the sidecar connection during bootstrap. Engine start reuses that connection, and generation and abort require a started engine.

Changes

Python runtime error mapping

Layer / File(s) Summary
Map sidecar runtime errors
lib/bindings/python/rust/backend.rs
Runtime errors from the sidecar runner now become PyRuntimeError. Startup argument errors continue to use sidecar_startup_to_pyerr.

TensorRT-LLM engine bootstrap and lifecycle

Layer / File(s) Summary
Discover the connection during bootstrap
lib/sidecar/trtllm/src/engine.rs, lib/sidecar/trtllm/src/client.rs
Async argument entry points now run bootstrap, which resolves the client and model limits before returning the engine and worker configuration. The client comment reflects async bootstrap and deferred LLMEngine::start.
Gate operations on engine start
lib/sidecar/trtllm/src/engine.rs, lib/sidecar/trtllm/src/tests/engine.rs
start reuses the resolved connection, records the started state, and sets the context length. generate and abort require a started engine. The regression test checks that bootstrap performs one metadata lookup, that start reuses it, and that generation works after start.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 999d9

Sidecar discovery remains cancellable, and discovery failures reach the expected Python error boundary. No identified issue prevents merging after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 38.46% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 13 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly describes the main change: making TensorRT-LLM sidecar bootstrap cancellable.
Description check Passed The description explains the startup issue, the implementation, and validation results. It references issue #15686, but does not include the template’s section headings or identify where the reviewer …
Full details: Description check

Explanation

The description explains the startup issue, the implementation, and validation results. It references issue #15686, but does not include the template’s section headings or identify where the reviewer should start.

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

⏹️ Dynamo PR CI cancelled — run 37860347222 (attempt 1) on 999d982ff6

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Other Jobs
planner ⏹️ 1 ✅ 4

Failure details

No jobs failed. The one cancelled job hit its job timeout; it was not a fail-fast, manual, or concurrency cancel.

  • ⏹️ planner / Compliance cpu, amd64: GitHub cancelled it with The job has exceeded the maximum execution time of 15m0s. The Compliance scan step's docker buildx build -f container/compliance/Dockerfile.extract (remote buildkit builder) got no further than load build definition for about 14.7 minutes. This looks like a stalled buildkit/infrastructure hang, so rerunning the failed jobs is likely enough. Logs: gh run view --job 113597450206 -R ai-dynamo/dynamo --log
For agents
{"pr": 15946, "run_id": 37860347222, "run_attempt": 1, "head_sha": "999d982ff68aee0d5523d77f0cf71d62c0fc55aa",
 "failures": [],
 "cancelled": [{"job": "planner / Compliance cpu, amd64", "job_id": 113597450206, "failed_step": "Compliance scan", "signature": "job exceeded 15m timeout; docker buildx compliance extract stalled after loading build definition", "tests": [], "log_cmd": "gh run view --job 113597450206 -R ai-dynamo/dynamo --log"}]}

Posted automatically by Devin for run 37860347222. Updated on every full-CI run of this PR.

@tanmayv25 tanmayv25 closed this Oct 16, 2026
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25 tanmayv25 reopened this Oct 17, 2026
@tanmayv25 tanmayv25 closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant