Skip to content

fix(vllm): stop engine monitors during handler cleanup - #16412

Open
rudrakshkarpe wants to merge 2 commits into
ai-dynamo:mainfrom
rudrakshkarpe:fix/vllm-monitor-shutdown
Open

rudrakshkarpe wants to merge 2 commits into
ai-dynamo:mainfrom
rudrakshkarpe:fix/vllm-monitor-shutdown

Conversation

@rudrakshkarpe

@rudrakshkarpe rudrakshkarpe commented Oct 14, 2026 •

Copy link
Copy Markdown

Summary

Stop vLLM health and stats monitoring from generation and embedding handler cleanup through a public, repeatable VllmEngineMonitor.stop() method. The worker factory retains ownership of AsyncLLM. The cleanup guide now describes that split.

Part of #16284. The transcription handler is in the still-open #16208; its author plans to rebase on this hook after this PR lands.

Validation

Added focused regression tests for running, unstarted, completed, and optional monitor tasks, plus repeated handler cleanup. The new cases fail against main and pass with this fix in local CPU checks using vLLM/runtime import stubs. Python compilation, pre-commit, and docs lint passed. Backend CI remains the integration check.

In the vLLM test image:

python3 -m pytest components/src/dynamo/vllm/tests/test_vllm_engine_monitor_stats.py components/src/dynamo/vllm/tests/test_vllm_handler_cleanup.py

Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
@rudrakshkarpe
rudrakshkarpe requested review from a team as code owners October 14, 2026 15:56
@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi rudrakshkarpe! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor backend::vllm Relates to the vllm backend labels Oct 14, 2026
@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 652b6930-d92d-4024-b563-1d06236e6309
📥 Commits

Reviewing files that changed from the base of the PR and between 53be534 and 57857ef.

📒 Files selected for processing: 4
  • components/src/dynamo/vllm/engine_monitor.py
  • components/src/dynamo/vllm/handlers.py
  • components/src/dynamo/vllm/tests/test_vllm_engine_monitor_stats.py
  • components/src/dynamo/vllm/tests/test_vllm_handler_cleanup.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

Walkthrough

The change adds explicit shutdown for vLLM engine monitor tasks. Worker handler cleanup calls the monitor’s stop() method. Tests cover monitor task states, repeated stop and cleanup calls, and preservation of engine and runtime ownership.

Changes

vLLM monitor shutdown

Layer / File(s) Summary
Monitor task shutdown
components/src/dynamo/vllm/engine_monitor.py, components/src/dynamo/vllm/tests/test_vllm_engine_monitor_stats.py
VllmEngineMonitor initializes task attributes before creating background tasks. Its new stop() method cancels tasks that exist and are unfinished, and __del__ delegates to stop(). Tests cover repeated calls, tasks that have not started or have completed, and tasks that are blocked.
Handler cleanup integration
components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/tests/test_vllm_handler_cleanup.py
BaseWorkerHandler.cleanup and EmbeddingWorkerHandler.cleanup call engine_monitor.stop(). Tests cover decode, prefill, and embedding handlers, including repeated cleanup and checks that cleanup does not shut down the engine client or runtime.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 57857

No actionable issue remains for the handlers included in this change. Backend vLLM CI remains a normal validation step.

Pre-merge checks: Passed 2 · Failed 3

Failed Failed checks: 3 warnings

Check name Status Explanation Resolution
Linked Issues check Warning The PR implements public VllmEngineMonitor.stop() with safe checks for missing and completed tasks. It calls stop() from generation and embedding handler cleanup, and tests cover monitor shutdown … Update transcription_handler.py cleanup to call the public VllmEngineMonitor.stop() without cancelling private tasks. Add the required cleanup documentation update. Add or extend focused tests for the transcription handler and the docum…
Docstring Coverage Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Warning The description explains the implementation, ownership split, validation, and related issue. It does not use the required Overview, Details, or Where should the reviewer start sections, and it uses “P… Add the required Overview, Details, and Where should the reviewer start sections. Replace or supplement “Part of #16284” with a supported reference such as “Relates to #16284” or “Closes #16284,” as appropriate.
Passed Passed checks: 2 passed
Check name Status Explanation
Out of Scope Changes check Passed The changed monitor lifecycle code, handler cleanup calls, and regression tests directly support issue #16284. The tests verify repeated cleanup, task shutdown, and preservation of factory-owned `Asyn…
Title check Passed The title clearly summarizes the main change: stopping vLLM engine monitors during handler cleanup.
Full details: Linked Issues check

Explanation

The PR implements public VllmEngineMonitor.stop() with safe checks for missing and completed tasks. It calls stop() from generation and embedding handler cleanup, and tests cover monitor shutdown and preserved AsyncLLM ownership. However, issue #16284 requires every vLLM handler that owns a monitor to call the public method, including transcription_handler.py. The PR explicitly excludes the transcription handler. The required cleanup documentation update is also not shown in the changed files.

Resolution

Update transcription_handler.py cleanup to call the public VllmEngineMonitor.stop() without cancelling private tasks. Add the required cleanup documentation update. Add or extend focused tests for the transcription handler and the documented lifecycle behavior.

Full details: Description check

Explanation

The description explains the implementation, ownership split, validation, and related issue. It does not use the required Overview, Details, or Where should the reviewer start sections, and it uses “Part of #16284” instead of the template’s specified issue-reference format.

  • Fix all pre-merge checks with AI
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: rudrakshkarpe <rudraksh.karpe@gmail.com>
@rudrakshkarpe
rudrakshkarpe requested a review from a team as a code owner October 14, 2026 16:24
@rudrakshkarpe
rudrakshkarpe temporarily deployed to external_collaborator October 14, 2026 16:25 — with GitHub Actions Inactive
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 14, 2026
@rudrakshkarpe

Copy link
Copy Markdown
Author

Updated the cleanup guide in 5a1a7f7 and clarified the PR scope. The transcription handler is still in #16208; its author plans to rebase on this stop hook after this PR lands.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

External PR triage

Blocker (failing-check-author): Docs Website Composition Check (and the aggregate pre-merge-status-check) fails on this PR (job) but passes on recent main commits, so this is specific to the PR or to its 5-day-old base. Trusted CI has also never run: no /ok to test has been posted.

Next action: @rudrakshkarpe: merge the latest main, check the job log, and fix the composition error in graceful-shutdown-architecture.md if it persists. After that, @ai-dynamo/dynamo-backend-vllm-codeowners @ai-dynamo/dynamo-observability-codeowners @ai-dynamo/dynamo-docs-codeowners should comment /ok to test <new head sha> and review.

Supply-chain audit of 5a1a7f7b: Safe to approve. The diff changes Python in components/src/dynamo/vllm/ (engine_monitor.py, handlers.py) plus unit tests and one doc page. No workflow/CI files, no dependency manifest changes (Cargo.toml/Cargo.lock/pyproject.toml/go.mod/requirements), no new download URLs, scripts, secret/credential access, or binary blobs. (Devin has not posted /ok to test or approved anything; a maintainer still has to vet and run CI.)

Automated triage, 2026-10-19. This comment is edited in place when the PR changes.

Written by Devin

This branch was previously deployed

1 inactive deployment
external_collaborator — 5a1a7f7b Deployed Oct 14, 2026 by rudrakshkarpe via ok-to-test #32382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant