Skip to content

fix(vllm): roll back scale_elastic_ep on a failed grow instead of wedging the engine - #12991

Merged
tzulingk merged 12 commits into
mainfrom
tzulingk/vllm-elastic-ep-scale-rollback
Sep 3, 2026
Merged

tzulingk merged 12 commits into
mainfrom
tzulingk/vllm-elastic-ep-scale-rollback

Conversation

@tzulingk

@tzulingk tzulingk commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

A failed scale_elastic_ep grow wedged the vLLM engine: the leader process survived but stopped serving, stuck at a data-parallel size it could not fulfill. This makes the grow atomic — on failure it rolls back to the previous dp and keeps serving; if the rollback itself fails, it reports the engine as unrecoverable (must restart) instead of a false recovery. Fixes DYN-3838.

Details:

Observed during elastic-EP testing (DYN-3686): when a new rank died mid-init while joining NCCL (cudaErrorNvlinkUncorrectable on a degraded node), the engine had already committed to the larger dp, so ranks 0/1 then expected a 3-way collective that never completes — inference hung/errored while the leader stayed alive (restart=0).

BaseWorkerHandler.scale_elastic_ep in components/src/dynamo/vllm/handlers.py now:

  • factors the ray-list_nodes-patched scale into a local _scale_engine(size) helper so the grow and any rollback go through the same path;
  • captures prev_dp = engine_client.vllm_config.parallel_config.data_parallel_size before the grow;
  • on grow failure, rolls back with a compensating _scale_engine(prev_dp) (vLLM's scale_elastic_ep is symmetric — a smaller size shrinks). Success → returns {"status": "error", "recoverable": true, "data_parallel_size": prev_dp} and the engine keeps serving;
  • if the rollback also fails → returns {"status": "error", "recoverable": false, ...} with "the engine ... must be restarted", because an uncorrectable NVLink error can poison the CUDA/NCCL context so the DP group cannot be rebuilt in process.

Two-tier by design: recoverable failures (unschedulable, or a clean init error before any collective) roll back in place; unrecoverable failures surface a clear restart signal — the design already tolerates a leader restart (the follower rejoins). Wiring recoverable: false to an automatic pod restart (liveness-probe failure / explicit exit) is a deliberate, separately-reviewable follow-up.

Validation:

  • New components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py (pure unit test; mocked engine client, ray stubbed in sys.modules; test_vllm_ prefix so it skips cleanly where vLLM is absent) covers: successful scale (grow called once), failed grow → rollback to prev dp (scale_elastic_ep awaited with the grow size then the prior size; recoverable=true), and failed rollback → recoverable=false with "must be restarted", plus the missing-field and below-floor rejections.
  • pre-commit run --files with the repo-pinned hooks (isort / black / flake8 / ruff / codespell / pytest-marker report) passes on both files.

Where should the reviewer start?

components/src/dynamo/vllm/handlers.py — the scale_elastic_ep failure path: the new _scale_engine helper and the grow/rollback try-nest (the except Exception as grow_err block). Then the three rollback cases in components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py.


Open in Devin Review

Summary by CodeRabbit

  • Bug Fixes

    • Improved elastic scaling reliability by using current cluster node information during scale operations.
    • Added automatic rollback when a scale-up fails.
    • Added clearer reporting for recoverable and unrecoverable scaling failures.
    • Added validation for missing or below-minimum scaling sizes.
  • Tests

    • Added coverage for successful scaling, validation errors, and rollback scenarios.

…ging the engine

A failed data-parallel grow left the vLLM engine wedged: the leader process
survived but stopped serving, stuck at a dp it could not fulfill. When a new
rank died mid-init (e.g. cudaErrorNvlinkUncorrectable while joining NCCL), the
engine had already committed to the larger dp and its remaining ranks then
expected a collective that could never complete, so inference hung/errored
while the leader stayed up (restart=0).

Make the grow atomic: capture the current dp up front and, on failure, roll
back to it so the engine keeps serving -- "either it grows, or nothing changes"
for the engine, not just the pod. If the rollback itself fails, the engine is
unrecoverable in process (an uncorrectable NVLink error can poison the
CUDA/NCCL context), so report recoverable=false and that the worker must be
restarted rather than reporting a false recovery.

The ray-list_nodes-patched scale is factored into a local _scale_engine(size)
helper so the grow and the rollback go through the same path. Adds
test_vllm_elastic_ep_handlers.py covering success, failed-grow-rolls-back, and
failed-rollback-is-unrecoverable.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@tzulingk
tzulingk requested review from a team as code owners August 11, 2026 01:59
@github-actions github-actions Bot added fix backend::vllm Relates to the vllm backend and removed fix labels Aug 11, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Open in Devin Review

Comment thread components/src/dynamo/vllm/handlers.py Resolved
Comment thread components/src/dynamo/vllm/handlers.py Resolved
@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The elastic EP scaling handler now patches Ray node discovery during engine resizing, records the previous size, and attempts rollback after failed growth. Tests cover successful scaling, validation, rollback outcomes, response fields, call ordering, and state cleanup.

Changes

Elastic EP scaling

Layer / File(s) Summary
Scaling and rollback flow
components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py
scale_elastic_ep uses _scale_engine with GCS-backed ray.nodes() discovery, tracks the previous data-parallel size, rolls back failed growth, and reports recoverable or unrecoverable errors. Tests cover success, validation, rollback behavior, response fields, call ordering, and cleanup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers the overview, implementation details, reviewer starting point, validation, and linked issue, but it omits the required Related Issues section. Add the required Related Issues section and select the linked-issue or no-issue path using the repository template.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the rollback behavior added for failed vLLM elastic data-parallel growth.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/vllm/handlers.py`:
- Around line 1496-1512: Update the elastic EP scaling flow around _scale_engine
to handle EngineDeadError before the broad grow and rollback catches, logging
and re-raising it without attempting rollback or returning an error response.
Restrict rollback to documented recoverable scaling failures, while logging and
re-raising unknown exceptions; add a test verifying an EngineDeadError shuts
down the engine and does not invoke rollback.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 420ef66f-c62b-437e-9eec-389889d1cc78

📥 Commits

Reviewing files that changed from the base of the PR and between e9cb2fe and b4a2508.

📒 Files selected for processing (2)
  • components/src/dynamo/vllm/handlers.py
  • components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py

Comment thread components/src/dynamo/vllm/handlers.py Resolved
Comment thread components/src/dynamo/vllm/handlers.py Resolved
@datadog-official

This comment has been minimized.

tzulingk and others added 3 commits August 11, 2026 16:04
Address the Devin review on #12991.

vLLM v0.26 records parallel_config.data_parallel_size only *after* a
successful reconfigure and short-circuits scale_elastic_ep(size) to a
no-op when size == the recorded size. After a failed grow the recorded
size is still prev_dp, so the compensating scale_elastic_ep(prev_dp)
call was silently skipped: the half-created ranks were never torn down,
yet the handler reported recoverable=True -- the exact false recovery
this PR set out to avoid. Advance the recorded size to the attempted
target before rolling back so the rollback drives a genuine scale-down;
if it still cannot complete it raises and we report recoverable=False.

Model vLLM's guard + record-after-success semantics in the unit test's
fake engine, and assert the rollback is a *real* reconfigure
([grow, rollback]) instead of a plain mock that hid the no-op.

Convert the four new ElasticEP log calls to lazy %s formatting per
.ai/python-guidelines.md.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address the CodeRabbit review on #12991.

vLLM defines EngineDeadError as unrecoverable. The broad `except
Exception` around the grow caught it and treated it as a rollback
candidate, so a dead engine core would be masked as a recovered engine.
Catch EngineDeadError before the rollback path and report
recoverable=false so the worker restarts instead of reporting a false
recovery -- consistent with how the rest of this handler treats engine
death. Add a unit test asserting a dead engine skips the rollback
entirely (only the failed grow reaches the engine).

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tzulingk
tzulingk requested a review from biswapanda August 13, 2026 03:39
tzulingk and others added 2 commits August 12, 2026 22:45
Resolve the add/add conflict in
components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py between
this PR and #12937 (derive elastic EP scale floor from TP). Both PRs touch
scale_elastic_ep and each added a test module of the same name:

  - #12937 rewrote the input validation / TP-derived EP-size floor and added
    validation tests;
  - this PR rewrote the failed-grow rollback + engine-death handling and added
    rollback tests.

handlers.py auto-merged cleanly (validation on top, rollback/EngineDeadError
below). The test module now carries BOTH test sets: #12937's validation tests
verbatim (with its importorskip/stub_ray/pytestmark structure) plus this PR's
rollback tests, whose _FakeVllmEngine now also exposes tensor_parallel_size so
it clears the new floor check and reaches the rollback path. The redundant
missing-field / below-floor rollback tests were dropped in favor of #12937's
validation coverage.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
The rollback path this PR adds captures prev_dp from
parallel_config.data_parallel_size up front on every scale. #12937's
_make_handler test helper (merged from main) only set tensor_parallel_size
and prefill_context_parallel_size, so after the merge its success-path tests
(test_integer_valued_sizes_are_accepted, test_dp_one/two_allowed_*) hit an
AttributeError that the handler's outer except turns into status "error" --
failing the "scale accepted" assertions in the vllm-runtime unit tests on
both amd64 and arm64. Real vLLM engines always set data_parallel_size; add
it to the helper (default 1) so those tests reach the reconfigure path again.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread components/src/dynamo/vllm/handlers.py Resolved
Comment thread components/src/dynamo/vllm/handlers.py Resolved
Comment thread components/src/dynamo/vllm/handlers.py Resolved

@biswapanda biswapanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left few comments in-line

tzulingk and others added 3 commits August 19, 2026 15:03
…back

Address @biswapanda's review on #12991.

vLLM performs no rollback or cleanup on a failed elastic-EP scale: its
_scale_up_elastic_ep has no try/except and its own /scale_elastic_ep
endpoint just returns 500, leaving the engine in a partial state (existing
ranks reconfigured toward a peer that never joined, wedged on a collective
that never completes). The previous in-handler rollback tried to recover on
top of an API that provides no rollback primitive, which:
  - raced vLLM's still-running grow tasks -- asyncio.gather does not cancel
    the siblings of the failed task (Thread B),
  - could not actually reach the previous size: the forged-config guard
    bypass is rejected by the inner DPLBAsyncMPClient, which keys off
    len(core_engines), or is redundant (Thread C), and
  - returned a `recoverable` field that nothing consumes, so a dead/wedged
    worker stayed registered and kept taking traffic (Thread A).

Since vLLM gives us no safe way to recover in process, fail fast: on any
failed grow shut the worker down (_shutdown_on_engine_dead for a dead
engine, _shutdown_worker otherwise) so it restarts clean at the previous
size and re-registers -- consistent with how every other engine path in
this handler already reacts to engine death. Drops the rollback, the
guard-bypass overwrite, and the inert recoverable dicts.

Tests now assert the worker restart is actually triggered (via a
BaseException stand-in for the NoReturn shutdown), not just that a boolean
was returned.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…equest

Complements the fail-fast restart tests: a request rejected up front by
validation (dp=1 at TP=1 collapses the EP world) returns an error and must
NOT restart the worker or touch the engine, scoping fail-fast to real scale
failures rather than bad input.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tzulingk
tzulingk requested a review from biswapanda August 19, 2026 20:10
Tighten the fail-fast rationale comment in scale_elastic_ep and the
test module/helper docstrings for readability; no behavior change.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tzulingk added a commit that referenced this pull request Aug 19, 2026
…gate (Phase 6)

Reads get_ep_capacity (Phase 5) over the Phase-3 leader Service and converges the engine's
data-parallel size on the owned capacity that has joined. Gated OFF by default
(DYN_OPERATOR_ENABLE_ELASTIC_EP_AUTOSCALE=1): until scale_elastic_ep gets a clean rollback
(#12991), a failed grow can leave the engine holding a reserved-but-dead rank and serving
nothing, so the operator must not fire scale calls on its own initiative without opt-in.

Core logic (pure, exhaustively tested -- internal/dynamo/elastic_ep_scale.go):
- DecideEPScale keeps three numbers distinct: current (engine size), observed (owned Ray nodes
  joined), desired (1 + follower count). Growth is driven by observation and bounded by desire;
  shrink is driven only by desire falling; observation dropping below current is a FAULT (a live
  rank's pod died), never a scale-down -- matching to it would shrink around dead ranks. Floor
  of one rank.
- OwnedRayNodeCount is the generation filter: it intersects what Ray reports with this
  deployment's current-generation pod IPs, so a superseded-generation pod still alive in Ray is
  counted as noise, not capacity.

Reconciler (internal/controller):
- HTTP client posts to /engine/control/{ep_capacity,scale_elastic_ep} on the system port over
  the network (the Phase-3 Service), not kubectl exec. scale_elastic_ep is accepted only when
  status==ok AND the echoed new_data_parallel_size matches the target; a busy reply requeues.
- A settle window (arrivals-stop dwell) so a spike of N followers costs one rebuild; a
  fire-once-per-change guard; grow waits for the window, shrink need not.
- Wired into the component program as a gated, non-fatal phase (a scale hiccup never wedges the
  DGD reconcile).

The follower-suffix const is duplicated locally to keep this a clean sibling of the Phase-3
branch (the shared const lands with Phase 4). Durable status fields (Phase 7b) and a periodic
requeue are noted as follow-ups.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Comment thread components/src/dynamo/vllm/handlers.py Resolved
tzulingk and others added 2 commits August 31, 2026 09:31
Address @biswapanda's review on #12991.

control/scale_elastic_ep is registered on every vLLM worker, but elastic EP
only works with the Ray DP backend and the feature enabled. On a default
worker (mp backend, elastic EP off) a valid scale request reaches vLLM's
scale_elastic_ep, which raises NotImplementedError / a Ray-backend assertion
-- which the fail-fast grow handler would turn into a restart of a perfectly
healthy worker that never mutated any scale state.

Gate on parallel_config.enable_elastic_ep and data_parallel_backend == 'ray'
before _scale_engine and return a nonfatal configuration error for
unsupported workers. Adds test_unsupported_config_is_rejected_without_restart;
the existing fixtures now advertise a supported config.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tzulingk

Copy link
Copy Markdown
Contributor Author

📌 Follow-up hardening PR: #14053 addresses the architecture-review findings on scale_elastic_ep — an outer timeout backstop so a hung scale can't wedge the endpoint forever, moving the heavy ray import off the event loop, plus two doc-only clarifications. It's built on top of this PR; please review/merge it after this one.

@tzulingk
tzulingk merged commit 09268a2 into main Sep 3, 2026
184 of 186 checks passed
@tzulingk
tzulingk deleted the tzulingk/vllm-elastic-ep-scale-rollback branch September 3, 2026 21:28
tzulingk added a commit that referenced this pull request Sep 3, 2026
Follow-up to #12991 addressing the architecture review of scale_elastic_ep:

- #1 Outer timeout backstop: wrap the scale in asyncio.wait_for
  (DYN_SCALE_EP_TIMEOUT_S, default 600s). Without it, a scale that hangs
  past vLLM's own limits leaves _scale_ep_in_progress stuck True and wedges
  the endpoint forever. A timed-out scale is a wedged engine -> fail fast.
- #2 Warm the heavy 'ray' import off the event loop (asyncio.to_thread) so
  the first scale on a worker does not block the loop.
- #3 Document that the ray.util.state.list_nodes patch mutates a
  process-global (safe: serialised by _scale_ep_in_progress, restored in
  finally).
- #4 Document that an externally-cancelled scale (shutdown / client
  disconnect) intentionally does not force a restart.

Adds test_hung_grow_times_out_and_restarts.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants