Repository navigation
fix(vllm): roll back scale_elastic_ep on a failed grow instead of wedging the engine - #12991
Conversation
…ging the engine A failed data-parallel grow left the vLLM engine wedged: the leader process survived but stopped serving, stuck at a dp it could not fulfill. When a new rank died mid-init (e.g. cudaErrorNvlinkUncorrectable while joining NCCL), the engine had already committed to the larger dp and its remaining ranks then expected a collective that could never complete, so inference hung/errored while the leader stayed up (restart=0). Make the grow atomic: capture the current dp up front and, on failure, roll back to it so the engine keeps serving -- "either it grows, or nothing changes" for the engine, not just the pod. If the rollback itself fails, the engine is unrecoverable in process (an uncorrectable NVLink error can poison the CUDA/NCCL context), so report recoverable=false and that the worker must be restarted rather than reporting a false recovery. The ray-list_nodes-patched scale is factored into a local _scale_engine(size) helper so the grow and the rollback go through the same path. Adds test_vllm_elastic_ep_handlers.py covering success, failed-grow-rolls-back, and failed-rollback-is-unrecoverable. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
WalkthroughThe elastic EP scaling handler now patches Ray node discovery during engine resizing, records the previous size, and attempts rollback after failed growth. Tests cover successful scaling, validation, rollback outcomes, response fields, call ordering, and state cleanup. ChangesElastic EP scaling
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/src/dynamo/vllm/handlers.py`:
- Around line 1496-1512: Update the elastic EP scaling flow around _scale_engine
to handle EngineDeadError before the broad grow and rollback catches, logging
and re-raising it without attempting rollback or returning an error response.
Restrict rollback to documented recoverable scaling failures, while logging and
re-raising unknown exceptions; add a test verifying an EngineDeadError shuts
down the engine and does not invoke rollback.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 420ef66f-c62b-437e-9eec-389889d1cc78
📒 Files selected for processing (2)
components/src/dynamo/vllm/handlers.pycomponents/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py
This comment has been minimized.
This comment has been minimized.
Address the Devin review on #12991. vLLM v0.26 records parallel_config.data_parallel_size only *after* a successful reconfigure and short-circuits scale_elastic_ep(size) to a no-op when size == the recorded size. After a failed grow the recorded size is still prev_dp, so the compensating scale_elastic_ep(prev_dp) call was silently skipped: the half-created ranks were never torn down, yet the handler reported recoverable=True -- the exact false recovery this PR set out to avoid. Advance the recorded size to the attempted target before rolling back so the rollback drives a genuine scale-down; if it still cannot complete it raises and we report recoverable=False. Model vLLM's guard + record-after-success semantics in the unit test's fake engine, and assert the rollback is a *real* reconfigure ([grow, rollback]) instead of a plain mock that hid the no-op. Convert the four new ElasticEP log calls to lazy %s formatting per .ai/python-guidelines.md. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-ep-scale-rollback
Address the CodeRabbit review on #12991. vLLM defines EngineDeadError as unrecoverable. The broad `except Exception` around the grow caught it and treated it as a rollback candidate, so a dead engine core would be masked as a recovered engine. Catch EngineDeadError before the rollback path and report recoverable=false so the worker restarts instead of reporting a false recovery -- consistent with how the rest of this handler treats engine death. Add a unit test asserting a dead engine skips the rollback entirely (only the failed grow reaches the engine). Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the add/add conflict in components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py between this PR and #12937 (derive elastic EP scale floor from TP). Both PRs touch scale_elastic_ep and each added a test module of the same name: - #12937 rewrote the input validation / TP-derived EP-size floor and added validation tests; - this PR rewrote the failed-grow rollback + engine-death handling and added rollback tests. handlers.py auto-merged cleanly (validation on top, rollback/EngineDeadError below). The test module now carries BOTH test sets: #12937's validation tests verbatim (with its importorskip/stub_ray/pytestmark structure) plus this PR's rollback tests, whose _FakeVllmEngine now also exposes tensor_parallel_size so it clears the new floor check and reaches the rollback path. The redundant missing-field / below-floor rollback tests were dropped in favor of #12937's validation coverage. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
The rollback path this PR adds captures prev_dp from parallel_config.data_parallel_size up front on every scale. #12937's _make_handler test helper (merged from main) only set tensor_parallel_size and prefill_context_parallel_size, so after the merge its success-path tests (test_integer_valued_sizes_are_accepted, test_dp_one/two_allowed_*) hit an AttributeError that the handler's outer except turns into status "error" -- failing the "scale accepted" assertions in the vllm-runtime unit tests on both amd64 and arm64. Real vLLM engines always set data_parallel_size; add it to the helper (default 1) so those tests reach the reconfigure path again. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
biswapanda
left a comment
There was a problem hiding this comment.
left few comments in-line
…back Address @biswapanda's review on #12991. vLLM performs no rollback or cleanup on a failed elastic-EP scale: its _scale_up_elastic_ep has no try/except and its own /scale_elastic_ep endpoint just returns 500, leaving the engine in a partial state (existing ranks reconfigured toward a peer that never joined, wedged on a collective that never completes). The previous in-handler rollback tried to recover on top of an API that provides no rollback primitive, which: - raced vLLM's still-running grow tasks -- asyncio.gather does not cancel the siblings of the failed task (Thread B), - could not actually reach the previous size: the forged-config guard bypass is rejected by the inner DPLBAsyncMPClient, which keys off len(core_engines), or is redundant (Thread C), and - returned a `recoverable` field that nothing consumes, so a dead/wedged worker stayed registered and kept taking traffic (Thread A). Since vLLM gives us no safe way to recover in process, fail fast: on any failed grow shut the worker down (_shutdown_on_engine_dead for a dead engine, _shutdown_worker otherwise) so it restarts clean at the previous size and re-registers -- consistent with how every other engine path in this handler already reacts to engine death. Drops the rollback, the guard-bypass overwrite, and the inert recoverable dicts. Tests now assert the worker restart is actually triggered (via a BaseException stand-in for the NoReturn shutdown), not just that a boolean was returned. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-ep-scale-rollback
…equest Complements the fail-fast restart tests: a request rejected up front by validation (dp=1 at TP=1 collapses the EP world) returns an error and must NOT restart the worker or touch the engine, scoping fail-fast to real scale failures rather than bad input. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tighten the fail-fast rationale comment in scale_elastic_ep and the test module/helper docstrings for readability; no behavior change. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gate (Phase 6) Reads get_ep_capacity (Phase 5) over the Phase-3 leader Service and converges the engine's data-parallel size on the owned capacity that has joined. Gated OFF by default (DYN_OPERATOR_ENABLE_ELASTIC_EP_AUTOSCALE=1): until scale_elastic_ep gets a clean rollback (#12991), a failed grow can leave the engine holding a reserved-but-dead rank and serving nothing, so the operator must not fire scale calls on its own initiative without opt-in. Core logic (pure, exhaustively tested -- internal/dynamo/elastic_ep_scale.go): - DecideEPScale keeps three numbers distinct: current (engine size), observed (owned Ray nodes joined), desired (1 + follower count). Growth is driven by observation and bounded by desire; shrink is driven only by desire falling; observation dropping below current is a FAULT (a live rank's pod died), never a scale-down -- matching to it would shrink around dead ranks. Floor of one rank. - OwnedRayNodeCount is the generation filter: it intersects what Ray reports with this deployment's current-generation pod IPs, so a superseded-generation pod still alive in Ray is counted as noise, not capacity. Reconciler (internal/controller): - HTTP client posts to /engine/control/{ep_capacity,scale_elastic_ep} on the system port over the network (the Phase-3 Service), not kubectl exec. scale_elastic_ep is accepted only when status==ok AND the echoed new_data_parallel_size matches the target; a busy reply requeues. - A settle window (arrivals-stop dwell) so a spike of N followers costs one rebuild; a fire-once-per-change guard; grow waits for the window, shrink need not. - Wired into the component program as a gated, non-fatal phase (a scale hiccup never wedges the DGD reconcile). The follower-suffix const is duplicated locally to keep this a clean sibling of the Phase-3 branch (the shared const lands with Phase 4). Durable status fields (Phase 7b) and a periodic requeue are noted as follow-ups. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Address @biswapanda's review on #12991. control/scale_elastic_ep is registered on every vLLM worker, but elastic EP only works with the Ray DP backend and the feature enabled. On a default worker (mp backend, elastic EP off) a valid scale request reaches vLLM's scale_elastic_ep, which raises NotImplementedError / a Ray-backend assertion -- which the fail-fast grow handler would turn into a restart of a perfectly healthy worker that never mutated any scale state. Gate on parallel_config.enable_elastic_ep and data_parallel_backend == 'ray' before _scale_engine and return a nonfatal configuration error for unsupported workers. Adds test_unsupported_config_is_rejected_without_restart; the existing fixtures now advertise a supported config. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-ep-scale-rollback
|
📌 Follow-up hardening PR: #14053 addresses the architecture-review findings on |
Follow-up to #12991 addressing the architecture review of scale_elastic_ep: - #1 Outer timeout backstop: wrap the scale in asyncio.wait_for (DYN_SCALE_EP_TIMEOUT_S, default 600s). Without it, a scale that hangs past vLLM's own limits leaves _scale_ep_in_progress stuck True and wedges the endpoint forever. A timed-out scale is a wedged engine -> fail fast. - #2 Warm the heavy 'ray' import off the event loop (asyncio.to_thread) so the first scale on a worker does not block the loop. - #3 Document that the ray.util.state.list_nodes patch mutates a process-global (safe: serialised by _scale_ep_in_progress, restored in finally). - #4 Document that an externally-cancelled scale (shutdown / client disconnect) intentionally does not force a restart. Adds test_hung_grow_times_out_and_restarts. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Overview:
A failed
scale_elastic_epgrow wedged the vLLM engine: the leader process survived but stopped serving, stuck at a data-parallel size it could not fulfill. This makes the grow atomic — on failure it rolls back to the previous dp and keeps serving; if the rollback itself fails, it reports the engine as unrecoverable (must restart) instead of a false recovery. Fixes DYN-3838.Details:
Observed during elastic-EP testing (DYN-3686): when a new rank died mid-init while joining NCCL (
cudaErrorNvlinkUncorrectableon a degraded node), the engine had already committed to the larger dp, so ranks 0/1 then expected a 3-way collective that never completes — inference hung/errored while the leader stayed alive (restart=0).BaseWorkerHandler.scale_elastic_epincomponents/src/dynamo/vllm/handlers.pynow:list_nodes-patched scale into a local_scale_engine(size)helper so the grow and any rollback go through the same path;prev_dp = engine_client.vllm_config.parallel_config.data_parallel_sizebefore the grow;_scale_engine(prev_dp)(vLLM'sscale_elastic_epis symmetric — a smaller size shrinks). Success → returns{"status": "error", "recoverable": true, "data_parallel_size": prev_dp}and the engine keeps serving;{"status": "error", "recoverable": false, ...}with "the engine ... must be restarted", because an uncorrectable NVLink error can poison the CUDA/NCCL context so the DP group cannot be rebuilt in process.Two-tier by design: recoverable failures (unschedulable, or a clean init error before any collective) roll back in place; unrecoverable failures surface a clear restart signal — the design already tolerates a leader restart (the follower rejoins). Wiring
recoverable: falseto an automatic pod restart (liveness-probe failure / explicit exit) is a deliberate, separately-reviewable follow-up.Validation:
components/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py(pure unit test; mocked engine client,raystubbed insys.modules;test_vllm_prefix so it skips cleanly where vLLM is absent) covers: successful scale (grow called once), failed grow → rollback to prev dp (scale_elastic_epawaited with the grow size then the prior size;recoverable=true), and failed rollback →recoverable=falsewith "must be restarted", plus the missing-field and below-floor rejections.pre-commit run --fileswith the repo-pinned hooks (isort / black / flake8 / ruff / codespell / pytest-marker report) passes on both files.Where should the reviewer start?
components/src/dynamo/vllm/handlers.py— thescale_elastic_epfailure path: the new_scale_enginehelper and the grow/rollback try-nest (theexcept Exception as grow_errblock). Then the three rollback cases incomponents/src/dynamo/vllm/tests/test_vllm_elastic_ep_handlers.py.Summary by CodeRabbit
Bug Fixes
Tests