Repository navigation
feat(operator): start a Ray head on a single-pod elastic EP component - #12943
Conversation
Elastic-EP wiring was reachable only through the isMultinode branch of UpdateContainer, so a component deployed as one pod never ran 'ray start --head'. Two things blocked it: the dispatch sat inside 'numberOfNodes > 1', and the leader arm of injectElasticEPRayLaunchFlags matched RoleLeader only, while a lone pod is expanded as RoleMain. Route single-pod components to the same injection and let the leader arm match RoleMain. Probes are left in place: unlike a multinode worker, this pod serves traffic. Gate the new branch on --data-parallel-backend ray as well as --enable-elastic-ep. Elastic EP only works on the Ray data-parallel backend - vLLM asserts it in scale_elastic_ep - so a Ray head is useless without it, and three single-pod templates carry --enable-elastic-ep of which only moe_elastic_ep_demo.yaml runs the Ray backend. Without the extra condition, moe_agg.yaml and moe_disagg.yaml would gain a Ray head nothing talks to, behind a 300s startup gate, on configurations that work today. The multinode branch keys on the elastic-EP flag alone and is left untouched: every multinode elastic-EP template sets the Ray backend, so tightening it would change nothing here while altering behaviour for anyone outside the tree. This is what lets a leader come up and serve on its own, with follower pods joining its Ray cluster at an arbitrary later time. Tracking: DYN-3686 Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
WalkthroughThe vLLM backend now detects Ray elastic-EP launches and starts Ray head services for eligible single-pod deployments. Tests cover Ray and non-Ray cases, ChangesElastic EP Ray launch
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
deploy/operator/internal/dynamo/backend_vllm_test.go (1)
202-210: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest readiness-probe preservation.
The fixture creates only liveness and startup probes. The assertion also checks only those probes. A regression that clears
ReadinessProbepasses this test.Add a readiness probe to the single-pod fixture and assert that it remains non-nil.
Also applies to: 288-292
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@deploy/operator/internal/dynamo/backend_vllm_test.go` around lines 202 - 210, Update the single-pod fixture in the relevant backend vLLM tests to include a non-nil ReadinessProbe alongside LivenessProbe and StartupProbe, then extend the probe-preservation assertions to verify ReadinessProbe remains non-nil in both referenced test cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@deploy/operator/internal/dynamo/backend_vllm.go`:
- Around line 488-495: Update hasFlagValue to recognize both separate-argument
flags and inline flag forms such as flag=value, so RoleMain detects the Ray
backend consistently. Add a regression test covering the inline syntax and
preserve the existing behavior for expanded arguments.
---
Nitpick comments:
In `@deploy/operator/internal/dynamo/backend_vllm_test.go`:
- Around line 202-210: Update the single-pod fixture in the relevant backend
vLLM tests to include a non-nil ReadinessProbe alongside LivenessProbe and
StartupProbe, then extend the probe-preservation assertions to verify
ReadinessProbe remains non-nil in both referenced test cases.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 590cc1ac-bf88-42bc-80f6-caafa8792ea2
📒 Files selected for processing (2)
deploy/operator/internal/dynamo/backend_vllm.godeploy/operator/internal/dynamo/backend_vllm_test.go
…stic EP isElasticEPRayLaunch gated the single-pod Ray-head injection on hasFlagValue, which only matched the space-separated "--data-parallel-backend ray" spelling. vLLM's argparse treats "--data-parallel-backend=ray" as equivalent, so a single-pod elastic-EP deployment written with the equals form was silently left without a Ray head and followers could never join the cluster. Reuse the existing hasArg helper, which matches both the "flag value" and "flag=value" forms, and drop the redundant hasFlagValue. Add a regression test for the inline spelling, and extend the single-pod fixture to carry a readiness probe and assert it survives the command rewrite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
This comment has been minimized.
This comment has been minimized.
…astic EP A solo elastic-EP leader starts a Ray head, which Ray registers under the pod IP. vLLM derives the same address for its data-parallel master, but at --data-parallel-size 1 ParallelConfig.__post_init__ takes its offline-SPMD branch and overwrites the derived value with VLLM_DP_MASTER_IP, whose default is 127.0.0.1. Engine startup then looks for a Ray node that does not exist and aborts with "The DP master node (ip: 127.0.0.1) is missing or dead". The overwrite discards both VLLM_HOST_IP and --data-parallel-address, so this env var is the only value that reaches the engine. Bind it to the pod IP behind the same predicate that starts the Ray head, leaving every other container untouched. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
julienmancuso
left a comment
There was a problem hiding this comment.
Requesting changes for three correctness and lifecycle issues in the new single-pod elastic-EP path. The Ray-backend gate and VLLM_DP_MASTER_IP=status.podIP injection otherwise look appropriately scoped, and go test ./internal/dynamo -count=1 plus git diff --check passed locally.
Addresses three review findings on the new single-pod elastic-EP path: - Recognize vLLM's documented `-dpb` short alias for `--data-parallel-backend` (v0.26.0), so `-dpb ray`/`-dpb=ray` also start the Ray head and bind VLLM_DP_MASTER_IP. - Detect the elastic-EP flags across the full container command line (Command + Args), not just Args, so a manifest that carries them in Command no longer bypasses the branch. isElasticEPRayLaunch now takes the container and scans the flattened command line. - exec the foreground vLLM command in the single-pod RoleMain path so it runs as the container's main process and receives SIGTERM directly for a graceful shutdown during rollout/eviction/deletion, instead of being SIGKILLed after the grace period with in-flight requests dropped. Scoped to RoleMain; the multinode RoleLeader keeps its historical no-exec form. Adds regression coverage for the `-dpb` split and equals forms and for flags placed in Command, and updates the RoleMain expectations for exec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
julienmancuso
left a comment
There was a problem hiding this comment.
Re-reviewed the updated head. The three previously requested fixes are present and their new regression cases pass locally. Two backward-compatibility problems remain in the new single-pod RoleMain shell wrapper: it cannot preserve an omitted image ENTRYPOINT, and its argv serialization corrupts valid argument values. go test ./internal/dynamo -count=1 and git diff --check otherwise pass.
…path Addresses two more review findings on the single-pod elastic-EP Ray path: - Preserve the image ENTRYPOINT when Command is empty. A single-pod spec may omit Command and run only the image ENTRYPOINT with Args (a CRD-supported shape). The Ray-head wrapper has no executable to reconstruct in that case, so it previously emitted `... && exec --model ...` and broke the pod. injectElasticEPRayLaunchFlags now returns whether it rewrote the container; for an empty Command it leaves the ENTRYPOINT+Args invocation intact, logs a warning, and the caller skips VLLM_DP_MASTER_IP for the head it did not start. - Serialize argv with an argv-preserving POSIX quoter. shellQuoteForBashC is not argv-safe: single quotes corrupt (it's -> it'"s), empty tokens vanish, and ;, |, & are left unquoted, so a --chat-template with quotes could break a formerly working pod. The elastic-EP leader arm now uses shellQuotePOSIX, which single-quotes any non-trivial token (and the empty string) so it round-trips through sh -c unchanged. Adds a no-Command regression case and an execution-level round-trip test that re-parses quoted tokens through a real /bin/sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
julienmancuso
left a comment
There was a problem hiding this comment.
Posting the remaining P1 from the read-only re-review.
The single-pod elastic-EP path tells vLLM the data-parallel master lives at status.podIP, then leaves Ray to choose where to register the head. vLLM looks up the Ray node under that exact address, so the two only agree as long as Ray's interface heuristic happens to land on the pod IP. On a pod with more than one network it need not, and the engine then aborts with the same "DP master node is missing or dead" that binding VLLM_DP_MASTER_IP exists to prevent. Bind POD_IP from the downward API and interpolate it into --node-ip-address, so both ends resolve from status.podIP and cannot disagree. Scoped to RoleMain. The multinode leader pins neither end, so vLLM and Ray run the same heuristic and agree with each other; adding the flag there without a matching env var would start the head with an empty address. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
…sion Per review, a silent operator Info log is not an acceptable diagnostic when a vLLM component requests elastic EP (--enable-elastic-ep with --data-parallel-backend ray, incl. the -dpb alias) but omits the main container Command. The operator starts the single-pod Ray head by wrapping that command and cannot start it from the image ENTRYPOINT, so elastic scaling would silently never work. Add an admission validation rule (validateElasticEPRequiresCommand) that fails closed with an actionable, user-visible error at DGD and standalone DCD admission, wired into the existing DGD component loop and DCD spec validator next to the vLLM-only inter-pod GMS check. Export IsElasticEPRayLaunch so the webhook reuses the exact same flag detection as the runtime injection. The runtime empty-Command guard stays as defense-in-depth for bypassed admission. Adds a table test for the rule and an end-to-end test proving it rejects at DGD admission. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
julienmancuso
left a comment
There was a problem hiding this comment.
One documentation follow-up from the latest review.
The single-pod Elastic EP path starts the Ray head by wrapping the main container command, so the operator now rejects an Elastic EP component that omits `command` at admission. Document that requirement in the multinode deployment guide's "For Elastic EP" section with a YAML snippet, so users who follow the published steps aren't surprised by the admission rejection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
The ElasticEPRayPoC gate was wrapped around two behaviours that shipped before it existed, so merely upgrading an operator -- which introduces the gate defaulted off -- changed running deployments. Cluster-verified on dynamo-aws-gb300 before this change: flipping the gate off rewrote each leader's pod template (generation 8->9), dropped the Ray head and POD_IP, deleted both follower DCDs, and deleted all three <leader>-ray Services including the Grove one. - The leader Ray-head render (#12943) is ungated again; the gate had been folded into it via IsSinglePodElasticEPLeader. - The Grove <leader>-ray Service (#13178) is ungated again. It is a separate call site, and its argument is a negated 'should I delete this', so gate-off evaluated to delete. - validateElasticEPRequiresCommand (#12943) is ungated again on the DGD path, matching the DCD validator. injectElasticEPRayLaunchFlags declines and only logs when there is no explicit command, so gating this rule made elastic EP silently no-op. - The follower Ray-join rewrite keys off the follower annotation rather than the gate. RoleFollower is assigned from that annotation with no gate term, so a follower outliving a flip rendered the leader's serve command instead. IsSinglePodElasticEPLeader is split into a gate-free shape predicate plus a gated wrapper, so each call site states which question it is asking. Also stop re-asserting the synthesized follower's resting zero on every reconcile. Generation re-derives the follower and stamps Replicas=0, and SyncResource treats an external scale as a manual change and overwrites it -- observed on cluster reverting replicas 1 to 0 within two seconds. Zero is seeded at creation; the scale client owns it after that. Without this the follower can never leave zero, so EP16 (one leader plus three followers at TP4) is unreachable and the feature cannot work at all. All four gate edits and the replica fix are mutation-verified: reverting each one fails its test. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
…before starting
Seeding N-1 followers is only half of a full-width launch. vLLM's
create_dp_placement_groups reads the Ray cluster ONCE at engine start and
allocates one placement group per rank against whatever it finds; nothing
retries. The leader was starting the engine as soon as its own Ray head came up,
so the launch was a race between that and N-1 pods being scheduled, pulled and
joined -- and it lost intermittently. On dynamo-aws-gb300 the same manifest
reached 4 Ray nodes on one attempt and 2 on the next, failing with
ValueError: Not enough resources to allocate 4 placement groups,
only created 2 placement groups
The leader now waits for its declared number of live Ray nodes between bringing
up the head and exec'ing the engine. Counted in nodes rather than GPUs because
the sizing rule is one pod per node per rank, which is the same rule
ElasticEPFollowerReplicas derives the follower count from -- so the two cannot
disagree about what full width means. Bounded at 20 minutes, and the timeout
message reports how many actually joined, because a leader that waits forever is
worse than one that fails.
Emitted only when the component declares more than one rank, so every deployment
that predates this renders byte-for-byte as before. That matters more than it
looks: this is the already-shipped #12943 render, and a pod-template change here
rolls a serving deployment.
Two other things this run surfaced, both fixed here.
An elastic-EP follower is no longer shrunk when the gate is off. Generation owns
the count with the gate off, but ownership is asymmetric: growing back to the
declared width adds ranks, which is safe, while shrinking to it deletes pods that
may hold live engine ranks, which is not. Nothing calls scale_elastic_ep first,
so a shrink leaves the engine committed to a data-parallel size whose members are
gone (DYN-3838, DYN-2660) -- exactly what deleteOrphanedElasticEPFollowers
already refuses to do. Turning the gate off means scaling stops, not that running
capacity is torn out from under a serving engine.
IsSinglePodElasticEPLeader is removed. The previous commit moved the gate off
synthesis, which left this with no non-test callers while its doc comment still
claimed to be "the predicate for everything the PoC introduced". That is the same
trap as the VLLMBackend field removed alongside it: a reviewer would reasonably
conclude the gate still governed rendering.
All three behaviours are mutation-verified -- removing the width gate, dropping
the shrink guard, and re-gating synthesis each fail their own test and nothing
else.
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
The comments added by this branch had grown to 55-70% of the added lines in the
files they touch, which makes the code harder to read rather than easier. This
is a comments-only pass: lead with the conclusion, keep the facts, drop the
repetition and the narration.
Structural fixes, which were the bigger readability problem than length:
- graph.go had a stranded paragraph. synthesizeElasticEPFollowerDCD's
description sat on top of ElasticEPFollowerReplicas' doc comment, so the
function had two half-comments in the wrong places and the reader met the
rationale before the thing it explains.
- The "deliberately not gated" argument appeared three times and the
"do not key on --data-parallel-size" argument twice. Each now has one
definition site; the other places point at it.
- The VLLMBackend doc referred to IsElasticEPLeader, which does not exist.
It now names IsSinglePodElasticEPShape.
Every load-bearing fact is kept: cluster observations, ticket ids (DYN-3838,
DYN-2660, grove#676, #12943), the verbatim vLLM errors, the timeouts and
bounds, the mutation-check notes, and each "why not X" rationale a reviewer
would otherwise re-propose. Three that a verification pass found dropped were
restored: the ordering constraint that --data-parallel-size-local must be
injected before the command is wrapped, why the gate is not plumbed into
VLLMBackend at all, and the pin-side failure of mis-gating the local-rank flag.
Proven comments-only mechanically, not by eye: each file was parsed with
comments stripped and gofmt-normalized before and after, and the two forms are
byte-identical. Full suite green (42 packages).
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
The gate's catalogue entry and the Helm chart still described the scope it had several commits ago: "enables the single-pod Ray head, the headless leader Service, the synthesized follower, and the Ray-specific admission rules", and "turning it back off removes the followers it synthesized". None of that is true now. The Ray head shipped in #12943, the leader Service in #13178, and follower synthesis renders a deployment's declared width -- all three are deliberately ungated, because gating them would strip a live deployment's Ray head on upgrade or render a fraction of the engine the user asked for. The follower's replica count freezes at either gate position, so turning the gate off removes nothing. What the gate governs is one rule: an elastic-EP leader may declare at most one replica. Both places now say that, and say what it deliberately does not cover, so an administrator reading either one is not told the default-off setting gives them a cluster with no elastic-EP behaviour. This is the administrator-facing contract, and it drifted as part of narrowing the gate in 6376649. README.md regenerated with `make generate-helm-docs` rather than hand-edited. Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Overview:
What we want. A Dynamo deployment where one pod runs the vLLM leader and serves on its own, and where extra pods can join later to widen the expert-parallel group. Elastic EP is the vLLM feature that makes that group resizable while the engine is running. It works by putting every participating pod into one Ray cluster, with the leader running the Ray head.
What is broken today. Deploy a component as a single pod, pass
--enable-elastic-ep, and nothing happens. No Ray head starts, so there is no cluster for anything to join, and the flag is silently ignored. Elastic EP works only if the component is declared multinode.Why. Two independent reasons, either of which alone is enough:
numberOfNodes > 1branch ofUpdateContainer, so a one-pod component never reaches it.injectElasticEPRayLaunchFlagsmatches onlyRoleLeader— and a one-pod component is expanded asRoleMain, so it would fall straight through.What this changes. Add a single-pod branch beside the multinode one, and let the leader arm match
RoleMainas well asRoleLeader. A solo leader then startsray start --headexactly as a multinode leader does, and is told where to find it viaVLLM_DP_MASTER_IP.This is worth having on its own, since single-pod elastic EP is simply broken today. It is also the first step toward a leader that serves alone with followers attaching at an arbitrary later time.
Details:
The wiring
UpdateContainerindeploy/operator/internal/dynamo/backend_vllm.gogains anelse ifbranch for single-pod components, andinjectElasticEPRayLaunchFlagsnow treatsRoleMainidentically toRoleLeader.Probes are deliberately left in place. The multinode path strips them from workers, but this pod is a leader that serves traffic, so it keeps them.
Why the branch also requires the Ray backend
The new branch fires only when the args carry both
--enable-elastic-epand--data-parallel-backend ray, expressed as a named predicateisElasticEPRayLaunch.The second condition is load-bearing, not defensive. Elastic EP only functions on the Ray data-parallel backend — vLLM asserts it outright in
scale_elastic_epwith "Only ray DP backend supports scaling elastic EP". On any other backend, a Ray head is a process nothing ever connects to, sitting behind a 300-second startup gate.That is not hypothetical. Three single-pod templates in this repo pass
--enable-elastic-ep:moe_elastic_ep_demo.yamlmoe_agg.yamlmoe_disagg.yamlThe latter two are not misconfigured.
--enable-elastic-epstill does real work for them: it requires EPLB, rejects pipeline parallelism, and selects thepyncclEPLB communicator. They simply cannot scale at runtime. Without the backend condition, this PR would have added a Ray head and a startup gate to two configurations that work today.Telling vLLM where the DP master is
Starting the Ray head is necessary but not sufficient, and the second half is not obvious.
Ray registers the head under the pod IP, since the whole point is that another pod reaches it later. vLLM derives the same address for its data-parallel master —
arg_utils.pycallsget_ip()for the Ray backend and logs "Using host IP %s as ray-based data parallel address". So far so good.Then
ParallelConfig.__post_init__throws it away. Its condition isdata_parallel_size > 1 or data_parallel_size_local == 0, and at--data-parallel-size 1neither holds, so it takes the offline-SPMD branch and overwrites the derived address withenvs.VLLM_DP_MASTER_IP— default127.0.0.1. Engine startup then asserts on a Ray node that does not exist:The root cause upstream is a conflated signal:
data_parallel_sizedefaults to 1, so vLLM cannot distinguish "DP not configured" from "DP configured with one rank". A solo elastic-EP leader is exactly the second, read as the first. Reported as vllm-project/vllm#51712; if it is fixed upstream this injection stops being load-bearing, but it is correct to carry it either way.Two things that look like they should fix it, and do not.
VLLM_HOST_IPonly steersget_ip(), whose result is then discarded.--data-parallel-addressis discarded by the same branch.VLLM_DP_MASTER_IPis the one value the fallback actually reads, so the operator binds it tostatus.podIPbehind the same predicate that starts the Ray head. This mirrors the existingVLLM_NIXL_SIDE_CHANNEL_HOSTinjection a few lines above.Note this only bites at dp=1.
moe_elastic_ep_demo.yamlruns dp=2 in one pod and takes the other branch, which is why single-pod elastic EP appeared to work in that shape.Naming the head's address, not guessing it
Telling vLLM the master is at
status.podIPis only half of a pair. vLLM then looks up the Ray node registered under that exact address, and Ray, left alone, picks an interface by its own heuristic. The two agree only as long as that heuristic lands on the pod IP — true on a single-interface pod, not guaranteed on one with a second network, and a mismatch there produces the identical "missing or dead" abort.Before this change nothing was pinned at either end, so vLLM's
get_ip()and Ray's detection ran the same heuristic and agreed with each other whatever they chose. Pinning one end alone would trade a self-consistent pair for a mismatched one.So the head is given its address explicitly:
POD_IPis bound from the downward API and interpolated into--node-ip-address. Both values resolve fromstatus.podIP, which is the point — they cannot drift. This also makes the generated command match the hand-written leader command already validated on a GB200 cluster.The flag is scoped to
RoleMain. A multinode leader gets noVLLM_DP_MASTER_IP, so adding--node-ip-addressthere would interpolate an unset variable and start the head with an empty address.What is deliberately left alone
The multinode branch keys on the elastic-EP flag alone and has the same looseness. It stays as it is: every multinode elastic-EP template in the tree already sets the Ray backend, so tightening it would change nothing here while altering behaviour for anyone outside the tree. Worth doing, but as its own change.
Tests
Six cases in
backend_vllm_test.go, all confirmed to fail without the corresponding production change:$POD_IP, and keeps its probes;RoleMaintakes the leader arm;Both env bindings are asserted on every case in the table rather than only the positive ones: present and bound to
status.podIPwhere a Ray head is started, absent everywhere else, including the whole multinode set. That makes it impossible for either to leak into a container that did not ask for a head, and the multinode expectations double as the guard that their rendered command is unchanged.Where should the reviewer start?
backend_vllm.go: the newelse ifinUpdateContainer, and theisElasticEPRayLaunchpredicate. The judgement call worth pressing on is whether requiring the Ray backend is right, or whether this should mirror the looser multinode branch instead.Then
TestVLLMBackend_UpdateContainer/single_node_elastic_EP_without_the_ray_backend_is_left_alone, which is the test that keepsmoe_agg.yamlandmoe_disagg.yamlrendering exactly as they do today.The
VLLM_DP_MASTER_IPbinding is the other place to look, and the question there is scope: it is gated on the same predicate as the Ray head, which deliberately leaves the multinode path alone even though the same vLLM behaviour would apply to a multinode leader at dp=1.Related Issues
🚫 This PR is NOT linked to an issue: