Skip to content

feat(operator): start a Ray head on a single-pod elastic EP component - #12943

Merged
tzulingk merged 8 commits into
mainfrom
tzulingk/elastic-ep-p2-ray-head
Aug 12, 2026
Merged

tzulingk merged 8 commits into
mainfrom
tzulingk/elastic-ep-p2-ray-head

Conversation

@tzulingk

@tzulingk tzulingk commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

What we want. A Dynamo deployment where one pod runs the vLLM leader and serves on its own, and where extra pods can join later to widen the expert-parallel group. Elastic EP is the vLLM feature that makes that group resizable while the engine is running. It works by putting every participating pod into one Ray cluster, with the leader running the Ray head.

What is broken today. Deploy a component as a single pod, pass --enable-elastic-ep, and nothing happens. No Ray head starts, so there is no cluster for anything to join, and the flag is silently ignored. Elastic EP works only if the component is declared multinode.

Why. Two independent reasons, either of which alone is enough:

  1. The elastic-EP dispatch sits inside the numberOfNodes > 1 branch of UpdateContainer, so a one-pod component never reaches it.
  2. Even if it did, the leader wiring in injectElasticEPRayLaunchFlags matches only RoleLeader — and a one-pod component is expanded as RoleMain, so it would fall straight through.

What this changes. Add a single-pod branch beside the multinode one, and let the leader arm match RoleMain as well as RoleLeader. A solo leader then starts ray start --head exactly as a multinode leader does, and is told where to find it via VLLM_DP_MASTER_IP.

This is worth having on its own, since single-pod elastic EP is simply broken today. It is also the first step toward a leader that serves alone with followers attaching at an arbitrary later time.

Details:

The wiring

UpdateContainer in deploy/operator/internal/dynamo/backend_vllm.go gains an else if branch for single-pod components, and injectElasticEPRayLaunchFlags now treats RoleMain identically to RoleLeader.

Probes are deliberately left in place. The multinode path strips them from workers, but this pod is a leader that serves traffic, so it keeps them.

Why the branch also requires the Ray backend

The new branch fires only when the args carry both --enable-elastic-ep and --data-parallel-backend ray, expressed as a named predicate isElasticEPRayLaunch.

The second condition is load-bearing, not defensive. Elastic EP only functions on the Ray data-parallel backend — vLLM asserts it outright in scale_elastic_ep with "Only ray DP backend supports scaling elastic EP". On any other backend, a Ray head is a process nothing ever connects to, sitting behind a 300-second startup gate.

That is not hypothetical. Three single-pod templates in this repo pass --enable-elastic-ep:

Template Ray DP backend Without the extra condition
moe_elastic_ep_demo.yaml yes Correct — gains the Ray head it needs
moe_agg.yaml no Would gain a Ray head nothing uses
moe_disagg.yaml no Same, on both of its components

The latter two are not misconfigured. --enable-elastic-ep still does real work for them: it requires EPLB, rejects pipeline parallelism, and selects the pynccl EPLB communicator. They simply cannot scale at runtime. Without the backend condition, this PR would have added a Ray head and a startup gate to two configurations that work today.

Telling vLLM where the DP master is

Starting the Ray head is necessary but not sufficient, and the second half is not obvious.

Ray registers the head under the pod IP, since the whole point is that another pod reaches it later. vLLM derives the same address for its data-parallel master — arg_utils.py calls get_ip() for the Ray backend and logs "Using host IP %s as ray-based data parallel address". So far so good.

Then ParallelConfig.__post_init__ throws it away. Its condition is data_parallel_size > 1 or data_parallel_size_local == 0, and at --data-parallel-size 1 neither holds, so it takes the offline-SPMD branch and overwrites the derived address with envs.VLLM_DP_MASTER_IP — default 127.0.0.1. Engine startup then asserts on a Ray node that does not exist:

The DP master node (ip: 127.0.0.1) is missing or dead

The root cause upstream is a conflated signal: data_parallel_size defaults to 1, so vLLM cannot distinguish "DP not configured" from "DP configured with one rank". A solo elastic-EP leader is exactly the second, read as the first. Reported as vllm-project/vllm#51712; if it is fixed upstream this injection stops being load-bearing, but it is correct to carry it either way.

Two things that look like they should fix it, and do not. VLLM_HOST_IP only steers get_ip(), whose result is then discarded. --data-parallel-address is discarded by the same branch. VLLM_DP_MASTER_IP is the one value the fallback actually reads, so the operator binds it to status.podIP behind the same predicate that starts the Ray head. This mirrors the existing VLLM_NIXL_SIDE_CHANNEL_HOST injection a few lines above.

Note this only bites at dp=1. moe_elastic_ep_demo.yaml runs dp=2 in one pod and takes the other branch, which is why single-pod elastic EP appeared to work in that shape.

Naming the head's address, not guessing it

Telling vLLM the master is at status.podIP is only half of a pair. vLLM then looks up the Ray node registered under that exact address, and Ray, left alone, picks an interface by its own heuristic. The two agree only as long as that heuristic lands on the pod IP — true on a single-interface pod, not guaranteed on one with a second network, and a mismatch there produces the identical "missing or dead" abort.

Before this change nothing was pinned at either end, so vLLM's get_ip() and Ray's detection ran the same heuristic and agreed with each other whatever they chose. Pinning one end alone would trade a self-consistent pair for a mismatched one.

So the head is given its address explicitly: POD_IP is bound from the downward API and interpolated into --node-ip-address. Both values resolve from status.podIP, which is the point — they cannot drift. This also makes the generated command match the hand-written leader command already validated on a GB200 cluster.

The flag is scoped to RoleMain. A multinode leader gets no VLLM_DP_MASTER_IP, so adding --node-ip-address there would interpolate an unset variable and start the head with an empty address.

What is deliberately left alone

The multinode branch keys on the elastic-EP flag alone and has the same looseness. It stays as it is: every multinode elastic-EP template in the tree already sets the Ray backend, so tightening it would change nothing here while altering behaviour for anyone outside the tree. Worth doing, but as its own change.

Tests

Six cases in backend_vllm_test.go, all confirmed to fail without the corresponding production change:

  • a single-pod elastic-EP component gets the Ray head pinned to $POD_IP, and keeps its probes;
  • RoleMain takes the leader arm;
  • four regression guards — no elastic-EP flag, the flag with no backend specified, and the flag on an explicitly non-Ray backend — each asserting the container is left untouched.

Both env bindings are asserted on every case in the table rather than only the positive ones: present and bound to status.podIP where a Ray head is started, absent everywhere else, including the whole multinode set. That makes it impossible for either to leak into a container that did not ask for a head, and the multinode expectations double as the guard that their rendered command is unchanged.

Where should the reviewer start?

backend_vllm.go: the new else if in UpdateContainer, and the isElasticEPRayLaunch predicate. The judgement call worth pressing on is whether requiring the Ray backend is right, or whether this should mirror the looser multinode branch instead.

Then TestVLLMBackend_UpdateContainer/single_node_elastic_EP_without_the_ray_backend_is_left_alone, which is the test that keeps moe_agg.yaml and moe_disagg.yaml rendering exactly as they do today.

The VLLM_DP_MASTER_IP binding is the other place to look, and the question there is scope: it is gated on the same predicate as the Ray head, which deliberately leaves the multinode path alone even though the same vLLM behaviour would apply to a multinode leader at dp=1.

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

Elastic-EP wiring was reachable only through the isMultinode branch of
UpdateContainer, so a component deployed as one pod never ran
'ray start --head'. Two things blocked it: the dispatch sat inside
'numberOfNodes > 1', and the leader arm of injectElasticEPRayLaunchFlags
matched RoleLeader only, while a lone pod is expanded as RoleMain.

Route single-pod components to the same injection and let the leader arm
match RoleMain. Probes are left in place: unlike a multinode worker, this pod
serves traffic.

Gate the new branch on --data-parallel-backend ray as well as
--enable-elastic-ep. Elastic EP only works on the Ray data-parallel backend -
vLLM asserts it in scale_elastic_ep - so a Ray head is useless without it, and
three single-pod templates carry --enable-elastic-ep of which only
moe_elastic_ep_demo.yaml runs the Ray backend. Without the extra condition,
moe_agg.yaml and moe_disagg.yaml would gain a Ray head nothing talks to,
behind a 300s startup gate, on configurations that work today. The multinode
branch keys on the elastic-EP flag alone and is left untouched: every
multinode elastic-EP template sets the Ray backend, so tightening it would
change nothing here while altering behaviour for anyone outside the tree.

This is what lets a leader come up and serve on its own, with follower pods
joining its Ray cluster at an arbitrary later time.

Tracking: DYN-3686
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@tzulingk
tzulingk requested a review from a team as a code owner August 10, 2026 16:30
@github-actions github-actions Bot added feat deployment::k8s Relates to dynamo deployment in kubernetes labels Aug 10, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The vLLM backend now detects Ray elastic-EP launches and starts Ray head services for eligible single-pod deployments. Tests cover Ray and non-Ray cases, RoleMain, and probe preservation.

Changes

Elastic EP Ray launch

Layer / File(s) Summary
Ray elastic-EP detection and startup
deploy/operator/internal/dynamo/backend_vllm.go
The backend requires both elastic-EP and Ray data-parallel flags. Single-pod RoleMain deployments now use Ray leader startup and readiness polling.
Ray launch regression coverage
deploy/operator/internal/dynamo/backend_vllm_test.go
Tests cover Ray and non-Ray launch behavior, RoleMain startup, and preservation of liveness and startup probes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: starting a Ray head for single-pod elastic EP components.
Description check ✅ Passed The description covers all required sections, explains the implementation and tests, identifies review points, and confirms that no related issue exists.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
deploy/operator/internal/dynamo/backend_vllm_test.go (1)

202-210: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test readiness-probe preservation.

The fixture creates only liveness and startup probes. The assertion also checks only those probes. A regression that clears ReadinessProbe passes this test.

Add a readiness probe to the single-pod fixture and assert that it remains non-nil.

Also applies to: 288-292

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/dynamo/backend_vllm_test.go` around lines 202 - 210,
Update the single-pod fixture in the relevant backend vLLM tests to include a
non-nil ReadinessProbe alongside LivenessProbe and StartupProbe, then extend the
probe-preservation assertions to verify ReadinessProbe remains non-nil in both
referenced test cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/operator/internal/dynamo/backend_vllm.go`:
- Around line 488-495: Update hasFlagValue to recognize both separate-argument
flags and inline flag forms such as flag=value, so RoleMain detects the Ray
backend consistently. Add a regression test covering the inline syntax and
preserve the existing behavior for expanded arguments.

---

Nitpick comments:
In `@deploy/operator/internal/dynamo/backend_vllm_test.go`:
- Around line 202-210: Update the single-pod fixture in the relevant backend
vLLM tests to include a non-nil ReadinessProbe alongside LivenessProbe and
StartupProbe, then extend the probe-preservation assertions to verify
ReadinessProbe remains non-nil in both referenced test cases.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 590cc1ac-bf88-42bc-80f6-caafa8792ea2

📥 Commits

Reviewing files that changed from the base of the PR and between 9eafac6 and f3b0403.

📒 Files selected for processing (2)
  • deploy/operator/internal/dynamo/backend_vllm.go
  • deploy/operator/internal/dynamo/backend_vllm_test.go

@tzulingk
tzulingk enabled auto-merge (squash) August 10, 2026 16:59
…stic EP

isElasticEPRayLaunch gated the single-pod Ray-head injection on
hasFlagValue, which only matched the space-separated
"--data-parallel-backend ray" spelling. vLLM's argparse treats
"--data-parallel-backend=ray" as equivalent, so a single-pod elastic-EP
deployment written with the equals form was silently left without a Ray
head and followers could never join the cluster.

Reuse the existing hasArg helper, which matches both the "flag value" and
"flag=value" forms, and drop the redundant hasFlagValue. Add a regression
test for the inline spelling, and extend the single-pod fixture to carry a
readiness probe and assert it survives the command rewrite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@datadog-official

This comment has been minimized.

…astic EP

A solo elastic-EP leader starts a Ray head, which Ray registers under the
pod IP. vLLM derives the same address for its data-parallel master, but at
--data-parallel-size 1 ParallelConfig.__post_init__ takes its offline-SPMD
branch and overwrites the derived value with VLLM_DP_MASTER_IP, whose
default is 127.0.0.1. Engine startup then looks for a Ray node that does
not exist and aborts with "The DP master node (ip: 127.0.0.1) is missing
or dead".

The overwrite discards both VLLM_HOST_IP and --data-parallel-address, so
this env var is the only value that reaches the engine. Bind it to the pod
IP behind the same predicate that starts the Ray head, leaving every other
container untouched.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

@julienmancuso julienmancuso left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for three correctness and lifecycle issues in the new single-pod elastic-EP path. The Ray-backend gate and VLLM_DP_MASTER_IP=status.podIP injection otherwise look appropriately scoped, and go test ./internal/dynamo -count=1 plus git diff --check passed locally.

Addresses three review findings on the new single-pod elastic-EP path:

- Recognize vLLM's documented `-dpb` short alias for
  `--data-parallel-backend` (v0.26.0), so `-dpb ray`/`-dpb=ray` also start
  the Ray head and bind VLLM_DP_MASTER_IP.
- Detect the elastic-EP flags across the full container command line
  (Command + Args), not just Args, so a manifest that carries them in
  Command no longer bypasses the branch. isElasticEPRayLaunch now takes the
  container and scans the flattened command line.
- exec the foreground vLLM command in the single-pod RoleMain path so it
  runs as the container's main process and receives SIGTERM directly for a
  graceful shutdown during rollout/eviction/deletion, instead of being
  SIGKILLed after the grace period with in-flight requests dropped. Scoped
  to RoleMain; the multinode RoleLeader keeps its historical no-exec form.

Adds regression coverage for the `-dpb` split and equals forms and for
flags placed in Command, and updates the RoleMain expectations for exec.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@tzulingk
tzulingk requested a review from julienmancuso August 10, 2026 17:49

@julienmancuso julienmancuso left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the updated head. The three previously requested fixes are present and their new regression cases pass locally. Two backward-compatibility problems remain in the new single-pod RoleMain shell wrapper: it cannot preserve an omitted image ENTRYPOINT, and its argv serialization corrupts valid argument values. go test ./internal/dynamo -count=1 and git diff --check otherwise pass.

…path

Addresses two more review findings on the single-pod elastic-EP Ray path:

- Preserve the image ENTRYPOINT when Command is empty. A single-pod spec may
  omit Command and run only the image ENTRYPOINT with Args (a CRD-supported
  shape). The Ray-head wrapper has no executable to reconstruct in that case,
  so it previously emitted `... && exec --model ...` and broke the pod.
  injectElasticEPRayLaunchFlags now returns whether it rewrote the container;
  for an empty Command it leaves the ENTRYPOINT+Args invocation intact, logs a
  warning, and the caller skips VLLM_DP_MASTER_IP for the head it did not start.

- Serialize argv with an argv-preserving POSIX quoter. shellQuoteForBashC is
  not argv-safe: single quotes corrupt (it's -> it'"s), empty tokens vanish,
  and ;, |, & are left unquoted, so a --chat-template with quotes could break a
  formerly working pod. The elastic-EP leader arm now uses shellQuotePOSIX,
  which single-quotes any non-trivial token (and the empty string) so it
  round-trips through sh -c unchanged.

Adds a no-Command regression case and an execution-level round-trip test that
re-parses quoted tokens through a real /bin/sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@tzulingk
tzulingk requested a review from julienmancuso August 10, 2026 18:21

@julienmancuso julienmancuso left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posting the remaining P1 from the read-only re-review.

tzulingk and others added 2 commits August 10, 2026 15:09
The single-pod elastic-EP path tells vLLM the data-parallel master lives at
status.podIP, then leaves Ray to choose where to register the head. vLLM looks
up the Ray node under that exact address, so the two only agree as long as
Ray's interface heuristic happens to land on the pod IP. On a pod with more
than one network it need not, and the engine then aborts with the same "DP
master node is missing or dead" that binding VLLM_DP_MASTER_IP exists to
prevent.

Bind POD_IP from the downward API and interpolate it into
--node-ip-address, so both ends resolve from status.podIP and cannot disagree.

Scoped to RoleMain. The multinode leader pins neither end, so vLLM and Ray run
the same heuristic and agree with each other; adding the flag there without a
matching env var would start the head with an empty address.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
…sion

Per review, a silent operator Info log is not an acceptable diagnostic when a
vLLM component requests elastic EP (--enable-elastic-ep with
--data-parallel-backend ray, incl. the -dpb alias) but omits the main container
Command. The operator starts the single-pod Ray head by wrapping that command
and cannot start it from the image ENTRYPOINT, so elastic scaling would silently
never work.

Add an admission validation rule (validateElasticEPRequiresCommand) that fails
closed with an actionable, user-visible error at DGD and standalone DCD
admission, wired into the existing DGD component loop and DCD spec validator
next to the vLLM-only inter-pod GMS check. Export IsElasticEPRayLaunch so the
webhook reuses the exact same flag detection as the runtime injection. The
runtime empty-Command guard stays as defense-in-depth for bypassed admission.

Adds a table test for the rule and an end-to-end test proving it rejects at DGD
admission.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>

@julienmancuso julienmancuso left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One documentation follow-up from the latest review.

The single-pod Elastic EP path starts the Ray head by wrapping the main
container command, so the operator now rejects an Elastic EP component that
omits `command` at admission. Document that requirement in the multinode
deployment guide's "For Elastic EP" section with a YAML snippet, so users who
follow the published steps aren't surprised by the admission rejection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
@tzulingk
tzulingk requested a review from a team as a code owner August 12, 2026 16:47
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 12, 2026
@tzulingk
tzulingk requested a review from julienmancuso August 12, 2026 16:48
@github-actions

Copy link
Copy Markdown
Contributor

@tzulingk
tzulingk requested a review from jh-nv August 12, 2026 19:15

@jh-nv jh-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc owner approval

@tzulingk
tzulingk merged commit 865afbb into main Aug 12, 2026
110 of 112 checks passed
@tzulingk
tzulingk deleted the tzulingk/elastic-ep-p2-ray-head branch August 12, 2026 19:35
tzulingk added a commit that referenced this pull request Sep 14, 2026
The ElasticEPRayPoC gate was wrapped around two behaviours that shipped
before it existed, so merely upgrading an operator -- which introduces the
gate defaulted off -- changed running deployments.

Cluster-verified on dynamo-aws-gb300 before this change: flipping the gate
off rewrote each leader's pod template (generation 8->9), dropped the Ray
head and POD_IP, deleted both follower DCDs, and deleted all three
<leader>-ray Services including the Grove one.

- The leader Ray-head render (#12943) is ungated again; the gate had been
  folded into it via IsSinglePodElasticEPLeader.
- The Grove <leader>-ray Service (#13178) is ungated again. It is a separate
  call site, and its argument is a negated 'should I delete this', so gate-off
  evaluated to delete.
- validateElasticEPRequiresCommand (#12943) is ungated again on the DGD path,
  matching the DCD validator. injectElasticEPRayLaunchFlags declines and only
  logs when there is no explicit command, so gating this rule made elastic EP
  silently no-op.
- The follower Ray-join rewrite keys off the follower annotation rather than
  the gate. RoleFollower is assigned from that annotation with no gate term,
  so a follower outliving a flip rendered the leader's serve command instead.

IsSinglePodElasticEPLeader is split into a gate-free shape predicate plus a
gated wrapper, so each call site states which question it is asking.

Also stop re-asserting the synthesized follower's resting zero on every
reconcile. Generation re-derives the follower and stamps Replicas=0, and
SyncResource treats an external scale as a manual change and overwrites it --
observed on cluster reverting replicas 1 to 0 within two seconds. Zero is
seeded at creation; the scale client owns it after that. Without this the
follower can never leave zero, so EP16 (one leader plus three followers at
TP4) is unreachable and the feature cannot work at all.

All four gate edits and the replica fix are mutation-verified: reverting each
one fails its test.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
tzulingk added a commit that referenced this pull request Sep 16, 2026
…before starting

Seeding N-1 followers is only half of a full-width launch. vLLM's
create_dp_placement_groups reads the Ray cluster ONCE at engine start and
allocates one placement group per rank against whatever it finds; nothing
retries. The leader was starting the engine as soon as its own Ray head came up,
so the launch was a race between that and N-1 pods being scheduled, pulled and
joined -- and it lost intermittently. On dynamo-aws-gb300 the same manifest
reached 4 Ray nodes on one attempt and 2 on the next, failing with

  ValueError: Not enough resources to allocate 4 placement groups,
              only created 2 placement groups

The leader now waits for its declared number of live Ray nodes between bringing
up the head and exec'ing the engine. Counted in nodes rather than GPUs because
the sizing rule is one pod per node per rank, which is the same rule
ElasticEPFollowerReplicas derives the follower count from -- so the two cannot
disagree about what full width means. Bounded at 20 minutes, and the timeout
message reports how many actually joined, because a leader that waits forever is
worse than one that fails.

Emitted only when the component declares more than one rank, so every deployment
that predates this renders byte-for-byte as before. That matters more than it
looks: this is the already-shipped #12943 render, and a pod-template change here
rolls a serving deployment.

Two other things this run surfaced, both fixed here.

An elastic-EP follower is no longer shrunk when the gate is off. Generation owns
the count with the gate off, but ownership is asymmetric: growing back to the
declared width adds ranks, which is safe, while shrinking to it deletes pods that
may hold live engine ranks, which is not. Nothing calls scale_elastic_ep first,
so a shrink leaves the engine committed to a data-parallel size whose members are
gone (DYN-3838, DYN-2660) -- exactly what deleteOrphanedElasticEPFollowers
already refuses to do. Turning the gate off means scaling stops, not that running
capacity is torn out from under a serving engine.

IsSinglePodElasticEPLeader is removed. The previous commit moved the gate off
synthesis, which left this with no non-test callers while its doc comment still
claimed to be "the predicate for everything the PoC introduced". That is the same
trap as the VLLMBackend field removed alongside it: a reviewer would reasonably
conclude the gate still governed rendering.

All three behaviours are mutation-verified -- removing the width gate, dropping
the shrink guard, and re-gating synthesis each fail their own test and nothing
else.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
tzulingk added a commit that referenced this pull request Sep 16, 2026
The comments added by this branch had grown to 55-70% of the added lines in the
files they touch, which makes the code harder to read rather than easier. This
is a comments-only pass: lead with the conclusion, keep the facts, drop the
repetition and the narration.

Structural fixes, which were the bigger readability problem than length:

  - graph.go had a stranded paragraph. synthesizeElasticEPFollowerDCD's
    description sat on top of ElasticEPFollowerReplicas' doc comment, so the
    function had two half-comments in the wrong places and the reader met the
    rationale before the thing it explains.
  - The "deliberately not gated" argument appeared three times and the
    "do not key on --data-parallel-size" argument twice. Each now has one
    definition site; the other places point at it.
  - The VLLMBackend doc referred to IsElasticEPLeader, which does not exist.
    It now names IsSinglePodElasticEPShape.

Every load-bearing fact is kept: cluster observations, ticket ids (DYN-3838,
DYN-2660, grove#676, #12943), the verbatim vLLM errors, the timeouts and
bounds, the mutation-check notes, and each "why not X" rationale a reviewer
would otherwise re-propose. Three that a verification pass found dropped were
restored: the ordering constraint that --data-parallel-size-local must be
injected before the command is wrapped, why the gate is not plumbed into
VLLMBackend at all, and the pin-side failure of mis-gating the local-rank flag.

Proven comments-only mechanically, not by eye: each file was parsed with
comments stripped and gofmt-normalized before and after, and the two forms are
byte-identical. Full suite green (42 packages).

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
tzulingk added a commit that referenced this pull request Sep 16, 2026
The gate's catalogue entry and the Helm chart still described the scope it had
several commits ago: "enables the single-pod Ray head, the headless leader
Service, the synthesized follower, and the Ray-specific admission rules", and
"turning it back off removes the followers it synthesized".

None of that is true now. The Ray head shipped in #12943, the leader Service in
#13178, and follower synthesis renders a deployment's declared width -- all
three are deliberately ungated, because gating them would strip a live
deployment's Ray head on upgrade or render a fraction of the engine the user
asked for. The follower's replica count freezes at either gate position, so
turning the gate off removes nothing.

What the gate governs is one rule: an elastic-EP leader may declare at most one
replica. Both places now say that, and say what it deliberately does not cover,
so an administrator reading either one is not told the default-off setting
gives them a cluster with no elastic-EP behaviour.

This is the administrator-facing contract, and it drifted as part of narrowing
the gate in 6376649.

README.md regenerated with `make generate-helm-docs` rather than hand-edited.

Signed-off-by: Tzu-Ling <tzulingk@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants