Repository navigation
Conversation
…card The frontend held a single `router_mode` and used it for both routers in a disaggregated pipeline: the decode router built in the watcher and the prefill router built underneath it. `--router-mode kv` meant KV on both hops and `--router-mode round-robin` meant round-robin on both, with no way to ask for one on prefill and the other on decode. Per-worker-set `RouterConfig` already rides in the model deployment card, and `register_model` already takes both `worker_type` and `router_config`, so a prefill worker can already declare how it wants to be routed to. Nothing read it: the watcher short-circuits prefill cards before applying `router_config`, and the prefill router inherited the decode set's mode instead. Resolve the prefill hop's mode from the prefill worker set's own cards at activation, falling back to the decode set's mode when the card declares nothing. `build_binding` already fetched those cards to read `enable_eagle`, so this reuses that read rather than adding one. A KV-routed prefill tier in front of a round-robin decode tier now works, as does the reverse. Three things this required getting right: - `router_mode` on `PrefillRouter` meant two things. Conditional disagg peeks the decode router, so it needs the decode set's mode; the Direct-routing guard is about the prefill dispatch, so it needs the prefill hop's. Split them: the field is now `decode_router_mode`, and the resolved prefill mode lives on `PrefillBinding` because it is only knowable once a prefill target is discovered. The Direct guard moved after the binding load accordingly. - A card that declares a mode may declare KV tuning with it. Honoring only half of a `RouterConfig` would be a trap, so an advertised `kv_router_config` is used too, with `router_track_active_blocks` still forced off as the decode set does. - An unreadable card now fails activation instead of silently inheriting. `drive_target` already retries with backoff, and a delayed activation costs far less than routing a whole deployment with the wrong strategy because of one transient discovery miss -- a downgrade that would persist until the binding was rebuilt. A prefill fleet mid-rolling-update can disagree about its mode; the first card wins, and the split is logged rather than passing silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Asserts the two hops route on their own terms rather than sharing one mode. Prefill mockers advertise RouterConfig(RouterMode.KV); decode mockers advertise nothing and inherit the frontend's --router-mode round-robin. Prefix-extending requests must converge on one prefill worker, which only KV routing produces, while spreading across more than one decode worker, which only round-robin produces. The second assertion is the one that fails if the per-role override is ignored and the pipeline silently runs in a single mode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Self-review follow-up to the prefill card resolution. `build_binding` now reads `enable_eagle` from the prefill card it already fetches, so the value threaded through `PrefillRouter::new` was dead. Every caller was computing the same lookup and handing it in: the frontend hardcoded `false` (already overridden by the card read), while ext-proc and the C bindings passed `card.runtime_config.enable_eagle` -- the same field, from a card. Both keep their copy for the decode chooser, which genuinely needs it. Removing the parameter is what makes that visible; keeping it would have left a public argument that is silently ignored. Also from the review: - Log cards that fail to deserialize. An unreadable card used to cost only an EAGLE hint; it now drops a worker out of the vote that decides how the hop is routed, so it should not vanish silently. - Drop `PrefillAdvertisement::advertised`; it was exactly `kv_router_config.is_some()` and existed only to feed one tracing field. - Stop shadowing the `kv_router_config` parameter in `build_binding`. - Remove the doubled error context around card resolution. - Trim comments that restated the code, and the test that asserted the override works in the reverse direction -- same assertion with the modes swapped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
…d too The prefill hop took its mode and KV tuning from the prefill card but its block size and session-affinity TTL from the decode set, so a prefill tier could not fully describe how it wants to be routed to. Resolve all four from the same place. Block size is the one that can be wrong rather than merely inflexible: the prefill pool's KV events are keyed on its own block size, so using the decode set's would index that pool at the wrong granularity. Disaggregation requires the two to match for KV transfer, which is why this has not bitten anyone, but the frontend should not be the reason it holds. A card reporting 0 never declared one, so the decode set's value still applies there. Session TTL rides in `router_config`, so it follows the same rule as the KV tuning: an advertisement replaces the decode set's configuration rather than merging with it. A card that advertises without a TTL therefore means "no affinity on this hop", not "inherit the frontend's" -- hence `Option<Option<_>>` to keep "advertised nothing" distinct from "advertised no TTL". Block size is not part of `router_config` and is intrinsic to the worker, so it is taken unconditionally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Asserts the prefill and decode hops take their session-affinity setting from their own cards rather than sharing one. Both tiers advertise round-robin and only prefill advertises a TTL, so affinity is the sole difference: requests carrying one session id stay on a single prefill worker while rotating across decode workers. Observed: prefill `[458, 458, 458, 458]`, decode `[501, 498, 501, 498]`. Giving decode a TTL as well collapses it to `[861, 861, 861, 861]` and fails the decode assertion, so the test is sensitive to the setting rather than to round-robin timing. This covers configuration, not expiry. Proving a TTL elapses means sleeping past it and asserting a rebind, which is timing-dependent and can legitimately re-select the same worker. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
The helper landed in a584b62 without the test that calls it: reverting a throwaway negative-control edit with `git checkout --` also discarded the wrapper, because that file had not been committed yet. The helper was dead code and the scenario never ran. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
An advertised card replaces the decode set's configuration rather than merging with it, so a worker that names a mode without restating tuning silently gets defaults. Nothing reported which values a hop ended up with, leaving "why did my temperature reset" answerable only by reading code. Log the resolved values -- mode, block size, session TTL, KV tuning -- once per activation, after resolution rather than before, so the line reflects what the hop actually uses rather than what the card asked for. The docs said "restate your tuning" without saying why. State the rule directly: on a worker an unset flag means the default, not "inherit the frontend's". Also note that both sides build from the same argument groups, so defaults are identical and values diverge only when the frontend is tuned and the worker is not -- and that a shared environment changes that answer, since the worker then picks the tuning up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
vLLM's NixlPushConnector inverts the disaggregated KV transfer: decode registers its freshly allocated blocks with prefill, which WRITEs into them, instead of decode READing prefill's blocks. That means the decode leg has to be able to name the prefill engine before prefill has finished, so the coordinates cannot come from the prefill response the way pull mode's do -- they have to be discoverable. Add them to the disaggregated endpoint a prefill worker already publishes. The field is optional and independent of the bootstrap pair, so a frontend that predates it keeps routing the hop sequentially, which push mode tolerates: the prefill worker holds its finished blocks under the existing lease until the late registration arrives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Push mode only pays off if the decode leg is already holding allocated blocks when prefill finishes -- that is the overlap the connector exists to buy. Awaiting the prefill response first, as the pull path must, leaves the prefill worker's writer polling for a registration that cannot arrive until the frontend has finished routing decode. When the selected prefill worker advertised NIXL push coordinates, synthesize the decode leg's handoff from them and route decode immediately, consuming the prefill stream in the background. This reuses the dispatch shape the bootstrap path already established; the handoff names the prefill engine and nothing else, since which blocks move and where they land is negotiated worker-to-worker over NIXL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
The PD protocol registry rejects any connector name it does not know, so a push-mode engine failed at request setup before reaching the transfer it was configured for. Register it, and include it in the set that triggers side-channel host resolution -- push and pull share that channel, and an unresolved host leaves the engine unreachable. The params themselves are unchanged from pull: push reverses who moves the blocks, but that is negotiated worker-to-worker over NIXL rather than through kv_transfer_params. Subclassing rather than aliasing keeps the configured connector visible and gives the divergence a home if upstream ever splits the shapes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Completes the overlapped path: the router can only pre-dispatch decode if it knows which prefill engine to point it at, and in push mode that engine is never named in the prefill response the way pull mode's is. Publish engine_id and the side-channel address from prefill workers running the push connector, recomputing the address the way vLLM's own NIXL scheduler does -- the connector lives in the engine core process and cannot be read back from registration. Decline, loudly, in the cases where advertised coordinates would be wrong rather than merely absent: a worker fronting several data-parallel ranks has a side channel per rank and no single address to publish, and an engine with no resolved side-channel host has no address at all. Each falls back to the sequential handoff, which still completes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Records why push mode overlaps the prefill and decode legs, what it gives up to do so, and the one deployment shape where it silently falls back: vLLM gives each data-parallel rank its own NIXL side channel, and a worker fronting several ranks has no single address to publish. Spells out which DP configurations are affected, since the answer is "only internal load balancing with more than one rank" and every recommended layout already advertises normally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
| port = ( | ||
| vllm_envs.VLLM_NIXL_SIDE_CHANNEL_PORT | ||
| + vllm_config.parallel_config.data_parallel_index | ||
| ) |
There was a problem hiding this comment.
🔴 Advertised prefill address can point at the wrong port when data parallelism is used
The port published for the prefill worker is computed from a data-parallel index (vllm_config.parallel_config.data_parallel_index at components/src/dynamo/vllm/nixl_push.py:48-51) that vLLM only fills in inside its engine process, so the registering process reads 0 (or nothing) and advertises an address that does not belong to this worker.
Impact: With data-parallel deployments at a non-zero rank, the decode side is pointed at the wrong prefill address, so the KV handoff never completes and requests stall or fail; if the field is unset the worker can fail outright while registering.
Why the index is not available in the registering process, and what the correct value is
publish_nixl_push_endpoint runs in the Dynamo worker (main) process from register_vllm_model (components/src/dynamo/vllm/main.py:769), using the main-process vllm_config. This repo already documents that data_parallel_index is assigned by vLLM inside the engine-core process (vllm/v1/engine/core.py sets parallel_config.data_parallel_index = dp_rank, then resets data_parallel_rank = 0) — see the comment and fallback logic in components/src/dynamo/vllm/instrumented_scheduler.py:1504-1516, which also handles the value being None/absent. In the main process the rank (not the index) is the populated field, and it is already computed for this call site: dp_range[0] from get_dp_range_for_worker (components/src/dynamo/vllm/handlers.py:1054-1077) returns parallel_config.data_parallel_rank, but publish_nixl_push_endpoint only uses dp_range[1].
Consequences:
- external DP load balancing with rank > 0 (the configuration the warning text recommends): the engine's NIXL side channel listens at base + rank, while discovery advertises base + 0, so the decode worker registers its blocks against the wrong port;
- if
data_parallel_indexisNonein this process,base_port + NoneraisesTypeErrorduring model registration.
The unit test components/src/dynamo/vllm/tests/test_nixl_push.py:77-90 pins the current behavior with a hand-built SimpleNamespace, so it cannot catch the mismatch.
Prompt for agents
In components/src/dynamo/vllm/nixl_push.py, _side_channel_endpoint derives the advertised NIXL side-channel port as VLLM_NIXL_SIDE_CHANNEL_PORT + vllm_config.parallel_config.data_parallel_index. That attribute is populated by vLLM inside the engine-core process (see the comment in components/src/dynamo/vllm/instrumented_scheduler.py::InstrumentedScheduler._resolve_dp_rank, which also tolerates it being None or absent), but publish_nixl_push_endpoint runs in the Dynamo main process during register_vllm_model (components/src/dynamo/vllm/main.py). There, the populated field is parallel_config.data_parallel_rank, which the caller already has as dp_range[0] from get_dp_range_for_worker. As written, an external-DP prefill worker at rank N advertises the rank-0 port (or crashes if the index is None), so decode registers its blocks with the wrong engine. Decide which rank vLLM's NIXL scheduler actually uses for its port offset in the pinned vLLM version, then derive the offset from a value that is valid in the registering process (e.g. dp_range[0]) and guard against None/absent attributes. Also confirm whether vLLM multiplies the offset by tensor_parallel_size; if so the formula needs that factor too. Update tests/test_nixl_push.py accordingly.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def _side_channel_endpoint(vllm_config: VllmConfig) -> Optional[tuple[str, int]]: | ||
| """Mirror how vLLM's NIXL scheduler derives its own listening address. | ||
|
|
||
| See ``NixlBaseConnectorScheduler.__init__``: the host is the env var | ||
| verbatim, and the port is the configured base offset by the engine's | ||
| data-parallel index. Recomputing it here rather than reading it back off | ||
| the connector is deliberate -- the connector is constructed inside the | ||
| engine core process and is not reachable from registration. | ||
| """ | ||
| from vllm import envs as vllm_envs | ||
|
|
||
| host = vllm_envs.VLLM_NIXL_SIDE_CHANNEL_HOST | ||
| if not host: | ||
| return None | ||
| port = ( | ||
| vllm_envs.VLLM_NIXL_SIDE_CHANNEL_PORT | ||
| + vllm_config.parallel_config.data_parallel_index | ||
| ) | ||
| return host, port |
There was a problem hiding this comment.
🔍 Side-channel port formula may be missing vLLM's tensor-parallel factor
Some vLLM releases derive the NIXL side-channel port as VLLM_NIXL_SIDE_CHANNEL_PORT + dp_rank * tensor_parallel_size (each TP rank of a DP group occupies a contiguous port block), rather than a bare + dp_rank. The docstring here cites NixlBaseConnectorScheduler.__init__ in the pinned vLLM 0.26.0, but vLLM is not installed in this checkout so the derivation could not be verified. If the upstream formula includes the TP factor, the advertised port is wrong for any tp_size > 1 deployment with a non-zero data-parallel rank. Worth pinning against the actual pinned vLLM source (and ideally an import-based contract test, like the Mooncake helper signature test in components/src/dynamo/vllm/tests/test_vllm_kv_connector_protocols.py:504-521).
Was this helpful? React with 👍 or 👎 to provide feedback.
| endpoint = _side_channel_endpoint(vllm_config) | ||
| if endpoint is None: | ||
| logger.warning( | ||
| "VLLM_NIXL_SIDE_CHANNEL_HOST is unset, so this prefill worker has " | ||
| "no address to publish. Not advertising push coordinates; the " | ||
| "handoff will run sequentially." | ||
| ) | ||
| return False | ||
|
|
||
| host, port = endpoint |
There was a problem hiding this comment.
🔍 Published host is taken verbatim, including wildcard/loopback values
host is the env var verbatim. ensure_side_channel_host (components/src/dynamo/vllm/args.py:701-711) only sets a routable IP when the var is unset, so an operator who explicitly exports VLLM_NIXL_SIDE_CHANNEL_HOST=0.0.0.0 (a common bind-everything choice) causes the worker to publish 0.0.0.0 to discovery. Decode would then try to register at an unroutable address. Pull mode never advertised this value, so the failure mode is new; a guard that declines (or substitutes the detected IP) for wildcard/loopback addresses would keep the "decline rather than mislead" principle stated in the module docstring consistent.
Was this helpful? React with 👍 or 👎 to provide feedback.
| fn nixl_push_handoff(endpoint: &NixlPushEndpoint) -> PrefillResult { | ||
| PrefillResult { | ||
| disaggregated_params: serde_json::json!({ | ||
| "kv_transfer_params": { | ||
| "do_remote_decode": false, | ||
| "do_remote_prefill": true, | ||
| "remote_engine_id": endpoint.engine_id, | ||
| "remote_host": endpoint.host, | ||
| "remote_port": endpoint.port, | ||
| "tp_size": endpoint.tensor_parallel_size, | ||
| "pp_size": endpoint.pipeline_parallel_size, | ||
| } | ||
| }), | ||
| // Pull mode lifts this off the prefill response. Decode is dispatched | ||
| // before that response exists here, so cached-prompt-token reporting is | ||
| // unavailable in push mode -- the same trade the bootstrap path makes. | ||
| prompt_tokens_details: None, | ||
| } | ||
| } |
There was a problem hiding this comment.
🔍 No cross-tier check that decode also runs the push connector
The router synthesizes do_remote_prefill: true params purely from the prefill worker's advertisement; nothing verifies the selected decode worker is configured with NixlPushConnector. A deployment that upgrades only the prefill tier (prefill push, decode pull) would send decode a handoff its connector cannot service, and the failure appears as a hung/timed-out request rather than a configuration error. Since decode workers already register runtime metadata, advertising the connector mode on the decode card (or at least logging a mismatch) would turn a silent stall into a diagnosable misconfiguration.
Was this helpful? React with 👍 or 👎 to provide feedback.
| bootstrap_info, | ||
| worker_id: prepared.worker_id, | ||
| } | ||
| } else if let Some(result) = prepared.push_handoff { |
There was a problem hiding this comment.
The overlapped NIXL push path is taken for multimodal vLLM requests even though the synthetic PrefillResult has no prefill-produced embedding_params, so decode-only workers reject or mishandle requests that need the multimodal handoff. Fix: fall back to the sequential prefill completion path for requests with multi_modal_data or otherwise carry the required multimodal handoff before dispatching decode.
🤖 AI Fix
In lib/llm/src/kv_router/prefill_router/mod.rs, update PrefillRouter::generate so the prepared.push_handoff branch is skipped when the PreprocessedRequest being routed has multi_modal_data.is_some(), causing those requests to consume the prefill stream normally and preserve embedding_params; add a unit test covering a multimodal request with a nixl_push endpoint that does not select PrefillOutcome::NixlPush.
This comment has been minimized.
This comment has been minimized.
The decode engine reads this key unconditionally when it stages the receive -- NixlConnectorMetadata.add_new_req_to_recv indexes kv_transfer_params["remote_request_id"] rather than .get()-ing it -- so omitting it took down EngineCore with a KeyError mid-step instead of failing the request. Decode echoes the id back in its heartbeats to tell prefill the blocks are still wanted. Both legs already run under the frontend's context id, so that id names the prefill-side request directly. Thread it through to the handoff. The existing shape test asserted the payload this code produced rather than the one vLLM consumes, so it passed throughout; add a test that pins the keys vLLM indexes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
vLLM renames the NIXL agent per data-parallel rank while spawning the engine core, so the identity decode has to name is `<base>_dp<rank>` -- but only when `data_parallel_size > 1 or rank > 0`. A dense engine, including TP and TEP with a single rank, keeps the unsuffixed base ID. The advertisement published the base ID unconditionally. That is correct only for the single-rank case; every external- or hybrid-load-balanced DP prefill worker advertised an agent that does not exist -- exactly the configurations push mode documents as supported. The failure is silent: the peer rejects the handshake with "Remote NIXL agent engine ID mismatch" and the transfer falls back to the sequential path instead of erroring, so the overlap is lost with no obvious cause. Mirror vLLM's own derivation rather than approximating it. Appending `_dp0` to a dense engine is wrong in the same way as omitting the suffix for a real rank, and was observed rejecting every handshake on a tensor/expert-parallel deployment. Also decline to advertise when the DP range Dynamo assigned this worker starts somewhere other than the rank vLLM reports, since one of the two is then wrong and the published identity cannot be trusted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
6f4c03f to
b34d6b7
Compare
Stacked on #13197 — base is
gluo/prefill-card-router-mode, so this diff is the five commits on top. Retarget tomainonce #13197 lands.Summary
Adds support for vLLM's push-mode NIXL KV connector (
NixlPushConnector, available in the pinned vLLM 0.26.0) for disaggregated serving.The default
NixlConnectoris pull-based: prefill runs, returns its block locations, and decode then READs them.NixlPushConnectorinverts this — decode registers its freshly allocated blocks with prefill, which WRITEs into them. The point of the inversion is that decode no longer needs anything computed during prefill, only the prefill engine's identity, so the two legs can be dispatched concurrently and the transfer starts the moment prefill finishes.Dynamo rejected the connector outright before this change (
Unsupported kv_connector=...at request setup). Registering it alone would have produced a working but slower-than-pull configuration: with a sequential handoff, push adds a notification round-trip and two engine-step boundaries onto the critical path that a one-sided READ does not have. So this change does both halves — recognize the connector, and give the router what it needs to overlap the legs.Advertisement — a prefill worker in push mode publishes its NIXL engine ID and side-channel address on the
disaggregated_endpointit already registers. The field is optional and independent of the existing bootstrap pair, so an older frontend keeps routing sequentially, which push mode tolerates.Routing — when the selected prefill worker advertised those coordinates,
PrefillRoutersynthesizes decode'skv_transfer_paramsfrom them and routes decode immediately, consuming the prefill stream in the background. This reuses the dispatch shape the SGLang bootstrap path already established.Backend —
NixlPushConnectoris registered as a PD-capable connector and included in the set that triggers side-channel host resolution. Thekv_transfer_paramsare unchanged from pull mode on both legs: the reversal is negotiated worker-to-worker over NIXL notifications, not through the params the handler sets.Enable with the same connector on both tiers:
--kv-transfer-config '{"kv_connector":"NixlPushConnector","kv_role":"kv_both"}'Validation
End-to-end on one GPU (RTX 5880 Ada, 49 GiB), Qwen3-0.6B, prefill + decode co-resident, no parallelism. Ran
examples/backends/vllm/launch/disagg_same_gpu.shunmodified as the pull-mode baseline, then the same script withNixlPushConnectorsubstituted into both--kv-transfer-configflags, against vLLM 0.26.0.Four greedy prompts (
temperature=0,max_tokens=64) produced byte-identical output between the two modes — same md5 over text and token counts. Correct output is the real check that the KV blocks arrived intact via WRITE rather than READ.The overlap was confirmed from the dispatch timeline rather than assumed, since a silent fall back to the sequential handoff would also produce correct output:
The prefill worker logged
Publishing NIXL push endpoint to discovery: engine_id=66113432-... 10.110.40.102:20097 (tp=1, pp=1), with the port matching its configuredVLLM_NIXL_SIDE_CHANNEL_PORT. 12 concurrent requests all returned 200 with no engine errors, watchdog expiries, or lease reaps.Static checks and unit tests:
New coverage: handoff wire shape plus the keys vLLM indexes unconditionally, and the fields it must omit (
lib/llm/src/kv_router/prefill_router/mod.rs);disaggregated_endpointserde round-trip with and without the new field (lib/llm/src/local_model/runtime_config.rs); push-connector registration,PdConnectornesting, and the side-channel-name set staying in sync with the registry; advertisement gating and address derivation (test_nixl_push.py).Note on the sixth commit. The first e2e run killed the decode EngineCore with
KeyError: 'remote_request_id'.NixlConnectorMetadata.add_new_req_to_recvindexes that key directly rather than.get()-ing it, and the synthesized handoff omitted it. The unit tests had asserted the payload the code produced rather than the one vLLM consumes, so they passed throughout — the added test now pins the consumed keys. Everything above was re-run after the fix.Known limitations
prompt_tokens_detailsoff the prefill response; decode is dispatched before that response exists. Same trade the bootstrap path makes.--data-parallel-size> 1; external and hybrid-with-one-local-rank both advertise normally. Documented in the vLLM reference guide. Liftable as a follow-up:data_parallel_index == data_parallel_rankupstream, so the router could offset the base port by the rank it selected — it needs a multi-rank deployment to validate.🤖 Generated with Claude Code