Skip to content

feat(vllm): support NixlPushConnector with overlapped prefill/decode dispatch - #13311

Closed
GuanLuo wants to merge 14 commits into
mainfrom
gluo/vllm-nixl-push-connector
Closed

GuanLuo wants to merge 14 commits into
mainfrom
gluo/vllm-nixl-push-connector

Conversation

@GuanLuo

@GuanLuo GuanLuo commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #13197 — base is gluo/prefill-card-router-mode, so this diff is the five commits on top. Retarget to main once #13197 lands.

Summary

Adds support for vLLM's push-mode NIXL KV connector (NixlPushConnector, available in the pinned vLLM 0.26.0) for disaggregated serving.

The default NixlConnector is pull-based: prefill runs, returns its block locations, and decode then READs them. NixlPushConnector inverts this — decode registers its freshly allocated blocks with prefill, which WRITEs into them. The point of the inversion is that decode no longer needs anything computed during prefill, only the prefill engine's identity, so the two legs can be dispatched concurrently and the transfer starts the moment prefill finishes.

Dynamo rejected the connector outright before this change (Unsupported kv_connector=... at request setup). Registering it alone would have produced a working but slower-than-pull configuration: with a sequential handoff, push adds a notification round-trip and two engine-step boundaries onto the critical path that a one-sided READ does not have. So this change does both halves — recognize the connector, and give the router what it needs to overlap the legs.

Advertisement — a prefill worker in push mode publishes its NIXL engine ID and side-channel address on the disaggregated_endpoint it already registers. The field is optional and independent of the existing bootstrap pair, so an older frontend keeps routing sequentially, which push mode tolerates.

Routing — when the selected prefill worker advertised those coordinates, PrefillRouter synthesizes decode's kv_transfer_params from them and routes decode immediately, consuming the prefill stream in the background. This reuses the dispatch shape the SGLang bootstrap path already established.

Backend — NixlPushConnector is registered as a PD-capable connector and included in the set that triggers side-channel host resolution. The kv_transfer_params are unchanged from pull mode on both legs: the reversal is negotiated worker-to-worker over NIXL notifications, not through the params the handler sets.

Enable with the same connector on both tiers:

--kv-transfer-config '{"kv_connector":"NixlPushConnector","kv_role":"kv_both"}'

Validation

End-to-end on one GPU (RTX 5880 Ada, 49 GiB), Qwen3-0.6B, prefill + decode co-resident, no parallelism. Ran examples/backends/vllm/launch/disagg_same_gpu.sh unmodified as the pull-mode baseline, then the same script with NixlPushConnector substituted into both --kv-transfer-config flags, against vLLM 0.26.0.

Four greedy prompts (temperature=0, max_tokens=64) produced byte-identical output between the two modes — same md5 over text and token counts. Correct output is the real check that the KV blocks arrived intact via WRITE rather than READ.

The overlap was confirmed from the dispatch timeline rather than assumed, since a silent fall back to the sequential handoff would also produce correct output:

push:  45.502596  prefill  request received
       45.504747  backend  request received   <-- decode dispatched 2.2ms later
       45.555116  prefill  request completed  <-- decode already running 50ms
pull:  09.368171  prefill  request received
       09.429478  prefill  request completed
       09.432664  backend  request received   <-- decode waits for prefill

The prefill worker logged Publishing NIXL push endpoint to discovery: engine_id=66113432-... 10.110.40.102:20097 (tp=1, pp=1), with the port matching its configured VLLM_NIXL_SIDE_CHANNEL_PORT. 12 concurrent requests all returned 200 with no engine errors, watchdog expiries, or lease reaps.

Static checks and unit tests:

cargo check --workspace --all-targets                      # clean
cargo clippy --workspace --no-deps --all-targets           # clean
cargo test -p dynamo-llm -p dynamo-backend-common --lib    # 2092 + 137 passed, 0 failed
pytest test_nixl_push.py test_vllm_kv_connector_protocols.py   # 48 passed
pytest test_vllm_unit.py                                   # 139 passed
pre-commit run --files <changed>                           # clean

New coverage: handoff wire shape plus the keys vLLM indexes unconditionally, and the fields it must omit (lib/llm/src/kv_router/prefill_router/mod.rs); disaggregated_endpoint serde round-trip with and without the new field (lib/llm/src/local_model/runtime_config.rs); push-connector registration, PdConnector nesting, and the side-channel-name set staying in sync with the registry; advertisement gating and address derivation (test_nixl_push.py).

Note on the sixth commit. The first e2e run killed the decode EngineCore with KeyError: 'remote_request_id'. NixlConnectorMetadata.add_new_req_to_recv indexes that key directly rather than .get()-ing it, and the synthesized handoff omitted it. The unit tests had asserted the payload the code produced rather than the one vLLM consumes, so they passed throughout — the added test now pins the consumed keys. Everything above was re-run after the fix.

Known limitations

  • No cached-prompt-token details in usage. Pull mode lifts prompt_tokens_details off the prefill response; decode is dispatched before that response exists. Same trade the bootstrap path makes.
  • Terminal prefill is not short-circuited. A request reaching a stop condition during the one-token prefill step still proceeds to decode. Also inherited from the bootstrap path.
  • Prefill failure surfaces late. An error after dispatch reaches the client as decode's registration-watchdog timeout rather than an immediate failure — inherent to not awaiting prefill.
  • One data-parallel rank per worker. vLLM gives each DP rank its own side channel; a worker fronting several has no single address to publish, so it logs a warning and runs the sequential handoff. Affects only vLLM's internal DP load balancing with --data-parallel-size > 1; external and hybrid-with-one-local-rank both advertise normally. Documented in the vLLM reference guide. Liftable as a follow-up: data_parallel_index == data_parallel_rank upstream, so the router could offset the base port by the rank it selected — it needs a multi-rank deployment to validate.
  • The Rust vLLM sidecar is untouched. It targets vLLM gRPC 0.25.1, which predates this connector. It never advertises, so it never takes the concurrent path.

🤖 Generated with Claude Code


Open in Devin Review

GuanLuo and others added 12 commits August 14, 2026 15:27
…card

The frontend held a single `router_mode` and used it for both routers in a
disaggregated pipeline: the decode router built in the watcher and the prefill
router built underneath it. `--router-mode kv` meant KV on both hops and
`--router-mode round-robin` meant round-robin on both, with no way to ask for
one on prefill and the other on decode.

Per-worker-set `RouterConfig` already rides in the model deployment card, and
`register_model` already takes both `worker_type` and `router_config`, so a
prefill worker can already declare how it wants to be routed to. Nothing read
it: the watcher short-circuits prefill cards before applying `router_config`,
and the prefill router inherited the decode set's mode instead.

Resolve the prefill hop's mode from the prefill worker set's own cards at
activation, falling back to the decode set's mode when the card declares
nothing. `build_binding` already fetched those cards to read `enable_eagle`,
so this reuses that read rather than adding one. A KV-routed prefill tier in
front of a round-robin decode tier now works, as does the reverse.

Three things this required getting right:

- `router_mode` on `PrefillRouter` meant two things. Conditional disagg peeks
  the decode router, so it needs the decode set's mode; the Direct-routing
  guard is about the prefill dispatch, so it needs the prefill hop's. Split
  them: the field is now `decode_router_mode`, and the resolved prefill mode
  lives on `PrefillBinding` because it is only knowable once a prefill target
  is discovered. The Direct guard moved after the binding load accordingly.

- A card that declares a mode may declare KV tuning with it. Honoring only
  half of a `RouterConfig` would be a trap, so an advertised
  `kv_router_config` is used too, with `router_track_active_blocks` still
  forced off as the decode set does.

- An unreadable card now fails activation instead of silently inheriting.
  `drive_target` already retries with backoff, and a delayed activation costs
  far less than routing a whole deployment with the wrong strategy because of
  one transient discovery miss -- a downgrade that would persist until the
  binding was rebuilt.

A prefill fleet mid-rolling-update can disagree about its mode; the first card
wins, and the split is logged rather than passing silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Asserts the two hops route on their own terms rather than sharing one mode.
Prefill mockers advertise RouterConfig(RouterMode.KV); decode mockers advertise
nothing and inherit the frontend's --router-mode round-robin.

Prefix-extending requests must converge on one prefill worker, which only KV
routing produces, while spreading across more than one decode worker, which
only round-robin produces. The second assertion is the one that fails if the
per-role override is ignored and the pipeline silently runs in a single mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Self-review follow-up to the prefill card resolution.

`build_binding` now reads `enable_eagle` from the prefill card it already
fetches, so the value threaded through `PrefillRouter::new` was dead. Every
caller was computing the same lookup and handing it in: the frontend hardcoded
`false` (already overridden by the card read), while ext-proc and the C
bindings passed `card.runtime_config.enable_eagle` -- the same field, from a
card. Both keep their copy for the decode chooser, which genuinely needs it.

Removing the parameter is what makes that visible; keeping it would have left a
public argument that is silently ignored.

Also from the review:
- Log cards that fail to deserialize. An unreadable card used to cost only an
  EAGLE hint; it now drops a worker out of the vote that decides how the hop is
  routed, so it should not vanish silently.
- Drop `PrefillAdvertisement::advertised`; it was exactly
  `kv_router_config.is_some()` and existed only to feed one tracing field.
- Stop shadowing the `kv_router_config` parameter in `build_binding`.
- Remove the doubled error context around card resolution.
- Trim comments that restated the code, and the test that asserted the override
  works in the reverse direction -- same assertion with the modes swapped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
…d too

The prefill hop took its mode and KV tuning from the prefill card but its block
size and session-affinity TTL from the decode set, so a prefill tier could not
fully describe how it wants to be routed to. Resolve all four from the same
place.

Block size is the one that can be wrong rather than merely inflexible: the
prefill pool's KV events are keyed on its own block size, so using the decode
set's would index that pool at the wrong granularity. Disaggregation requires
the two to match for KV transfer, which is why this has not bitten anyone, but
the frontend should not be the reason it holds. A card reporting 0 never
declared one, so the decode set's value still applies there.

Session TTL rides in `router_config`, so it follows the same rule as the KV
tuning: an advertisement replaces the decode set's configuration rather than
merging with it. A card that advertises without a TTL therefore means "no
affinity on this hop", not "inherit the frontend's" -- hence `Option<Option<_>>`
to keep "advertised nothing" distinct from "advertised no TTL". Block size is
not part of `router_config` and is intrinsic to the worker, so it is taken
unconditionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Asserts the prefill and decode hops take their session-affinity setting from
their own cards rather than sharing one. Both tiers advertise round-robin and
only prefill advertises a TTL, so affinity is the sole difference: requests
carrying one session id stay on a single prefill worker while rotating across
decode workers.

Observed: prefill `[458, 458, 458, 458]`, decode `[501, 498, 501, 498]`. Giving
decode a TTL as well collapses it to `[861, 861, 861, 861]` and fails the decode
assertion, so the test is sensitive to the setting rather than to round-robin
timing.

This covers configuration, not expiry. Proving a TTL elapses means sleeping past
it and asserting a rebind, which is timing-dependent and can legitimately
re-select the same worker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
The helper landed in a584b62 without the test that calls it: reverting a
throwaway negative-control edit with `git checkout --` also discarded the
wrapper, because that file had not been committed yet. The helper was dead code
and the scenario never ran.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
An advertised card replaces the decode set's configuration rather than merging
with it, so a worker that names a mode without restating tuning silently gets
defaults. Nothing reported which values a hop ended up with, leaving "why did my
temperature reset" answerable only by reading code.

Log the resolved values -- mode, block size, session TTL, KV tuning -- once per
activation, after resolution rather than before, so the line reflects what the
hop actually uses rather than what the card asked for.

The docs said "restate your tuning" without saying why. State the rule directly:
on a worker an unset flag means the default, not "inherit the frontend's". Also
note that both sides build from the same argument groups, so defaults are
identical and values diverge only when the frontend is tuned and the worker is
not -- and that a shared environment changes that answer, since the worker then
picks the tuning up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
vLLM's NixlPushConnector inverts the disaggregated KV transfer: decode
registers its freshly allocated blocks with prefill, which WRITEs into
them, instead of decode READing prefill's blocks. That means the decode
leg has to be able to name the prefill engine before prefill has
finished, so the coordinates cannot come from the prefill response the
way pull mode's do -- they have to be discoverable.

Add them to the disaggregated endpoint a prefill worker already
publishes. The field is optional and independent of the bootstrap pair,
so a frontend that predates it keeps routing the hop sequentially, which
push mode tolerates: the prefill worker holds its finished blocks under
the existing lease until the late registration arrives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Push mode only pays off if the decode leg is already holding allocated
blocks when prefill finishes -- that is the overlap the connector exists
to buy. Awaiting the prefill response first, as the pull path must,
leaves the prefill worker's writer polling for a registration that
cannot arrive until the frontend has finished routing decode.

When the selected prefill worker advertised NIXL push coordinates,
synthesize the decode leg's handoff from them and route decode
immediately, consuming the prefill stream in the background. This reuses
the dispatch shape the bootstrap path already established; the handoff
names the prefill engine and nothing else, since which blocks move and
where they land is negotiated worker-to-worker over NIXL.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
The PD protocol registry rejects any connector name it does not know, so
a push-mode engine failed at request setup before reaching the transfer
it was configured for. Register it, and include it in the set that
triggers side-channel host resolution -- push and pull share that
channel, and an unresolved host leaves the engine unreachable.

The params themselves are unchanged from pull: push reverses who moves
the blocks, but that is negotiated worker-to-worker over NIXL rather
than through kv_transfer_params. Subclassing rather than aliasing keeps
the configured connector visible and gives the divergence a home if
upstream ever splits the shapes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Completes the overlapped path: the router can only pre-dispatch decode if
it knows which prefill engine to point it at, and in push mode that engine
is never named in the prefill response the way pull mode's is.

Publish engine_id and the side-channel address from prefill workers
running the push connector, recomputing the address the way vLLM's own
NIXL scheduler does -- the connector lives in the engine core process and
cannot be read back from registration.

Decline, loudly, in the cases where advertised coordinates would be
wrong rather than merely absent: a worker fronting several data-parallel
ranks has a side channel per rank and no single address to publish, and
an engine with no resolved side-channel host has no address at all. Each
falls back to the sequential handoff, which still completes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Records why push mode overlaps the prefill and decode legs, what it
gives up to do so, and the one deployment shape where it silently falls
back: vLLM gives each data-parallel rank its own NIXL side channel, and
a worker fronting several ranks has no single address to publish.

Spells out which DP configurations are affected, since the answer is
"only internal load balancing with more than one rank" and every
recommended layout already advertises normally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
@GuanLuo
GuanLuo requested review from a team as code owners August 15, 2026 19:06
@github-actions github-actions Bot added feat documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend router Relates to routing, KV-aware routing, etc. labels Aug 15, 2026
@github-actions

github-actions Bot commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 4 potential issues.

Open in Devin Review

Comment on lines +48 to +51
port = (
vllm_envs.VLLM_NIXL_SIDE_CHANNEL_PORT
+ vllm_config.parallel_config.data_parallel_index
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Advertised prefill address can point at the wrong port when data parallelism is used

The port published for the prefill worker is computed from a data-parallel index (vllm_config.parallel_config.data_parallel_index at components/src/dynamo/vllm/nixl_push.py:48-51) that vLLM only fills in inside its engine process, so the registering process reads 0 (or nothing) and advertises an address that does not belong to this worker.
Impact: With data-parallel deployments at a non-zero rank, the decode side is pointed at the wrong prefill address, so the KV handoff never completes and requests stall or fail; if the field is unset the worker can fail outright while registering.

Why the index is not available in the registering process, and what the correct value is

publish_nixl_push_endpoint runs in the Dynamo worker (main) process from register_vllm_model (components/src/dynamo/vllm/main.py:769), using the main-process vllm_config. This repo already documents that data_parallel_index is assigned by vLLM inside the engine-core process (vllm/v1/engine/core.py sets parallel_config.data_parallel_index = dp_rank, then resets data_parallel_rank = 0) — see the comment and fallback logic in components/src/dynamo/vllm/instrumented_scheduler.py:1504-1516, which also handles the value being None/absent. In the main process the rank (not the index) is the populated field, and it is already computed for this call site: dp_range[0] from get_dp_range_for_worker (components/src/dynamo/vllm/handlers.py:1054-1077) returns parallel_config.data_parallel_rank, but publish_nixl_push_endpoint only uses dp_range[1].

Consequences:

  • external DP load balancing with rank > 0 (the configuration the warning text recommends): the engine's NIXL side channel listens at base + rank, while discovery advertises base + 0, so the decode worker registers its blocks against the wrong port;
  • if data_parallel_index is None in this process, base_port + None raises TypeError during model registration.

The unit test components/src/dynamo/vllm/tests/test_nixl_push.py:77-90 pins the current behavior with a hand-built SimpleNamespace, so it cannot catch the mismatch.

Prompt for agents
In components/src/dynamo/vllm/nixl_push.py, _side_channel_endpoint derives the advertised NIXL side-channel port as VLLM_NIXL_SIDE_CHANNEL_PORT + vllm_config.parallel_config.data_parallel_index. That attribute is populated by vLLM inside the engine-core process (see the comment in components/src/dynamo/vllm/instrumented_scheduler.py::InstrumentedScheduler._resolve_dp_rank, which also tolerates it being None or absent), but publish_nixl_push_endpoint runs in the Dynamo main process during register_vllm_model (components/src/dynamo/vllm/main.py). There, the populated field is parallel_config.data_parallel_rank, which the caller already has as dp_range[0] from get_dp_range_for_worker. As written, an external-DP prefill worker at rank N advertises the rank-0 port (or crashes if the index is None), so decode registers its blocks with the wrong engine. Decide which rank vLLM's NIXL scheduler actually uses for its port offset in the pinned vLLM version, then derive the offset from a value that is valid in the registering process (e.g. dp_range[0]) and guard against None/absent attributes. Also confirm whether vLLM multiplies the offset by tensor_parallel_size; if so the formula needs that factor too. Update tests/test_nixl_push.py accordingly.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +34 to +52
def _side_channel_endpoint(vllm_config: VllmConfig) -> Optional[tuple[str, int]]:
"""Mirror how vLLM's NIXL scheduler derives its own listening address.

See ``NixlBaseConnectorScheduler.__init__``: the host is the env var
verbatim, and the port is the configured base offset by the engine's
data-parallel index. Recomputing it here rather than reading it back off
the connector is deliberate -- the connector is constructed inside the
engine core process and is not reachable from registration.
"""
from vllm import envs as vllm_envs

host = vllm_envs.VLLM_NIXL_SIDE_CHANNEL_HOST
if not host:
return None
port = (
vllm_envs.VLLM_NIXL_SIDE_CHANNEL_PORT
+ vllm_config.parallel_config.data_parallel_index
)
return host, port

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Side-channel port formula may be missing vLLM's tensor-parallel factor

Some vLLM releases derive the NIXL side-channel port as VLLM_NIXL_SIDE_CHANNEL_PORT + dp_rank * tensor_parallel_size (each TP rank of a DP group occupies a contiguous port block), rather than a bare + dp_rank. The docstring here cites NixlBaseConnectorScheduler.__init__ in the pinned vLLM 0.26.0, but vLLM is not installed in this checkout so the derivation could not be verified. If the upstream formula includes the TP factor, the advertised port is wrong for any tp_size > 1 deployment with a non-zero data-parallel rank. Worth pinning against the actual pinned vLLM source (and ideally an import-based contract test, like the Mooncake helper signature test in components/src/dynamo/vllm/tests/test_vllm_kv_connector_protocols.py:504-521).

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +98 to +107
endpoint = _side_channel_endpoint(vllm_config)
if endpoint is None:
logger.warning(
"VLLM_NIXL_SIDE_CHANNEL_HOST is unset, so this prefill worker has "
"no address to publish. Not advertising push coordinates; the "
"handoff will run sequentially."
)
return False

host, port = endpoint

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Published host is taken verbatim, including wildcard/loopback values

host is the env var verbatim. ensure_side_channel_host (components/src/dynamo/vllm/args.py:701-711) only sets a routable IP when the var is unset, so an operator who explicitly exports VLLM_NIXL_SIDE_CHANNEL_HOST=0.0.0.0 (a common bind-everything choice) causes the worker to publish 0.0.0.0 to discovery. Decode would then try to register at an unroutable address. Pull mode never advertised this value, so the failure mode is new; a guard that declines (or substitutes the detected IP) for wildcard/loopback addresses would keep the "decline rather than mislead" principle stated in the module docstring consistent.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +141 to +159
fn nixl_push_handoff(endpoint: &NixlPushEndpoint) -> PrefillResult {
PrefillResult {
disaggregated_params: serde_json::json!({
"kv_transfer_params": {
"do_remote_decode": false,
"do_remote_prefill": true,
"remote_engine_id": endpoint.engine_id,
"remote_host": endpoint.host,
"remote_port": endpoint.port,
"tp_size": endpoint.tensor_parallel_size,
"pp_size": endpoint.pipeline_parallel_size,
}
}),
// Pull mode lifts this off the prefill response. Decode is dispatched
// before that response exists here, so cached-prompt-token reporting is
// unavailable in push mode -- the same trade the bootstrap path makes.
prompt_tokens_details: None,
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 No cross-tier check that decode also runs the push connector

The router synthesizes do_remote_prefill: true params purely from the prefill worker's advertisement; nothing verifies the selected decode worker is configured with NixlPushConnector. A deployment that upgrades only the prefill tier (prefill push, decode pull) would send decode a handoff its connector cannot service, and the failure appears as a hung/timed-out request rather than a configuration error. Since decode workers already register runtime metadata, advertising the connector mode on the decode card (or at least logging a mismatch) would turn a silent stall into a diagnosable misconfiguration.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

bootstrap_info,
worker_id: prepared.worker_id,
}
} else if let Some(result) = prepared.push_handoff {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overlapped NIXL push path is taken for multimodal vLLM requests even though the synthetic PrefillResult has no prefill-produced embedding_params, so decode-only workers reject or mishandle requests that need the multimodal handoff. Fix: fall back to the sequential prefill completion path for requests with multi_modal_data or otherwise carry the required multimodal handoff before dispatching decode.

🤖 AI Fix

In lib/llm/src/kv_router/prefill_router/mod.rs, update PrefillRouter::generate so the prepared.push_handoff branch is skipped when the PreprocessedRequest being routed has multi_modal_data.is_some(), causing those requests to consume the prefill stream normally and preserve embedding_params; add a unit test covering a multimodal request with a nixl_push endpoint that does not select PrefillOutcome::NixlPush.

@datadog-official

This comment has been minimized.

GuanLuo and others added 2 commits August 15, 2026 21:34
The decode engine reads this key unconditionally when it stages the
receive -- NixlConnectorMetadata.add_new_req_to_recv indexes
kv_transfer_params["remote_request_id"] rather than .get()-ing it -- so
omitting it took down EngineCore with a KeyError mid-step instead of
failing the request. Decode echoes the id back in its heartbeats to tell
prefill the blocks are still wanted.

Both legs already run under the frontend's context id, so that id names
the prefill-side request directly. Thread it through to the handoff.

The existing shape test asserted the payload this code produced rather
than the one vLLM consumes, so it passed throughout; add a test that
pins the keys vLLM indexes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
vLLM renames the NIXL agent per data-parallel rank while spawning the engine
core, so the identity decode has to name is `<base>_dp<rank>` -- but only when
`data_parallel_size > 1 or rank > 0`. A dense engine, including TP and TEP with
a single rank, keeps the unsuffixed base ID.

The advertisement published the base ID unconditionally. That is correct only
for the single-rank case; every external- or hybrid-load-balanced DP prefill
worker advertised an agent that does not exist -- exactly the configurations
push mode documents as supported. The failure is silent: the peer rejects the
handshake with "Remote NIXL agent engine ID mismatch" and the transfer falls
back to the sequential path instead of erroring, so the overlap is lost with no
obvious cause.

Mirror vLLM's own derivation rather than approximating it. Appending `_dp0` to
a dense engine is wrong in the same way as omitting the suffix for a real rank,
and was observed rejecting every handshake on a tensor/expert-parallel
deployment.

Also decline to advertise when the DP range Dynamo assigned this worker starts
somewhere other than the rank vLLM reports, since one of the two is then wrong
and the published identity cannot be trusted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
@GuanLuo
GuanLuo force-pushed the gluo/prefill-card-router-mode branch from 6f4c03f to b34d6b7 Compare August 20, 2026 16:35
@GuanLuo
GuanLuo requested a review from a team as a code owner August 20, 2026 16:35
Base automatically changed from gluo/prefill-card-router-mode to main August 22, 2026 00:17
@GuanLuo GuanLuo closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation feat router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant