Skip to content

feat: [AMD Support via SemiAnalysis][2/4] vLLM MoRI-IO integration - #11

Closed
cquil11 wants to merge 1 commit into
agent/amd-multinode-runtimefrom
agent/vllm-mori-integration
Closed

cquil11 wants to merge 1 commit into
agent/amd-multinode-runtimefrom
agent/vllm-mori-integration

Conversation

@cquil11

@cquil11 cquil11 commented Sep 8, 2026 •

Copy link
Copy Markdown

Summary

Integrate native vLLM Router with MoRI-IO prefill/decode workers. Static router registration and NIXL port handling do not describe MoRI's ZMQ discovery path; this PR supplies the connector-specific launch, port, and readiness contract for it on the seams main now has: a connector table row on the vLLM backend, discovery hooks on the vLLM Router frontend, and per-process listeners from the port allocator.

Stacked on #431, which combines AMD hardware enablement and required cluster runtime. #433 follows this PR; ATOM uses Mooncake and has no engine dependency on MoRI.

Review map

  • backends/vllm.py: moriio is a KVConnector row with discovery=True and a mode-dependent role (the row type, the connector_for_mode resolver, and the discovery seam landed on main in #491, so no connector name is compared in a branch here). kv_transfer_config(mode, process, runtime) builds the worker JSON; the topology arguments are optional and only a discovery row reads them. For a discovery connector the extra config carries proxy_ip (head node), proxy_ping_port, http_port, host_ip (the worker's own routable IP), handshake_port, notify_port, read_mode: the keys moriio_common.py reads (vllm-project/vllm 9679173788). Upstream's fixed defaults of 6301 and 61005 collide as soon as two workers share a node, so the allocator reserves separate handshake and notification blocks of one port per local rank (vLLM adds the DP and TP rank to each base), using default bases 26000 and 27000 below this cluster’s ephemeral port range; a worker built without them is refused. Such a worker gets no VLLM_PORT scan range because MoRI-IO opens listeners inside TP children that inherit it.
  • frontends/vllm_router.py, ports.py: in discovery mode the vLLM Router adds --kv-connector <name> --vllm-discovery-address 0.0.0.0:36367 (RouterArgs in vllm-project/router) and validates the discovery rules (both roles on the connector, a P/D topology, one router on the head node). Readiness is the Router's own /health, which answers 503 until a prefill and a decode have registered (vllm-project/router 43140bc8e2, VllmPDRouter::health); the probe reads the recipe #491 hands it, so core/health.py has no per-frontend branch and no completions POST. The probe_ready docstring records why /health and not /workers, and what the upstream fix would be.
  • core/schema.py: refuses a discovery connector with any frontend other than vllm-router; the Router's own rules live in its validate.
  • core/topology.py: Process.moriio_handshake_port and moriio_notify_port.
  • Docs: docs/vllm-router.md gains a MoRI-IO discovery section with the full contract table; config-reference.md points at it; the port tables in CLAUDE.md and docs/architecture.md list the two kinds. examples/vllm/vllm-router-moriio-disagg.yaml (colocated P/D) is in the examples matrix and in every example-driven test.
  • Tests: worker JSON against the realized topology, refusal without listeners, colocated listener allocation, the role override, the discovery command, the bootstrap port, the /health probe, and the recipe rules.

Review the stack

The upstream PRs target NVIDIA main and are manually stacked, so later upstream diffs include preceding layers. The downstream fork PRs form an actual branch-based stack (main → #1 → #11 → #7 → #14); each fork PR's Files changed tab isolates its layer for easier review.

Merge in that order, then rebase the remaining layers. ATOM uses Mooncake, not MoRI; its stack position is review ordering, not an engine dependency. The core runtime does not require the observation layer, but the InferenceX launcher uses its srtctl wait command.

E2E validation is maintained only in the base PR: upstream #431 / fork #1.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/srtctl/backends/vllm.py Resolved
@cquil11
cquil11 force-pushed the agent/vllm-mori-integration branch from 094e542 to 05c9395 Compare September 13, 2026 21:46
@cquil11 cquil11 changed the title feat(vllm): add native MoRI-IO router integration [AMD Support][2/3] vLLM MoRI-IO integration Sep 13, 2026
@cquil11
cquil11 force-pushed the agent/vllm-mori-integration branch from 05c9395 to 26bbe1c Compare September 15, 2026 14:07
@cquil11
cquil11 removed this pull request from stack #12 September 15, 2026 14:09
@cquil11
cquil11 changed the base branch from agent/amd-multinode-runtime to review/runtime-portability September 15, 2026 14:09
@cquil11
cquil11 added this pull request to stack #17 September 15, 2026 14:09
@cquil11 cquil11 changed the title [AMD Support][2/3] vLLM MoRI-IO integration [AMD Support][3/4] vLLM MoRI-IO integration Sep 15, 2026
@cquil11
cquil11 force-pushed the agent/vllm-mori-integration branch from 26bbe1c to fdfc9ed Compare September 16, 2026 01:11

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/srtctl/backends/vllm.py Resolved
@cquil11
cquil11 removed this pull request from stack #17 September 16, 2026 05:23
@cquil11
cquil11 changed the base branch from review/runtime-portability to agent/amd-multinode-runtime September 16, 2026 05:23
@cquil11
cquil11 force-pushed the agent/vllm-mori-integration branch from fdfc9ed to 65893f4 Compare September 16, 2026 05:23
@cquil11
cquil11 added this pull request to stack #18 September 16, 2026 05:23
@cquil11 cquil11 changed the title [AMD Support][3/4] vLLM MoRI-IO integration [AMD Support][2/4] vLLM MoRI-IO integration Sep 16, 2026
@cquil11
cquil11 force-pushed the agent/vllm-mori-integration branch 2 times, most recently from 7368036 to 9935d90 Compare September 18, 2026 21:36
@cquil11 cquil11 changed the title [AMD Support][2/4] vLLM MoRI-IO integration feat: [AMD Support via SemiAnalysis][2/4] vLLM MoRI-IO integration Sep 18, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9935d90. Configure here.

Comment thread src/srtctl/backends/vllm.py Outdated
"read_mode": True,
},
}
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MoRI workers share default control ports

High Severity

Managed MoRIIOConnector JSON omits per-worker handshake_port and notify_port, so every engine falls back to the same defaults. Co-located workers, including multiple endpoints on one node, then collide on those listeners and on engine_id.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 9935d90. Configure here.

@cquil11

cquil11 commented Sep 22, 2026

Copy link
Copy Markdown
Author

Hardware validation of the MoRI listener-allocation fix: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35756972563/job/106845265742

MI355X, vLLM v0.28.0 + MoRI-IO, MiniMax-M3-MXFP4, two-node TP4 prefill / TP4 decode. All seven measured points at concurrency 1, 2, 4, 8, 16, 32, and 64 passed: 1,276 requests completed, zero failed. Both roles became healthy and all four decode handshake/notify listeners bound successfully. Runtime: 049a6b5804fb3f9bfbc959a7f7379935fe8952a7, containing fix 6a780cce61cb24fc710810ff35e1ec365fa68bcf.

Stopped at the owner’s request while concurrency 128 was warming up; concurrency 256 was not tested. The workflow is intentionally cancelled, not a fully green sweep. Slurm allocation 45492 was cancelled and both nodes released. No model, serving-option, or workload changes were made for this fix.

@cquil11

cquil11 commented Oct 5, 2026

Copy link
Copy Markdown
Author

Closing because vLLM MoRI-IO integration was merged upstream in NVIDIA#432.

@cquil11 cquil11 closed this Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant