Repository navigation
Conversation
1870c89 to
094e542
Compare
094e542 to
05c9395
Compare
05c9395 to
26bbe1c
Compare
26bbe1c to
fdfc9ed
Compare
fdfc9ed to
65893f4
Compare
7368036 to
9935d90
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9935d90. Configure here.
| "read_mode": True, | ||
| }, | ||
| } | ||
| ) |
There was a problem hiding this comment.
MoRI workers share default control ports
High Severity
Managed MoRIIOConnector JSON omits per-worker handshake_port and notify_port, so every engine falls back to the same defaults. Co-located workers, including multiple endpoints on one node, then collide on those listeners and on engine_id.
Reviewed by Cursor Bugbot for commit 9935d90. Configure here.
|
Hardware validation of the MoRI listener-allocation fix: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35756972563/job/106845265742 MI355X, vLLM v0.28.0 + MoRI-IO, MiniMax-M3-MXFP4, two-node TP4 prefill / TP4 decode. All seven measured points at concurrency 1, 2, 4, 8, 16, 32, and 64 passed: 1,276 requests completed, zero failed. Both roles became healthy and all four decode handshake/notify listeners bound successfully. Runtime: Stopped at the owner’s request while concurrency 128 was warming up; concurrency 256 was not tested. The workflow is intentionally cancelled, not a fully green sweep. Slurm allocation 45492 was cancelled and both nodes released. No model, serving-option, or workload changes were made for this fix. |
6a780cc to
6631c48
Compare
|
Closing because vLLM MoRI-IO integration was merged upstream in NVIDIA#432. |


Summary
Integrate native vLLM Router with MoRI-IO prefill/decode workers. Static router registration and NIXL port handling do not describe MoRI's ZMQ discovery path; this PR supplies the connector-specific launch, port, and readiness contract for it on the seams
mainnow has: a connector table row on the vLLM backend, discovery hooks on the vLLM Router frontend, and per-process listeners from the port allocator.Stacked on #431, which combines AMD hardware enablement and required cluster runtime. #433 follows this PR; ATOM uses Mooncake and has no engine dependency on MoRI.
Review map
backends/vllm.py:moriiois aKVConnectorrow withdiscovery=Trueand a mode-dependent role (the row type, theconnector_for_moderesolver, and the discovery seam landed onmainin #491, so no connector name is compared in a branch here).kv_transfer_config(mode, process, runtime)builds the worker JSON; the topology arguments are optional and only a discovery row reads them. For a discovery connector the extra config carriesproxy_ip(head node),proxy_ping_port,http_port,host_ip(the worker's own routable IP),handshake_port,notify_port,read_mode: the keysmoriio_common.pyreads (vllm-project/vllm 9679173788). Upstream's fixed defaults of 6301 and 61005 collide as soon as two workers share a node, so the allocator reserves separate handshake and notification blocks of one port per local rank (vLLM adds the DP and TP rank to each base), using default bases 26000 and 27000 below this cluster’s ephemeral port range; a worker built without them is refused. Such a worker gets noVLLM_PORTscan range because MoRI-IO opens listeners inside TP children that inherit it.frontends/vllm_router.py,ports.py: in discovery mode the vLLM Router adds--kv-connector <name> --vllm-discovery-address 0.0.0.0:36367(RouterArgsin vllm-project/router) and validates the discovery rules (both roles on the connector, a P/D topology, one router on the head node). Readiness is the Router's own/health, which answers 503 until a prefill and a decode have registered (vllm-project/router 43140bc8e2,VllmPDRouter::health); the probe reads the recipe #491 hands it, socore/health.pyhas no per-frontend branch and no completions POST. Theprobe_readydocstring records why/healthand not/workers, and what the upstream fix would be.core/schema.py: refuses a discovery connector with any frontend other thanvllm-router; the Router's own rules live in itsvalidate.core/topology.py:Process.moriio_handshake_portandmoriio_notify_port.docs/vllm-router.mdgains a MoRI-IO discovery section with the full contract table;config-reference.mdpoints at it; the port tables inCLAUDE.mdanddocs/architecture.mdlist the two kinds.examples/vllm/vllm-router-moriio-disagg.yaml(colocated P/D) is in the examples matrix and in every example-driven test./healthprobe, and the recipe rules.Review the stack
The upstream PRs target NVIDIA
mainand are manually stacked, so later upstream diffs include preceding layers. The downstream fork PRs form an actual branch-based stack (main→ #1 → #11 → #7 → #14); each fork PR's Files changed tab isolates its layer for easier review.Merge in that order, then rebase the remaining layers. ATOM uses Mooncake, not MoRI; its stack position is review ordering, not an engine dependency. The core runtime does not require the observation layer, but the InferenceX launcher uses its
srtctl waitcommand.E2E validation is maintained only in the base PR: upstream #431 / fork #1.