Repository navigation
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
115f21e to
0d8c044
Compare
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
0d8c044 to
9dbc5e8
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
…VLS multicast The tail fusion builds NVLS multicast mappings during runner construction. When multicast is unavailable on the TP group, e.g. with multi-GPU NVIDIA Confidential Computing, every rank fails at startup with "routed NVLS multicast mapping is unavailable", and there is no way to turn the fusion off. Probe multicast support on every TP rank before initializing the tail op, as the MiniMax-M3 fused MoE finalize and the custom all-reduce already do, and take the default latent-MoE path unless all ranks support it. Fixes vllm-project#65568 Signed-off-by: Yuki Kitaya <72787572+yeighta@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9dbc5e8 to
6dc118d
Compare
Overview
On a TP group without NVLS multicast, Kimi K3 currently fails at startup in the SM100 latent-MoE tail fusion (TP 8/16). With this change it falls back to the default latent-MoE path, as the MiniMax-M3 fused MoE finalize already does. Multi-GPU NVIDIA Confidential Computing is one setup without multicast.
Fixes #65568.
Claims
LatentMoERunnernow probes multicast on every TP rank and requires all ranks to agree before it buildsKimiK3LatentMoETailOp. If any rank lacks multicast, it logs a warning and uses the default path instead of raisingrouted NVLS multicast mapping is unavailableon every rank.Validation
Probe on the affected machine. Intel TDX TD, 8× B300 (SM103), Blackwell multi-GPU CC with
nvidia-smi conf-compute -mgm= NVLE, guest driver 595.91.07, torch 2.13.0+cu130 (vllm/vllm-openai:v0.32.0). Both checks return "no multicast" on all 8 GPUs:These are
_SymmetricMemory.has_multicast_support(DeviceType.CUDA, i), the probe used here, andcuDeviceGetAttribute(CU_DEVICE_ATTRIBUTE_MULTICAST_SUPPORTED).End to end on the same TD. Kimi K3, TP8, vLLM 0.32.0 with this change applied (the
vllm/diff applies cleanly; the TD also carries #59981 for Confidential Computing and a backport of #63989):Same setup with the V2 model runner, and no env var or flag set:
RuntimeError: routed NVLS multicast mapping is unavailable(log in #65568)Application startup complete, KV cache 3,704,968 tokensKimiK3LatentMoETailOp.initializeWARNING [latent_moe_runner.py:120] K3 latent-MoE tail fusion requires NVLS multicast memory on every TP rank, which is unavailable. Falling back to the default path.(logged once, on the local first rank)vllm bench serve, random 1024/1024,--ignore-eos, c=16No
Tracebackappears in the server log after the prompts and the benchmark.For reference, at the same benchmark point the default latent-MoE path, with the fusion forced off by a local patch, gave 820 tok/s and a median TPOT of 18.4 ms. That was one run each. The fallback path performs like the default path, which is expected because they are the same path.
Unit test.
test_tail_fusion_falls_back_without_multicastmakes the tail op raise what the real NVLS rendezvous raises when multicast is unavailable, and makes the device probe report no multicast.The Kimi K3 B200 CI step (
models/kimi_k3) selects this file. CI cannot cover the multicast-less path end to end, because it needs 8 SM100 GPUs without multicast. The TD run above stands in for that.Details
Root cause.
LatentMoERunner.__init__enables the tail fusion from the device family and TP size alone.KimiK3LatentMoETailOp.initializethen needs NVLS multicast mappings from torch symmetric memory. When no multicast object can be created, every rank raisesrouted NVLS multicast mapping is unavailablefromallreduce_rmsnorm_reduce_scatter_early_exit.py. Nothing can switch the fusion off, so the model cannot start on such a node.Under multi-GPU CC (NVLE), the GPUs are connected over NVLink, but the driver reports no multicast support (
CU_DEVICE_ATTRIBUTE_MULTICAST_SUPPORTED= 0). Other multicast users in vLLM already check for this and fall back: the custom all-reduce MNNVL path and MiniMax-M3'ssupports_fused_moe_finalize.Design.
_has_local_multicast_supportand_all_ranks_truefromcustom_all_reduce.py, the same pairsupports_fused_moe_finalizeuses. No new probe is added.Related PRs.
INTRA_NODE, so it does not cover this case.multicastflag to thebuild_runnertest helper.Limitation. The opt-in latency-specialized decode (
decode_backend=latency_specialized) also needs multicast and still raises without it. It is off by default and outside this change.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.