Skip to content

[ROCm] Route AITER Triton MXFP4 MoE over local experts under EP - #63800

Open
Rohan138 wants to merge 9 commits into
vllm-project:mainfrom
ROCm:ep-w4a16-local-routing
Open

Rohan138 wants to merge 9 commits into
vllm-project:mainfrom
ROCm:ep-w4a16-local-routing

Conversation

@Rohan138

@Rohan138 Rohan138 commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

Under EP, route the AITER Triton W4A16 and W4A4 MoE over local experts only, as ROCm/ATOM does: the aiter top-k feeds ep_sort_routing. Follow-up to #59255.

Claims

  • DeepSeek-V4-Pro EP8: 1.1–2.6x output throughput vs [ROCm] Enable expert parallelism for AITER Triton W4A16 MoE #59255 (global routing). It now beats triton_unfused at every concurrency.
  • gpt-oss-120b EP8: 1.1–2.0x, with no regression at conc 1.
  • W4A4 EP fix: it already accepted EP but indexed local weights with global expert ids.

Validation

gfx950:

Model Backend tok/s gsm8k
DeepSeek-V4-Pro this PR 68 / 481 / 1402 / 2159 / 3322 0.935
DeepSeek-V4-Pro #59255 63 / 311 / 651 / 888 / 1275 0.930
DeepSeek-V4-Pro triton_unfused 59 / 356 / 883 / 1258 / 1684 0.935
gpt-oss-120b this PR 286 / 1722 / 6138 / 11080 / 17622 0.905
gpt-oss-120b #59255 263 / 1384 / 3722 / 5784 / 8920 0.920
gpt-oss-120b triton (triton_kernels) 180 / 1185 / 3499 / 5286 / 7545 0.905

gfx942:

Model Routing tok/s gsm8k
DeepSeek-V4-Flash this PR 96 / 592 / 1732 / 2584 / 3660 0.945
DeepSeek-V4-Flash #59255 87 / 402 / 920 / 1335 / 1924 0.970
gpt-oss-120b this PR 261 / 1280 / 3570 / 6531 / 9428 0.900
gpt-oss-120b #59255 243 / 883 / 2099 / 3127 / 4449 0.880

Note: W4A4 EP is partially tested on gfx1250, cc @jpvillam-amd

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

@mergify mergify Bot added the rocm Related to AMD ROCm label Oct 26, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 26, 2026
@Rohan138
Rohan138 force-pushed the ep-w4a16-local-routing branch 2 times, most recently from 463a0ef to 76ea15b Compare October 28, 2026 16:43
Rohan138 and others added 3 commits October 28, 2026 16:45
Under expert parallelism, run only the top-k stage of the aiter routing and
sort the gates with aiter's ep_sort_routing, as ROCm/ATOM does. Non-local
gates are binned out of the histogram, so the GEMMs schedule tiles for this
rank's experts only and index the local weight shard directly; gate_valid
still masks the dead gates in the combine. TP routing is unchanged.

Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
W4A4 already accepted naive EP but routed globally and indexed the local
weight shard with global expert ids. Route over local experts with aiter's
ep_sort_routing, as the W4A16 expert does, and mask dead gates with
gate_valid.

Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The DSV4 hash-routing and AITER W4A16 EP selection tests exercise the MXFP4
MoE oracle, not the softplus-sqrt top-k kernels; keep them next to the other
MXFP4 oracle tests.

Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Rohan138
Rohan138 force-pushed the ep-w4a16-local-routing branch from 76ea15b to d76a6c4 Compare October 28, 2026 16:45
@Rohan138
Rohan138 marked this pull request as ready for review October 28, 2026 16:51

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@Rohan138 Rohan138 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 28, 2026
@Rohan138

Copy link
Copy Markdown
Collaborator Author

/ci run

@Rohan138

Copy link
Copy Markdown
Collaborator Author

/amd-ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98013 for commit a757c34185bc.

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite AMD CI #15635 for commit a757c34185bc.

@mergify

mergify Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Hi @Rohan138, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

gammas=None if apply_router_weight_on_input else gammas,
swizzle_mx_scale=swizzle_mx_scale,
apply_swiglu=False,
gate_valid=gate_valid,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @jpvillam-amd: #65716 moves these W4A4 GEMMs into a new _aiter_w4a4_gemms helper that takes gate_valid, and its monolithic forward calls that helper with None.

@okorzh-amd

Copy link
Copy Markdown
Contributor

Related PRs:
#65716 CONFLICT (W4A4 GEMM refactor plus ep_sort_routing for MoRI DP+EP; merges cleanly but passes gate_valid=None)
#63396 CONFLICT (deletes the gfx1250 gather block that this PR clamps; 1 textual conflict, gate_valid survives; preshuffled weights keep experts as dim 0, so the local expert count still holds) ·
#65759/#65758 CONFLICT (drafts that edit the moved tests in test_topk_softplus_sqrt.py)
#59255 ALREADY-MERGED (global routing plus in-kernel expert_map, replaced here)
#66159 composes (naive all-gather DP; local routing works on the gathered rows)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
@mergify

mergify Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Rohan138.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 12, 2026
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>

# Conflicts:
#	tests/kernels/moe/test_ocp_mx_moe.py
@Rohan138

Copy link
Copy Markdown
Collaborator Author

/ci run

@Rohan138

Copy link
Copy Markdown
Collaborator Author

/amd-ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102257 for commit 369c8282c99c.

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite AMD CI #15819 for commit 369c8282c99c.

@mergify mergify Bot removed the needs-rebase label Nov 12, 2026
@Rohan138

Copy link
Copy Markdown
Collaborator Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103578 for commit dc4eee3a295e, running 1 failed step(s) from Buildkite CI #102257.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gpt-oss Related to GPT-OSS models ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Todo
Status: To Triage

Development

Successfully merging this pull request may close these issues.

3 participants