Skip to content

[ROCm][CI] Gate MoRI EPv2 modular-kernel cases on the production EPv2 checks - #65184

Merged
AndreasKaratzas merged 2 commits into
vllm-project:mainfrom
djramic:mk-gate-mori-epv2
Nov 2, 2026
Merged

AndreasKaratzas merged 2 commits into
vllm-project:mainfrom
djramic:mk-gate-mori-epv2

Conversation

@djramic

@djramic djramic commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Overview

Since #61361, :amd: (MI300) MoE Kernels Multi-GPU fails on main (amd-ci#15761): all 3 MoriEPv2PrepareAndFinalize cases hit a MoRI assertion on MI300X. This test-only PR registers MoRI EPv2 in the modular-kernel harness only where production would select it.

Claims

  • The MI300 Multi-GPU step passes again. The MoRI EPv1 and DeepEP cases are unchanged.
  • MoRI EPv2 cases are generated only on gfx950/gfx1250 with MoRI >= 1.2.4, the conditions RocmPlatform.check_and_update_config uses.

Validation

MI300X, rocm/vllm-dev:ci_base with MoRI 1.2.3.post1 (as in CI), 2 GPUs:

cd tests
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 pytest -v -s kernels/moe/test_modular_kernel_combinations.py -k multigpu
main this PR
Collected 33 30
MoRI EPv2 / EPv1 cases 3 / 8 0 / 8
Result 3 failed, 30 passed 30 passed

On main, the EPv2 cases fail locally with the CI error: AssertionError: combine block_num 304 exceeds xdb_flag_slots 256.

Details

The harness sets mori_ep_v2=True for the "mori" backend. That skips the EPv1 fallback production uses unless MoRI >= 1.2.4 is installed and the GPU is gfx950/gfx1250. On MI300X, MoRI's untuned EPv2 path uses one combine block per CU (304), above its fixed limit of 256. ROCm/mori#876 fixes this, but no MoRI release includes it yet.

AI-assisted tooling was used to analyze the CI logs, write the change, and run the validation. The change and the results were manually reviewed.

@djramic
djramic requested a review from bnellnm as a code owner November 2, 2026 15:52

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the rocm Related to AMD ROCm label Nov 2, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 2, 2026
@djramic djramic added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

✅ @djramic, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@djramic

djramic commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

/ci run

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #99242 for commit 683d39ed132d.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 683d39ed132d before starting CI. Pull the branch before pushing more changes.

… checks

Signed-off-by: Djordje Ramic <djoramic@amd.com>
@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #99247 for commit 1bd3f8aa348f.

ℹ️ This PR is 15 commits behind upstream main. Merged upstream main into this branch as 1bd3f8aa348f before starting CI. Pull the branch before pushing more changes.

@AndreasKaratzas
AndreasKaratzas merged commit 4890c7f into vllm-project:main Nov 2, 2026
32 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants