Skip to content

[ROCm] More gfx1250 models, di, and perf fixes - #65716

Open
jpvillam-amd wants to merge 40 commits into
vllm-project:mainfrom
ROCm:jpvillam/gfx1250_perf2
Open

jpvillam-amd wants to merge 40 commits into
vllm-project:mainfrom
ROCm:jpvillam/gfx1250_perf2

Conversation

@jpvillam-amd

Copy link
Copy Markdown
Contributor

Upstream the gfx1250 (MI455X) work from the ROCm gfx1250_perf2 branch. Everything new is gated on on_gfx1250() or is opt-in, so gfx942/gfx950 and CUDA behavior should not change. The
one exception is the rocm_block32_gemm fused-split gate, listed in the fixes table below.

New features

  • --moe-backend aiter_flydsl_mxfp4_bf16 (new Mxfp4MoeBackend.AITER_FLYDSL_MXFP4_BF16, AiterFlydslMxfp4Experts)
    • Runs the MXFP4 SiTU MoE through AITER's FlyDSL grouped GEMM on gfx1250.
    • Takes BF16 activations; AITER quantizes them internally to FP8 (a8w4) or FP4 (a4w4).
    • Weights are preshuffled with gate/up interleaving at load time.
    • Opt-in only, never auto-selected. Used by Kimi-K3.
    • On gfx1250, VLLM_ROCM_USE_AITER_MOE_SITUV2=auto now resolves to a8w4 instead of a4w4.
  • Modular AiterW4A4Experts, for DP+EP over MoRI EPv1 on aiter_triton_mxfp4_mxfp4
    • MoRI dispatches packed MXFP4 activations together with their e8m0 scales.
    • ep_sort_routing sorts the dispatched global top-k onto the local experts.
    • gate_valid lets GEMM2 skip gate slots owned by other ranks.
    • GEMM1, activation and GEMM2 live in _aiter_w4a4_gemms, which this class shares with the monolithic path.
    • The class expects quantized inputs only when use_mori_epv1_kernels is set, and does not support EPLB.
  • MuseSpark1x MXFP4 on gfx1250 (MuseSparkAiterW4A4Experts, on aiter_triton_mxfp4_mxfp4)
    • Each route is a top-1 one-hot row.
    • The gate/up clamp is disabled because the experts use unclamped SiLU.
    • BF16 layers stay on Triton when an MXFP4 backend is selected.
    • EP is not supported yet.
  • MoRI EPv1 on gfx1250
    • The arch check is relaxed from gfx942/gfx950 only to CDNA3 and newer.
    • gfx1250 gets its own launch config: 32 warps per block intranode, 16 warps / 64 blocks / 32 RDMA blocks internode, and gpu_per_node=4.
  • MoRIIO backend: "fabric" (UALink)
    • Creates a MoRI FabricBackendConfig backend.
    • Allocates the KV cache under MoRI's fabric-exportable memory pool (Worker._maybe_get_memory_pool_context), so MoRI can export fabric handles for cross-node transfers.

gfx1250 correctness fixes

Area Fix
FLA chunk_delta_h and Kimi-K3 KDA recompute_w_u Force num_stages=1 on gfx1250. Deeper pipelines corrupt state and drop gsm8k to 0. The root cause is not established.
DeepSeek-V4 compress_norm_rope_store_triton Use 16 warps for C128 on gfx1250. With 4 warps, the LLVM register allocator hangs.
DeepSeek-V4.1 ROCm attention Skip the gemm_a8w8_blockscale_bpreshuffle preprocessing, which has no gfx1250 path. Don't fuse the inverse RoPE into sparse decode.
rocm_fp8_paged_mqa_logits Use AITER paged-MQA logits on gfx1250, except for the C4A block-flat indexer K cache.
MLA prefill FlashAttnPrefillBackend gfx1250 uses AITER Triton FA. Pass return_lse, then transpose the LSE into vLLM's layout.
gpt-oss MXFP4 scale swizzle Pass activation through to the weight conversion. Use scale_kwidth=8 for AITER_TRITON_MXFP4_BF16 with SwiGLU-OAI and 4 otherwise, so the swizzle
matches the kernel that reads it.
AITER_MXFP4_BF16 on gfx1250 Remove the special case that routed it through the Triton weight format.
Unquantized MoE Fall back to Triton instead of AITER on gfx1250, which has no BF16/FP16 AITER MoE kernel.
rocm_unquantized_gemm_impl Call AITER gemm_a16w16 only for 2D activations.
rocm_block32_gemm Run in-launch fused split-K only on gfx950.
Kimi-K3 Keep the KDA gluon decode disabled on gfx1250, where it hasn't been validated yet.

Docker (gfx1250 only)

  • Dockerfile.rocm_base_gfx1250:
    • ROCm SDK 10.1 → 10.3.0a20261030, from the new wheel index.
    • PyTorch 2.11 → 2.13, Triton 3.9, MoRI 2958fce6.
    • FlashAttention moves from a fork to upstream Dao-AILab 98c1cee, built against the system AITER. The FA stage now runs after the AITER stage.
    • Adds torchvision[device-gfx1250] and the gRPC/protobuf dev packages.
  • Dockerfile.rocm_gfx1250:
    • Bumps the Triton pin to 12225a8, which fixes an a16w16 gluon GEMM race.

Test Result

Model MoE backend gsm8k flexible-extract
GLM-5.3-Flash triton 0.99 ± 0.010
Llama-3.1-405B-Instruct-MXFP4-Preview — 0.97 ± 0.017
DeepSeek-R1-0528-MXFP4 aiter_triton_mxfp4_mxfp4 0.95 ± 0.022
DeepSeek-V4-Flash aiter_triton_mxfp4_mxfp4 0.94 ± 0.024
DeepSeek-V4.1-Flash aiter_triton_mxfp4_mxfp4 0.93 ± 0.026
gpt-oss-120b-w-mxfp4-a-fp8 auto 0.92 ± 0.027
gpt-oss-120b aiter_triton_mxfp4_bf16 0.91 ± 0.029
Qwen3.8-Flash-Next auto 0.89 ± 0.031

Juan Villamizar and others added 24 commits November 3, 2026 17:12
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* Enable Gluon MoE path for oai-gpt-oss

* DSV4 Core dump fix

Co-authored-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* Mori changes

* Mori connector patch

Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>

* changing triton pin to enable oss tp4

Signed-off-by: Daniel <danichan@amd.com>

---------

Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Daniel <danichan@amd.com>
Co-authored-by: jpvillam <juan.villamizar@amd.com>
Co-authored-by: Daniel <danichan@amd.com>
Upgrade TORCH_VERSION/TORCHVISION_VERSION to the 2.13.0/0.28.0 ROCm
10.1 nightly builds.

Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Daniel <danichan@amd.com>
Co-authored-by: Daniel <danichan@amd.com>
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* flatten 3D ViT activation & dont select AITER unquantized bf16 MOE

* applying  2D gate so gfx1250 fallbacks to F.linear for qwen

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>

---------

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
…its() (#1360)

and reading the indexer K cache in wrong layout. the indexer K cache is written
preshuffled and gfx942/gfx950 read it at decode with deepgemm_fp8_paged_mqa_logits(Preshuffle=True)
but gfx1250 fell through to deepgemm_fp8_paged_mqa_logits_stage1, which reads the cache row major.
so decode time top-k indexes were wrong. after this fix gfx1250 also takes the preshuffle path.

Co-authored-by: Muhammad Ahmed <mm.ahmed2202@gmail.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
* changing rocm whl index and sdk version

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>

* bumping to newer mori commit to be compatible with sdk

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>

* switching to torch 2.13

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>

---------

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
…a16w16 gluon gemm race issues

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
…, flag not required anymore

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Kimi-K3 enablement on gfx1250:
- Opt-in AITER FlyDSL MXFP4 SiTU MoE backend (--moe-backend
  aiter_flydsl_mxfp4_bf16); SiTUv2 auto resolves to a8w4 on gfx1250.
- Pin the KDA chunk kernels to num_stages=1 on gfx1250.
- Route gfx1250 MLA prefill to AITER's Triton flash attention.
- Keep gfx1250 on the Triton KDA decode; AITER's Gluon KDA decode is not
  yet validated there.

On gfx1250 serve with AITER_FORCE_A8W4=1 and --kda-prefill-backend triton:
AITER picks a8w4 there from that flag alone (a4w4 otherwise), and the auto
KDA prefill picks AITER's Gluon chunk_kda, which fails to compile there.

Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
* using 10/27 sdk

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>

* moving sdk to oct 30 to solve a0 issues

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>

---------

Signed-off-by: Daniel Chan <Daniel.Chan@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
- AiterW4A4Experts: use_mori_kernels was split upstream into
  use_mori_epv1_kernels / use_mori_epv2_kernels. Gate the modular expert on
  MoRI EPv1, which dispatches packed MXFP4 + e8m0 scales; EPv2 dispatches
  BF16 and remaps expert ids, and is untested with ep_sort_routing.
- ruff format fixes on rebased files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Juan Villamizar <jpvillam@amd.com>
The int32 stride cast and req_idx < 0 guard in _gather_block_tables_kernel
are no longer needed with the current Triton pin: DeepSeek-V4.1-Flash on
gfx1250 runs gsm8k (0.92) and long-prompt stress at concurrency up to 128
with no faults on the upstream kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci cancel

@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Requested cancellation of 1 CI build for jpvillam/gfx1250_perf2: #102216.

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102242 for commit d5257ad0bef4.

@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 2 failed job(s) for retry in Buildkite CI #102242.

@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102539 for commit 628e59dd2a62, running 1 failed step(s) from Buildkite CI #102242.

Comment thread docker/Dockerfile.rocm_base_gfx1250 Resolved
@mergify

mergify Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jpvillam-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 17, 2026
Signed-off-by: Juan Villamizar <jpvillam@amd.com>

# Conflicts:
#	vllm/models/muse_spark/amd/muse_moe_runner.py
#	vllm/v1/attention/ops/rocm_aiter_mla_sparse.py
@mergify mergify Bot removed the needs-rebase label Nov 17, 2026
Signed-off-by: Juan Villamizar <jpvillam@amd.com>
@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ No failed, timed-out, or expired jobs need retrying in Buildkite CI #102539.

@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

❌ This PR is 12 commits behind upstream main. The bot could not merge upstream main into this branch (API returned 403: user doesn't have permission to update head repository). No new CI build was started. Either merge or rebase onto the latest main yourself and rerun /ci run, or use /ci run --allow-stale to test this branch as is.

@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

❌ This PR is 2 commits behind upstream main. The bot could not merge upstream main into this branch (API returned 403: user doesn't have permission to update head repository). No new CI build was started. Either merge or rebase onto the latest main yourself and rerun /ci run, or use /ci run --allow-stale to test this branch as is.

@jpvillam-amd

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103607 for commit 166913049739.

Signed-off-by: Juan Villamizar <jpvillam@amd.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build deepseek Related to DeepSeek models DSv4 DSv4.1 Related to DeepSeek-V4.1 models k3 kimi kv-connector quantization ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

6 participants