Skip to content

[Triton/Gluon] [Config] [Perf] Tune gfx950 FP8 blockscale selections for DSv4 decode - #7614

Open
XinyuJiangCMU wants to merge 2 commits into
ROCm:mainfrom
XinyuJiangCMU:amd/dsv4-decode-gfx950-tuned-20261107
Open

XinyuJiangCMU wants to merge 2 commits into
ROCm:mainfrom
XinyuJiangCMU:amd/dsv4-decode-gfx950-tuned-20261107

Conversation

@XinyuJiangCMU

Copy link
Copy Markdown
Contributor

Problem

DeepSeek-V4 FP8 decode runs more slowly on 256-CU gfx950 GPUs because small projection GEMMs fall back to slower default kernels.

Change

Add bpreshuffle selections for N1536/K4096 and N8192/K1024, and retune the N1536/K4096 Triton buckets through M128.

Validation

The measured kernel selections match the original outputs within a relative squared error of 1e-4 (FP64, over each whole output and each 512-element block). On MI350X, fixed-length batch-32 plain decode with the full 43-layer model and TP4/EP4 averaged 525.8→549.1 decode tok/s/GPU (+4.4%), with 13.97→13.33 ms GPU decode steps.

Co-author-with: @JessicaJiang-123

Add 57 tuned gfx950 FP8 blockscale entries for DeepSeek-V4 decode shapes
to the bpreshuffle table.

Retune the N=1536, K=4096 Triton buckets for M <= 128 to select faster
small-M kernels.

Signed-off-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
@XinyuJiangCMU
XinyuJiangCMU requested review from a team and a balanced review from Copilot November 7, 2026 21:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot changed the title [Config] [Perf] Tune gfx950 FP8 blockscale selections for DSv4 decode [Triton/Gluon] [Config] [Perf] Tune gfx950 FP8 blockscale selections for DSv4 decode Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7614 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

@aiter-bot

aiter-bot commented Nov 7, 2026 •

Copy link
Copy Markdown

WAITING ON AUTHOR

PR Self-Review

Revision 13520fc45b01 · mode shadow

Waiting for the PR author. The readiness report will appear after the self-review answers are complete for this revision.

Questions to answer in the PR description

  1. What behavior does this PR change, and why is the existing behavior insufficient?
  2. What did you validate, and what remains unverified? Include commands, hardware/configuration and results.
  3. Which exact workload keys consume these configuration changes, and what evidence shows that unrelated shapes keep their previous selection?
  4. For every changed op_tests/ file, what behavior does it cover and why is it not redundant? Write None if no op test changed.

Edit the PR Self-Review section in the PR description. Include the current revision SHA; this comment refreshes automatically.

Missing or invalid answers

  • FAIL — missing ### Reviewed revision in the PR description
  • FAIL — missing ### Behavior and intent in the PR description
  • FAIL — missing ### Validation and unverified scope in the PR description
  • FAIL — missing ### PR-specific question in the PR description
  • FAIL — missing ### Changed op tests in the PR description
  • FAIL — Reviewed revision must contain at least the first 7 characters of the current head SHA
  • FAIL — Behavior and intent has no substantive answer
  • FAIL — Validation and unverified scope has no substantive answer
  • FAIL — PR-specific question has no substantive answer

Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.

@zufayu
zufayu requested review from a team and Dewei-Wang-sh November 10, 2026 01:17
Rescreen every candidate kernel for the 57 DeepSeek-V4 decode shapes
on gfx950 with rotating weights. Nine N=8192, K=1024 shapes (M=72, 120
and 136-184) now select the ASM 32x128 or 48x128 blockscale kernels;
the other 48 keep their kernels. The timing columns are refreshed from
the new measurements. The Triton configuration is unchanged.

Signed-off-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 18:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@XinyuJiangCMU

Copy link
Copy Markdown
Contributor Author

@valarLip @demonsan Could you help review this PR? Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants