Skip to content

feat(config): add TokenSpeed Muse Spark Open MXFP4 MI355X AgentX with decode monokernels / feat(config): 新增 TokenSpeed Muse Spark Open MXFP4 MI355X AgentX(decode monokernel)配置 - #5103

Open
bangtianliu wants to merge 5 commits into
SemiAnalysisAI:mainfrom
bangtianliu:tokenspeed-musespark-mi355x-monokernel

Conversation

@bangtianliu

@bangtianliu bangtianliu commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

Add musespark-fp4-mi355x-tokenspeed-agentic-dflash: Muse Spark Open MXFP4 (amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4) AgentX on MI355X with TokenSpeed nightly wheels, the DFlash2 draft (7 draft tokens, served as it ships in BF16), FP8 KV cache and 1M context (recipe, README).

  • TP8 / EP1, 8 GPUs, decode-monokernel on: running-request cap 1 at concurrency 1; cap 2 with prefill/decode interval 384 at 2, 4, 8, 10 and 16; cap 1 with interval 384 at 6; cap 8 with interval 256 at 28. Stream interval 50, 40 and 20 at 6, 10 and 28.
  • TP4 / EP4, 4 GPUs, regular decode path: concurrency 28, 36 and 40 with caps 56, 72 and 80, prefill/decode interval 96 with minimum interval 48, short-prefill-first, a 250 GB per GPU host KV store and stream interval 20.
  • Pinned: TokenSpeed 0.1.1.post20261117, ROCm 7.2 kernel 0.1.5.post20261117, kernel-amd 0.1.4.post20261117 and engine 0.1.1.post20261117 by release URL and SHA256; scheduler 0.1.58 and tokenspeed-smg 1.16.0.post20261115 from PyPI; lightseekorg/tokenspeed-runner-amd:rocm7.2.4-pytorch-2.14.0-gfx950 by digest; checkpoint c7bc74fd64371315a091bc15f9723882eead0dc1 and DFlash2 draft 46c90f412549c472941a68382d91a99e5e6372a0. tokenspeed-nightly-rocm.sh installs them and keeps the image's ROCm torch.
  • Golden AL: the connector and spec identity accept TokenSpeed DFLASH. Throughput uses golden acceptance (4.05) with the greedy sampler; evals verify for real with the triton sampler and speculative sampling.

What the monokernel covers: the decode monokernel in this nightly serves up to two requests a launch. The cap-1 and cap-2 points take it on every decode step; at concurrency 28 only steps with one or two running requests take it, and the TP4 points do not use it.

Wheel sources: TokenSpeed, the ROCm kernel and engine are built from upstream 20c1bc26; kernel-amd from 2c3d0cea, whose tokenspeed-kernel-amd tree is identical.

Previous sweep (run 43973158102, November 16 nightly, one request per monokernel launch): all 11 points passed; GSM8K at TP4 C40 97.6%. This commit changes the TP8 caps at concurrency 2 to 16 and needs a new sweep.

Overlap with #4962 and #5101: this PR carries #4962's tokenspeed-nightly-rocm.sh, workload.py entry, DFLASH connector change and tests, and defines the same config key with a different recipe, so only one of this PR and #4962 should merge. #5101 extends the same connector for DSPARK; whichever lands second needs a small rebase of synthetic_acceptance.py, its tests and the golden-AL READMEs.

Additional detail
  • Draft precision: shipped BF16 (RadixArk/Muse-Spark-Open-DFlash2); effective BF16 (no draft quantization flag; TokenSpeed's default handling).
AI model disclosure
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, a maintainer with write access to the repository has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

新增 musespark-fp4-mi355x-tokenspeed-agentic-dflash:在 MI355X 上用 TokenSpeed nightly wheel 运行 Muse Spark Open MXFP4(amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4)的 AgentX 配置,使用 DFlash2 草稿模型(7 个草稿 token,按发布原样以 BF16 运行)、FP8 KV cache 和 1M 上下文。

  • TP8 / EP1,8 张 GPU,开启 decode-monokernel: 并发 1 的运行请求上限为 1;并发 2、4、8、10、16 的上限为 2,prefill/decode 间隔为 384;并发 6 的上限为 1,间隔 384;并发 28 的上限为 8,间隔 256。并发 6、10、28 的 stream interval 分别为 50、40、20。
  • TP4 / EP4,4 张 GPU,常规 decode 路径: 并发 28、36、40 的运行请求上限分别为 56、72、80,prefill/decode 间隔 96、最小间隔 48,启用 short-prefill-first,每张 GPU 配 250 GB 的 Host KV store,stream interval 为 20。
  • 固定版本: TokenSpeed 0.1.1.post20261117、ROCm 7.2 kernel 0.1.5.post20261117、kernel-amd 0.1.4.post20261117、engine 0.1.1.post20261117,均按 release URL 和 SHA256 固定;scheduler 0.1.58 与 tokenspeed-smg 1.16.0.post20261115 从 PyPI 安装;runner 镜像按 digest 固定;checkpoint 和 DFlash2 草稿模型按 revision 固定。tokenspeed-nightly-rocm.sh 安装这些包并保留镜像自带的 ROCm torch。
  • Golden AL: 连接器和 spec identity 支持 TokenSpeed DFLASH。吞吐测试使用 golden 接受长度(4.05)和 greedy 采样器;评测使用 triton 采样器和 speculative sampling 进行真实验证。

monokernel 的覆盖范围: 该 nightly 中的 decode monokernel 每次 launch 最多服务两个请求。上限为 1 或 2 的并发点每个 decode step 都走 monokernel;并发 28 只有在运行一到两个请求的 step 才走 monokernel;TP4 并发点不使用 monokernel。

Wheel 来源: TokenSpeed、ROCm kernel 和 engine 构建自上游 20c1bc26;kernel-amd 构建自 2c3d0cea,其 tokenspeed-kernel-amd 代码与前者相同。

上一次 sweep(run 43973158102,11 月 16 日 nightly,monokernel 每次 launch 一个请求):11 个点全部通过;TP4 C40 的 GSM8K 为 97.6%。本提交修改了并发 2 到 16 的 TP8 上限,需要重新 sweep。

与 #4962 和 #5101 的重叠: 本 PR 包含 #4962 的 tokenspeed-nightly-rocm.sh、workload.py 条目、DFLASH 连接器改动及测试,并以不同的 recipe 定义了相同的配置键,因此本 PR 与 #4962 只应合入其一。#5101 为 DSPARK 扩展了同一连接器;后合入的一方需要对 synthetic_acceptance.py、相关测试和 golden-AL README 做少量 rebase。

… decode monokernels

Add musespark-fp4-mi355x-tokenspeed-agentic-dflash: Muse Spark Open MXFP4 on
MI355X with TokenSpeed nightly wheels and the DFlash2 draft. Attention TP8
with MoE TP8/EP1 and decode-monokernel at concurrency 1, 2, 4, 6, 8, 10, 16
and 28; attention TP4 with EP4 at concurrency 28, 36 and 40.

Add tokenspeed-nightly-rocm.sh, which installs the pinned wheels and keeps the
runner image's ROCm torch, and let the golden-AL connector select DFlash2
acceptance for TokenSpeed DFLASH.
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@bangtianliu
bangtianliu marked this pull request as ready for review November 16, 2026 22:15
@bangtianliu
bangtianliu requested a review from a team November 16, 2026 22:15

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +81 -16

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@raikonenfnu raikonenfnu added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision 5cdffd15983302821d8e93202aebdad20f404911.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 5cdffd15983302821d8e93202aebdad20f404911 调度trusted sweep run。
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

…with two-request monokernels

Pin the TokenSpeed 0.1.1.post20261117, ROCm 7.2 kernel 0.1.5.post20261117,
kernel-amd 0.1.4.post20261117 and engine 0.1.1.post20261117 wheels, and
scheduler 0.1.58. This nightly's decode monokernel serves up to two
requests a launch.

TP8 running caps: 1 at concurrency 1; 2 with prefill/decode interval 384 at
2, 4, 8, 10 and 16; 1 with interval 384 at 6 (lowest time per token);
8 with interval 256 at 28. TP4 points are unchanged.

Signed-off-by: Bangtian Liu <5477224+bangtianliu@users.noreply.github.com>
Resolve perf-changelog.yaml: keep main's entries unchanged and move this PR's entry to the end.

Signed-off-by: Bangtian Liu <5477224+bangtianliu@users.noreply.github.com>
@raikonenfnu

Copy link
Copy Markdown
Collaborator

@raikonenfnu can you re-apply benchmarks?

@bangtianliu

Copy link
Copy Markdown
Author

@raikonenfnu could you help me to rerun the benchmark

@raikonenfnu raikonenfnu added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision 90a34d6dd6575f551baff5ebf4a856c4f26972d9.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 90a34d6dd6575f551baff5ebf4a856c4f26972d9 调度trusted sweep run。
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants