Skip to content

feat(config): add TokenSpeed Muse Spark Open MXFP4 MI355X AgentX / feat(config): 新增 TokenSpeed Muse Spark Open MXFP4 MI355X AgentX 配置 - #4962

Open
efric wants to merge 14 commits into
SemiAnalysisAI:mainfrom
efric:tokenspeed-musespark-mi355x-agentx
Open

efric wants to merge 14 commits into
SemiAnalysisAI:mainfrom
efric:tokenspeed-musespark-mi355x-agentx

Conversation

@efric

@efric efric commented Nov 10, 2026 •

Copy link
Copy Markdown

Summary

Add musespark-fp4-mi355x-tokenspeed-agentic-dflash: Muse Spark Open MXFP4 (amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4) AgentX on 8x MI355X with TokenSpeed, attention TP8 / MoE TP8 EP1, the DFlash2 draft (7 draft tokens), FP8 KV cache, at concurrency 1, 2, 4, 8 and 16 with per-concurrency running-request caps and prefill/decode intervals (recipe, README).

  • Pinned: TokenSpeed nightly 0.1.1.post20261110, ROCm 7.2 kernel and kernel-amd 0.1.4.post20261110, engine 0.1.0.post20261110 (all from lightseekorg/tokenspeed@e8b98dd83), scheduler 0.1.53, lightseekorg/tokenspeed-runner-amd:rocm7.2.4-pytorch-2.14.0-gfx950 by digest. SHA256s are in the recipe; tokenspeed-nightly-rocm.sh installs them and keeps the image's ROCm torch.
  • Revisions pinned on the worker: checkpoint c7bc74fd64371315a091bc15f9723882eead0dc1, DFlash2 draft 46c90f412549c472941a68382d91a99e5e6372a0 (--speculative-draft-model-revision). DFlash2 draft served as it ships (BF16).
  • The golden-AL connector and spec identity accept TokenSpeed DFLASH: throughput uses golden acceptance (4.05) with the greedy sampler; evals verify with the triton sampler and speculative sampling.
Additional detail
  • Draft precision: shipped BF16 (RadixArk/Muse-Spark-Open-DFlash2); effective BF16 (no draft quantization flag; TokenSpeed's default handling).
AI model disclosure
  • Model/version: claude-opus-5-5[1m] (Claude Code)
  • Role: prepared the recipe, setup script, connector changes, tests and this description; reviewed and directed by the author.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

新增 musespark-fp4-mi355x-tokenspeed-agentic-dflash:在 8 张 MI355X 上用 TokenSpeed 运行 Muse Spark Open MXFP4(amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4)的 AgentX 配置。注意力 TP8,MoE TP8 EP1,使用 DFlash2 草稿模型(7 个草稿 token),FP8 KV cache,并发 1、2、4、8、16,每个并发点设置各自的运行请求上限和 prefill/decode 间隔。

  • 固定版本:TokenSpeed nightly 0.1.1.post20261110,ROCm 7.2 kernel 与 kernel-amd 0.1.4.post20261110,engine 0.1.0.post20261110(均构建自 lightseekorg/tokenspeed@e8b98dd83),scheduler 0.1.53,runner 镜像按 digest 固定。SHA256 写在 recipe 中;tokenspeed-nightly-rocm.sh 安装这些 wheel 并保留镜像自带的 ROCm torch。
  • worker 固定 revision:checkpoint c7bc74fd64371315a091bc15f9723882eead0dc1,DFlash2 草稿模型 46c90f412549c472941a68382d91a99e5e6372a0(--speculative-draft-model-revision)。DFlash2 草稿模型按发布原样(BF16)运行。
  • golden-AL 连接器和 spec identity 支持 TokenSpeed DFLASH:吞吐测试使用 golden 接受长度(4.05)和 greedy 采样器;评测使用 triton 采样器和 speculative sampling 进行真实验证。
  • 草稿模型精度:发布精度 BF16,实际运行精度 BF16(未设置草稿量化参数,使用 TokenSpeed 默认处理)。

efric added a commit to efric/InferenceX that referenced this pull request Nov 10, 2026
Add a TP8/EP1 TokenSpeed AgentX configuration for Muse Spark Open MXFP4 on
MI355X with the DFlash2 draft, at concurrency 1, 2, 4, 8 and 16.

- Recipe, README and amd-master entry; the public 20261109 TokenSpeed
  nightly, kernel, kernel-amd and engine wheels pinned by SHA256, and the
  runner image by digest.
- tokenspeed-nightly-rocm.sh installs the pinned wheels and keeps the
  image's ROCm torch.
- The golden-AL connector and spec identity accept TokenSpeed DFLASH:
  throughput uses golden acceptance with the greedy sampler; evals verify
  with the triton sampler and speculative sampling.

Signed-off-by: Eric Feng <Eric.Feng@amd.com>
Signed-off-by: Eric Feng <Eric.Feng@amd.com>
…k revisions

- Pin TokenSpeed 0.1.1.post20261110, kernel and kernel-amd 0.1.4.post20261110
  and engine 0.1.0.post20261110 (all from e8b98dd83) by SHA256, with scheduler
  0.1.53.
- Pin the checkpoint with revision and the DFlash2 draft with
  speculative-draft-model-revision, which this nightly resolves offline.

Signed-off-by: Eric Feng <Eric.Feng@amd.com>
…eval sampling

- The connector accepts TokenSpeed MTP and DFLASH; throughput removes
  enable-speculative-sampling under the greedy sampler.
- Eval-only runs verify MTP with flashinfer and DFLASH with triton and
  speculative sampling.

Signed-off-by: Eric Feng <Eric.Feng@amd.com>
@efric
efric force-pushed the tokenspeed-musespark-mi355x-agentx branch from 9ad0914 to 6cf38d7 Compare November 11, 2026 02:18
@efric
efric marked this pull request as ready for review November 11, 2026 03:15
@efric
efric requested a review from a team November 11, 2026 03:15

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +81 -16

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

Keep the infx diff minimal: TokenSpeed's spec_parameters builds one spec
for MTP and DFLASH, the speculative-config error keeps its original
wording, and the identity test for the DFlash2 curve is dropped since the
DFLASH synthetic-acceptance test already selects that curve. Connector
overrides are unchanged for throughput, eval-only, non-agentic and
non-speculative runs.

Signed-off-by: Eric Feng <Eric.Feng@amd.com>
…mi355x-agentx

Signed-off-by: Eric Feng <Eric.Feng@amd.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
…mi355x-agentx

Signed-off-by: Eric Feng <Eric.Feng@amd.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@functionstackx functionstackx added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 12, 2026
…mi355x-agentx

Signed-off-by: Eric Feng <Eric.Feng@amd.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@efric

efric commented Nov 12, 2026

Copy link
Copy Markdown
Author

@functionstackx thanks for helping add the label to trigger CI! can you also help approve the workflow? also @raikonenfnu

@efric

efric commented Nov 12, 2026

Copy link
Copy Markdown
Author

It seems like the label also needs to be removed + readded to trigger CI even after workflows have been approved after a rebase unfortuantely

@raikonenfnu raikonenfnu added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision eb799a0f52c1be62faeb04c821bfc9c41b7cf702.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 eb799a0f52c1be62faeb04c821bfc9c41b7cf702 调度trusted sweep run。
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

efric and others added 2 commits November 12, 2026 10:22
…pes set

The TokenSpeed Muse Spark MI355X recipe set PYTHONNOUSERSITE,
PYTHONUNBUFFERED, PYTHONSAFEPATH and NO_COLOR in its global environment,
which srt-slurm also exports into the batch script and so into the
cluster's host_setup hook; no other MI355X recipe sets any of them
globally. Follow the MI355X recipes: PYTHONNOUSERSITE moves to
roles.agg.env, where most of them set it, and the rest are dropped.
Also drop TOKENSPEED_JIT_COMPILE_CHECK=warn, which is the engine default,
and TOKENSPEED_HEALTH_CHECK_TIMEOUT, which the pinned TokenSpeed build no
longer reads. The TokenSpeed wheel pins stay global because the install
script also runs in the SMG gateway's container.

Signed-off-by: Eric Feng <Eric.Feng@amd.com>
…mi355x-agentx

Signed-off-by: Eric Feng <Eric.Feng@amd.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@raikonenfnu raikonenfnu added full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision 5e9c84eb57fccf549f96c32c0d0953bce4417940.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 5e9c84eb57fccf549f96c32c0d0953bce4417940 调度trusted sweep run。
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

…mi355x-agentx

Signed-off-by: Eric Feng <Eric.Feng@amd.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants