Repository navigation
feat(config): add TokenSpeed Muse Spark Open MXFP4 MI355X AgentX with decode monokernels / feat(config): 新增 TokenSpeed Muse Spark Open MXFP4 MI355X AgentX(decode monokernel)配置 - #5103
Conversation
… decode monokernels Add musespark-fp4-mi355x-tokenspeed-agentic-dflash: Muse Spark Open MXFP4 on MI355X with TokenSpeed nightly wheels and the DFlash2 draft. Attention TP8 with MoE TP8/EP1 and decode-monokernel at concurrency 1, 2, 4, 6, 8, 10, 16 and 28; attention TP4 with EP4 at concurrency 28, 36 and 40. Add tokenspeed-nightly-rocm.sh, which installs the pinned wheels and keeps the runner image's ROCm torch, and let the golden-AL connector select DFlash2 acceptance for TokenSpeed DFLASH.
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
…with two-request monokernels Pin the TokenSpeed 0.1.1.post20261117, ROCm 7.2 kernel 0.1.5.post20261117, kernel-amd 0.1.4.post20261117 and engine 0.1.1.post20261117 wheels, and scheduler 0.1.58. This nightly's decode monokernel serves up to two requests a launch. TP8 running caps: 1 at concurrency 1; 2 with prefill/decode interval 384 at 2, 4, 8, 10 and 16; 1 with interval 384 at 6 (lowest time per token); 8 with interval 256 at 28. TP4 points are unchanged. Signed-off-by: Bangtian Liu <5477224+bangtianliu@users.noreply.github.com>
Resolve perf-changelog.yaml: keep main's entries unchanged and move this PR's entry to the end. Signed-off-by: Bangtian Liu <5477224+bangtianliu@users.noreply.github.com>
|
@raikonenfnu can you re-apply benchmarks? |
|
@raikonenfnu could you help me to rerun the benchmark |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
Summary
Add
musespark-fp4-mi355x-tokenspeed-agentic-dflash: Muse Spark Open MXFP4 (amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4) AgentX on MI355X with TokenSpeed nightly wheels, the DFlash2 draft (7 draft tokens, served as it ships in BF16), FP8 KV cache and 1M context (recipe, README).decode-monokernelon: running-request cap 1 at concurrency 1; cap 2 with prefill/decode interval 384 at 2, 4, 8, 10 and 16; cap 1 with interval 384 at 6; cap 8 with interval 256 at 28. Stream interval 50, 40 and 20 at 6, 10 and 28.0.1.1.post20261117, ROCm 7.2 kernel0.1.5.post20261117, kernel-amd0.1.4.post20261117and engine0.1.1.post20261117by release URL and SHA256; scheduler0.1.58andtokenspeed-smg1.16.0.post20261115from PyPI;lightseekorg/tokenspeed-runner-amd:rocm7.2.4-pytorch-2.14.0-gfx950by digest; checkpointc7bc74fd64371315a091bc15f9723882eead0dc1and DFlash2 draft46c90f412549c472941a68382d91a99e5e6372a0.tokenspeed-nightly-rocm.shinstalls them and keeps the image's ROCm torch.What the monokernel covers: the decode monokernel in this nightly serves up to two requests a launch. The cap-1 and cap-2 points take it on every decode step; at concurrency 28 only steps with one or two running requests take it, and the TP4 points do not use it.
Wheel sources: TokenSpeed, the ROCm kernel and engine are built from upstream
20c1bc26; kernel-amd from2c3d0cea, whosetokenspeed-kernel-amdtree is identical.Previous sweep (run 43973158102, November 16 nightly, one request per monokernel launch): all 11 points passed; GSM8K at TP4 C40 97.6%. This commit changes the TP8 caps at concurrency 2 to 16 and needs a new sweep.
Overlap with #4962 and #5101: this PR carries #4962's
tokenspeed-nightly-rocm.sh,workload.pyentry, DFLASH connector change and tests, and defines the same config key with a different recipe, so only one of this PR and #4962 should merge. #5101 extends the same connector for DSPARK; whichever lands second needs a small rebase ofsynthetic_acceptance.py, its tests and the golden-AL READMEs.Additional detail
RadixArk/Muse-Spark-Open-DFlash2); effective BF16 (no draft quantization flag; TokenSpeed's default handling).AI model disclosure
claude-fable-5-2(Claude Code); the November 17 nightly update byclaude-opus-5-5(Claude Code)claude-opus-5-5[1m](Claude Code), as that PR discloses.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied./use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
新增
musespark-fp4-mi355x-tokenspeed-agentic-dflash:在 MI355X 上用 TokenSpeed nightly wheel 运行 Muse Spark Open MXFP4(amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4)的 AgentX 配置,使用 DFlash2 草稿模型(7 个草稿 token,按发布原样以 BF16 运行)、FP8 KV cache 和 1M 上下文。decode-monokernel: 并发 1 的运行请求上限为 1;并发 2、4、8、10、16 的上限为 2,prefill/decode 间隔为 384;并发 6 的上限为 1,间隔 384;并发 28 的上限为 8,间隔 256。并发 6、10、28 的 stream interval 分别为 50、40、20。0.1.1.post20261117、ROCm 7.2 kernel0.1.5.post20261117、kernel-amd0.1.4.post20261117、engine0.1.1.post20261117,均按 release URL 和 SHA256 固定;scheduler0.1.58与tokenspeed-smg1.16.0.post20261115从 PyPI 安装;runner 镜像按 digest 固定;checkpoint 和 DFlash2 草稿模型按 revision 固定。tokenspeed-nightly-rocm.sh安装这些包并保留镜像自带的 ROCm torch。monokernel 的覆盖范围: 该 nightly 中的 decode monokernel 每次 launch 最多服务两个请求。上限为 1 或 2 的并发点每个 decode step 都走 monokernel;并发 28 只有在运行一到两个请求的 step 才走 monokernel;TP4 并发点不使用 monokernel。
Wheel 来源: TokenSpeed、ROCm kernel 和 engine 构建自上游
20c1bc26;kernel-amd 构建自2c3d0cea,其tokenspeed-kernel-amd代码与前者相同。上一次 sweep(run 43973158102,11 月 16 日 nightly,monokernel 每次 launch 一个请求):11 个点全部通过;TP4 C40 的 GSM8K 为 97.6%。本提交修改了并发 2 到 16 的 TP8 上限,需要重新 sweep。
与 #4962 和 #5101 的重叠: 本 PR 包含 #4962 的
tokenspeed-nightly-rocm.sh、workload.py条目、DFLASH 连接器改动及测试,并以不同的 recipe 定义了相同的配置键,因此本 PR 与 #4962 只应合入其一。#5101 为 DSPARK 扩展了同一连接器;后合入的一方需要对synthetic_acceptance.py、相关测试和 golden-AL README 做少量 rebase。claude-fable-5-2(Claude Code)编写了 recipe、README、master config 条目、changelog 条目和本说明,并沿用了 feat(config): add TokenSpeed Muse Spark Open MXFP4 MI355X AgentX / feat(config): 新增 TokenSpeed Muse Spark Open MXFP4 MI355X AgentX 配置 #4962 的安装脚本、连接器改动和测试;所有改动均由作者审阅和指导。沿用自 feat(config): add TokenSpeed Muse Spark Open MXFP4 MI355X AgentX / feat(config): 新增 TokenSpeed Muse Spark Open MXFP4 MI355X AgentX 配置 #4962 的文件由claude-opus-5-5[1m](Claude Code)编写,见该 PR 的说明。