Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
set -eo pipefail
source /infmax-workspace/benchmarks/check_env.sh
check_env_vars TOKENSPEED_WHEEL_URL TOKENSPEED_KERNEL_WHEEL_URL TOKENSPEED_KERNEL_AMD_WHEEL_URL TOKENSPEED_ENGINE_WHEEL_URL TOKENSPEED_SCHEDULER_VERSION ROUTER_VERSION

pip_install=(python3 -m pip install)
if python3 -m pip install --help 2>/dev/null | grep -q -- --break-system-packages; then
pip_install+=(--break-system-packages)
fi
# The runner image provides ROCm torch. Upgrading only what the pinned wheels
# need keeps it: PyPI's torch is a CUDA build.
"${pip_install[@]}" --upgrade --upgrade-strategy only-if-needed \
--index-url https://pypi.org/simple \
"$TOKENSPEED_WHEEL_URL" "$TOKENSPEED_KERNEL_WHEEL_URL" \
"$TOKENSPEED_KERNEL_AMD_WHEEL_URL" "$TOKENSPEED_ENGINE_WHEEL_URL" \
"tokenspeed-scheduler==$TOKENSPEED_SCHEDULER_VERSION" \
"tokenspeed-smg==$ROUTER_VERSION"
python3 -c 'import torch, sys; sys.exit(0 if torch.version.hip else "torch is not a ROCm build")'
python3 -m pip check
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Muse Spark Open MXFP4 on TokenSpeed (MI355X)

The `agentic.yaml` recipe backs the
`musespark-fp4-mi355x-tokenspeed-agentic-dflash` matrix: one TokenSpeed
worker on eight MI355X GPUs with attention TP8 and MoE TP8/EP1, so every GPU
holds all 256 experts and no tokens move between GPUs. The DFlash2 draft
proposes seven tokens per step and is served as it ships, in BF16. The KV
cache is FP8, the context is the checkpoint's native 1M tokens, and prefill
runs in 8K chunks.

Each concurrency has its own override with the running-request cap
(`max-num-seqs`) and prefill/decode interval from TokenSpeed's Muse Spark
MI355X recipe:

| Concurrency | `max-num-seqs` | `prefill-decode-interval` |
| ----------- | -------------- | ------------------------- |
| 1 | 1 | 0 |
| 2 | 2 | 192 |
| 4 | 2 | 96 |
| 8 | 2 | 256 |
| 16 | 3 | 384 |

A cap below the concurrency queues the extra requests. With an interval of N,
the scheduler runs N decode rounds after each prefill chunk. Both keep decode
batches small, which lowers TPOT, at the cost of time to first token.

The recipe installs the unmodified TokenSpeed `0.1.1.post20261110` nightly,
the ROCm 7.2 Python 3.12 kernel and kernel-amd `0.1.4.post20261110`, and
`tokenspeed-engine` `0.1.0.post20261110` by SHA256, all built from upstream
revision `e8b98dd83f65795d554bb51b54684e088c69552c`, with scheduler `0.1.53`.
The engine embeds the scheduler, so it must come from the same revision as the
runtime. The runner image, pinned by digest in the master config, provides
ROCm torch; `tokenspeed-nightly-rocm.sh` installs the pinned wheels without
replacing it and checks that torch is still a ROCm build. The worker pins the
checkpoint with `revision` and the DFlash2 draft with
`speculative-draft-model-revision`.

Workers launch through `python3 -m tokenspeed.cli serve --headless` with
SMG's ZMQ transport. srtctl runs SMG directly, so the recipe passes the
gateway settings `ts serve` uses for Muse Spark: the `muse` reasoning and tool
parsers under the Symphony parser, and the engine's xgrammar backend. The two
HIP graph settings in the worker environment keep the two-stream speculative
decode graph off the runtime's slow launch path.

For synthetic throughput, the golden-AL connector derives acceptance from the
committed Muse Spark DFlash2 curve at seven draft tokens and selects the greedy
sampler, which verifies by exact match. Eval runs remove synthetic acceptance
and verify for real with the triton sampler and speculative sampling, the one
TokenSpeed verifier that reads the DFlash2 draft distribution.
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
base:
schema: 2
name: musespark-fp4-mi355x-tokenspeed-agentic-dflash
identity:
model:
revision: c7bc74fd64371315a091bc15f9723882eead0dc1
container: {}
frameworks:
tokenspeed: 0.1.1.post20261110
tokenspeed-kernel: 0.1.4.post20261110
tokenspeed-kernel-amd: 0.1.4.post20261110
tokenspeed-engine: 0.1.0.post20261110
resources:
gpu_type: mi355x
gpus_per_node: 8
observability:
enabled: false
tachometer:
enabled: false
# Weight loading, graph capture and the first kernel compilations.
health_check:
max_attempts: 720
interval_seconds: 10
frontend:
type: smg
enable_multiple_frontends: false
# srtctl runs SMG directly: these are the gateway settings `ts serve` uses for Muse Spark.
args:
reasoning-parser: muse
tool-call-parser: muse
parser: symphony
policy: round_robin
disable-circuit-breaker: true
disable-retries: true
disable-load-monitoring: true
tokenizer-cache-enable-l0: true
tokenizer-cache-enable-l1: true
health-check-timeout-secs: 610
engine: tokenspeed
setup_script: tokenspeed-nightly-rocm.sh
environment:
TOKENSPEED_WHEEL_URL: https://github.com/lightseekorg/whl/releases/download/tokenspeed-nightly-v0.1.1.post20261110/tokenspeed-0.1.1.post20261110-py3-none-any.whl#sha256=5efa0ecdb4e84e44a9407f48d8ca1b1e83feb71057483e2eacaabc88aa5c065b
TOKENSPEED_KERNEL_WHEEL_URL: https://github.com/lightseekorg/whl/releases/download/tokenspeed-kernel-nightly-v0.1.4.post20261110-rocm72/tokenspeed_kernel-0.1.4.post20261110-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=46c32a9bbad19134083667a0b8f1989bb558ae9e2b63d1fd4d6d058973ce5ce7
TOKENSPEED_KERNEL_AMD_WHEEL_URL: https://github.com/lightseekorg/whl/releases/download/tokenspeed-kernel-amd-nightly-v0.1.4.post20261110/tokenspeed_kernel_amd-0.1.4.post20261110-py3-none-any.whl#sha256=b340c4cff4e4492b31202d0bc7b13cac393097f3f0a3cf900b2f835c01ef3707
TOKENSPEED_ENGINE_WHEEL_URL: https://github.com/lightseekorg/whl/releases/download/tokenspeed-engine-nightly-v0.1.0.post20261110/tokenspeed_engine-0.1.0.post20261110-cp312-cp312-linux_x86_64.whl#sha256=13aa5eb7408419f53466e0186bb0b7ab2946314b95cd8be72bc541f37f600909
TOKENSPEED_SCHEDULER_VERSION: 0.1.53
roles:
agg:
nodes: 1
workers: 1
gpus: 8
env:
PYTHONNOUSERSITE: '1'
# One HIP queue per graph keeps the two-stream speculative decode graph
# off the runtime's slow per-node launch path.
DEBUG_HIP_FORCE_GRAPH_QUEUES: '1'
DEBUG_HIP_GRAPH_BATCH_SIZE: '1'
# Serve from the pre-seeded shared HF cache with no Hub API calls.
HF_HUB_OFFLINE: '1'
args:
served-model-name: amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4
revision: c7bc74fd64371315a091bc15f9723882eead0dc1
dtype: bfloat16
quantization: mxfp4
# Attention TP8; MoE TP8 with EP1 keeps all 256 experts on every GPU.
attn-tp-size: 8
data-parallel-size: 1
moe-tp-size: 8
ep-size: 1
pipeline-parallel-size: 1
init-expert-location: trivial
moe-backend: gluon
all2all-backend: none
kv-cache-dtype: fp8_e4m3
max-model-len: 1048576
chunked-prefill-size: 8192
max-prefill-tokens: 8192
cudagraph-capture-sizes: [1, 2, 3, 4, 5, 6, 8, 12, 16, 32, 64]
max-cudagraph-capture-size: 64
disable-prefill-graph: true
enable-prefix-caching: true
disable-kvstore: true
kvstore-ratio: 0.0
gpu-memory-utilization: 0.92
reasoning-parser: muse
grammar-backend: xgrammar
# DFlash2 drafts seven tokens per step; the verify window adds the current token.
speculative-algorithm: DFLASH
speculative-draft-model-path: RadixArk/Muse-Spark-Open-DFlash2
speculative-draft-model-revision: 46c90f412549c472941a68382d91a99e5e6372a0
speculative-num-steps: 7
speculative-eagle-topk: 1
speculative-num-draft-tokens: 8
# Real verification. The connector selects greedy sampling only for synthetic throughput.
sampling-backend: triton
enable-speculative-sampling: true
enable-metrics: true
metrics-host: 0.0.0.0
benchmark:
env:
# Warmup can leave bytes unacknowledged past AIPerf's 30 s default.
AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'tokenspeed:'

override_tp8_c1:
roles:
agg:
args:
max-num-seqs: 1
prefill-decode-interval: 0
benchmark:
env:
CONC: '1'
KV_OFFLOADING: none

override_tp8_c2:
roles:
agg:
args:
max-num-seqs: 2
prefill-decode-interval: 192
benchmark:
env:
CONC: '2'
KV_OFFLOADING: none

override_tp8_c4:
roles:
agg:
args:
max-num-seqs: 2
prefill-decode-interval: 96
benchmark:
env:
CONC: '4'
KV_OFFLOADING: none

override_tp8_c8:
roles:
agg:
args:
max-num-seqs: 2
prefill-decode-interval: 256
benchmark:
env:
CONC: '8'
KV_OFFLOADING: none

override_tp8_c16:
roles:
agg:
args:
max-num-seqs: 3
prefill-decode-interval: 384
benchmark:
env:
CONC: '16'
KV_OFFLOADING: none
16 changes: 16 additions & 0 deletions inferencex-e2e/configs/amd-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1167,6 +1167,22 @@ musespark-fp4-mi355x-sglang-agentic-dflash:
- { tp: 8, ep: 8, kv-offloading: none, spec-decoding: draft_model, conc-list: [6, 10, 28], srt-recipe: agentic.yaml }
- { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: draft_model, conc-list: [28, 36, 40], srt-recipe: agentic.yaml }

musespark-fp4-mi355x-tokenspeed-agentic-dflash:
image: lightseekorg/tokenspeed-runner-amd:rocm7.2.4-pytorch-2.14.0-gfx950@sha256:25949b7a6306e9fbc0165091cc67d9bded9965ed1d664b55f9966e7df0b05e51
model: amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4
model-prefix: musespark
runner: cluster:mi355x-amds
precision: fp4
framework: tokenspeed
multinode: false
router: { name: smg, version: 1.13.0.post20261027 }
srt-recipe-dir: musespark/tokenspeed/mi355x-fp4
scenarios:
agentic-coding:
- dram-utilization: 0.85
search-space:
- { tp: 8, ep: 1, kv-offloading: none, spec-decoding: draft_model, conc-list: [1, 2, 4, 8, 16], srt-recipe: agentic.yaml }

kimik3.1-fp4-mi355x-sglang-agentic-dflash:
image: lmsysorg/sglang:dev-kimi-k31-pro-mi35x@sha256:5901db1401ad7b8bc62ac67175f6f013b1b868e2a2dab063ee8848e7290fa9a7
model: moonshotai/Kimi-K3.1-Pro
Expand Down
12 changes: 7 additions & 5 deletions inferencex-e2e/infx/golden_al_distribution/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,16 @@ SGLang MTP synthetic throughput, the connector rejects a generation role whose
`speculative-num-steps + 1`. Manual acceptance-length settings are not needed.
Eval-only runs remove synthetic settings and use real verification.

TokenSpeed MTP recipes use the same automatic selection from
TokenSpeed MTP and DFLASH recipes use the same automatic selection from
`speculative-num-steps`, with `speculative-num-draft-tokens` equal to the step
count plus one and `speculative-eagle-topk: 1`. The connector sets
`synthetic-acceptance-length` and the required `sampling-backend: greedy` for
AgentX throughput. Eval-only and fixed-sequence runs remove that length and
select `sampling-backend: flashinfer` for real verification. TokenSpeed's
connector accepts explicit MTP flags; other speculative algorithms and the
JSON `speculative-config` shorthand are rejected.
AgentX throughput, and removes `enable-speculative-sampling`, which the greedy
sampler rejects. Eval-only and fixed-sequence runs remove that length and
select real verification: `sampling-backend: flashinfer` for MTP, and
`sampling-backend: triton` with `enable-speculative-sampling: true` for DFLASH.
TokenSpeed's connector accepts explicit MTP and DFLASH flags; other speculative
algorithms and the JSON `speculative-config` shorthand are rejected.

vLLM supports this through synthetic rejection sampling. For example, an EAGLE3 run can inject the selected YAML value through `synthetic_acceptance_length`:

Expand Down
8 changes: 5 additions & 3 deletions inferencex-e2e/infx/golden_al_distribution/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,13 @@ AL 取决于工作负载:草稿模型的预测在某些领域比其他领域

对于 Qwen3.5 的 srt-slurm 启动,srt 驱动会运行 `infx/srt_slurm/synthetic_acceptance.py` 连接器,在应用调用方覆盖设置后,根据生成角色的 `speculative-num-steps` 和 `THINKING_MODE` 选择已提交的曲线。验证 token 不计为额外的 MTP 步数。对于采用合成接受的 SGLang MTP 吞吐测试,如果生成角色缺少 `speculative-num-draft-tokens`,或者其值不是等于 `speculative-num-steps + 1` 的整数,连接器会拒绝启动。无需手动设置接受长度。仅评测运行会移除合成设置并使用真实验证。

TokenSpeed MTP 配置同样根据 `speculative-num-steps` 自动选择黄金 AL,要求
TokenSpeed MTP 和 DFLASH 配置同样根据 `speculative-num-steps` 自动选择黄金 AL,要求
`speculative-num-draft-tokens` 等于步数加一,且 `speculative-eagle-topk: 1`。
连接器为 AgentX 吞吐测试设置 `synthetic-acceptance-length` 和所需的
`sampling-backend: greedy`;仅评测和固定序列长度运行会移除该接受长度,并选择
`sampling-backend: flashinfer` 进行真实验证。TokenSpeed 连接器接受显式 MTP 参数,
`sampling-backend: greedy`,并移除 greedy 采样器不接受的 `enable-speculative-sampling`;
仅评测和固定序列长度运行会移除该接受长度,并选择真实验证:MTP 使用
`sampling-backend: flashinfer`,DFLASH 使用 `sampling-backend: triton` 和
`enable-speculative-sampling: true`。TokenSpeed 连接器接受显式 MTP 和 DFLASH 参数,
拒绝其他投机算法和 JSON `speculative-config` 简写。

vLLM 通过合成拒绝采样支持这一策略。例如,EAGLE3 运行可以通过 `synthetic_acceptance_length` 注入所选 YAML 值:
Expand Down
22 changes: 15 additions & 7 deletions inferencex-e2e/infx/srt_slurm/synthetic_acceptance.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,12 @@ def spec_parameters(role: Mapping[str, Any], engine: str) -> dict[str, Any]:
algorithm = str(args.get("speculative-algorithm", "")).lower()
if not algorithm:
return {}
if algorithm != "mtp":
raise ValueError("TokenSpeed AgentX currently supports only MTP speculation")
return {
"method": "mtp",
"num_speculative_tokens": args.get("speculative-num-steps"),
}
if algorithm not in ("mtp", "dflash"):
raise ValueError("TokenSpeed AgentX supports MTP and DFLASH speculation")
spec = {"method": algorithm, "num_speculative_tokens": args.get("speculative-num-steps")}
if algorithm == "dflash":
spec["model"] = args.get("speculative-draft-model-path", "")
return spec
if engine == "tilert":
if args.get("with-mtp") is not True:
return {}
Expand Down Expand Up @@ -320,13 +320,21 @@ def build_overrides(
]
elif worker_engine == "tokenspeed":
key = "synthetic-acceptance-length"
sampled = "enable-speculative-sampling"
if worker_al is not None and worker_spec:
overrides += ["--set", f"{prefix}.args.{key}={worker_al:g}"]
overrides += ["--set", f'{prefix}.args.sampling-backend="greedy"']
# The greedy sampler verifies by exact match and rejects sampled verification.
if sampled in (role.get("args") or {}):
overrides += ["--unset", f"{prefix}.args.{sampled}"]
else:
if key in (role.get("args") or {}):
overrides += ["--unset", f"{prefix}.args.{key}"]
if worker_spec:
if worker_spec.get("method") == "dflash":
# Only the triton sampler reads the DFlash2 draft distribution.
overrides += ["--set", f'{prefix}.args.sampling-backend="triton"']
overrides += ["--set", f"{prefix}.args.{sampled}=true"]
elif worker_spec:
overrides += ["--set", f'{prefix}.args.sampling-backend="flashinfer"']
elif worker_engine == "atom":
key = "spec-decode-acceptance-length"
Expand Down
1 change: 1 addition & 0 deletions inferencex-e2e/infx/srt_slurm/workload.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@
FABRIC_REFERENCE = "@fabric."
INSTALLERS = {
"tokenspeed-nightly.sh": ("router", "smg"),
"tokenspeed-nightly-rocm.sh": ("router", "smg"),
"glm5.3-tilert-rocm.sh": ("router", "tilert-pd-router"),
"dsv41flash-tilert-nv.sh": ("router", "tilert-pd-router"),
"vllm-mooncake.sh": ("kv-offload-backend", "mooncake"),
Expand Down
27 changes: 27 additions & 0 deletions inferencex-e2e/infx/tests/srt_slurm/test_setup_scripts.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,30 @@ def test_tokenspeed_installer_uses_direct_private_wheels_and_one_public_index(tm
"--flashinfer-version", "0.7.2.post1"],
["-m", "pip", "check"],
]


def test_tokenspeed_rocm_installer_keeps_the_image_torch_and_checks_it(tmp_path):
result, installs = run(tmp_path, "tokenspeed-nightly-rocm.sh", {
"TOKENSPEED_WHEEL_URL": "https://example.test/tokenspeed.whl#sha256=abc",
"TOKENSPEED_KERNEL_WHEEL_URL": "https://example.test/kernel.whl#sha256=def",
"TOKENSPEED_KERNEL_AMD_WHEEL_URL": "https://example.test/kernel_amd.whl#sha256=456",
"TOKENSPEED_ENGINE_WHEEL_URL": "https://example.test/engine.whl#sha256=123",
"TOKENSPEED_SCHEDULER_VERSION": "0.1.52",
"ROUTER_VERSION": "1.13.0.post20261027",
})

assert result.returncode == 0, result.stdout + result.stderr
assert installs == [[
"-m", "pip", "install", "--break-system-packages", "--upgrade",
"--upgrade-strategy", "only-if-needed",
"--index-url", "https://pypi.org/simple",
"https://example.test/tokenspeed.whl#sha256=abc",
"https://example.test/kernel.whl#sha256=def",
"https://example.test/kernel_amd.whl#sha256=456",
"https://example.test/engine.whl#sha256=123",
"tokenspeed-scheduler==0.1.52",
"tokenspeed-smg==1.13.0.post20261027",
]]
calls = [json.loads(line) for line in (tmp_path / "python3.jsonl").read_text().splitlines()]
assert calls[-2][0] == "-c" and "torch.version.hip" in calls[-2][1]
assert calls[-1] == ["-m", "pip", "check"]
Loading
Loading