Skip to content

[TokenSpeed] DeepSeek-V4.1-Flash FP4 GB300 AgentX / [TokenSpeed] 新增 DeepSeek-V4.1-Flash FP4 GB300 AgentX - #5105

Open
fanyang-real wants to merge 4 commits into
mainfrom
tokenspeed-dsv41-gb300-agentx
Open

fanyang-real wants to merge 4 commits into
mainfrom
tokenspeed-dsv41-gb300-agentx

Conversation

@fanyang-real

@fanyang-real fanyang-real commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds dsv41flash-fp4-gb300-tokenspeed-agentic-dspark: DeepSeek-V4.1-Flash FP4 AgentX on GB300 with the unmodified TokenSpeed image lightseekorg/tokenspeed:rc-377b78, pinned by digest (sha256:c5e3c416…). The upstream nightly Docker workflow built it from lightseekorg/tokenspeed@377b782, and it bundles the tokenspeed-smg 1.16.0.post20261115 gateway. There is no setup script and nothing is installed at job start.

  • Serving: one DEP4 server per node (attention DP4 with TP1 attention, EP4 MegaMoE), five-step DSpark from the checkpoint's draft layers, FP8 Engram tables in host memory, 1M context, 32K prefill chunks, prefill CUDA graphs up to 16384 tokens, gpu-memory-utilization 0.85, no host KVStore.
  • Gateway: SMG with the settings ts serve applies; --grouped-dp-sticky hash over x-dynamo-parent-session-id, then x-correlation-id, keeps a session tree on one attention-DP rank.
  • Rows: a per-concurrency bundle (agg-variants.yaml) at concurrency 128, 176, 216, 240, 256, 288, 320, 352, 368, 384, 448, 512, 768 and 896, each on the measured frontier. Each row sets the running-request cap, the decode CUDA graphs and the prefill gate intervals. Rows up to 352 add --prefill-single-round, and rows up to 256 capture every decode graph size up to the cap / 4.
  • Connector: the golden-AL connector accepts TokenSpeed DSPARK and selects the DeepSeek-V4.1-Flash DSpark curve from speculative-num-steps.

Pre-submission validation on our own GB300 nodes with the pinned image (Unofficial; details below): all 14 rows completed. Dashboard tiers vs the current GB300 best (#4844): 840.8k / 766.4k / 638.1k / 550.6k / 514.3k / 442.3k / 271.8k tok/s/GPU at P90 50 / 75 / 100 / 120 / 125 / 150 / 200 (+11.4% / +27.2% / +22.2% / +16.2% / +11.0% / +6.0% / +2.2%). GSM8K at concurrency 768: strict-match 0.975 / 0.973 in two runs (pass mark 0.90).

中文

新增 dsv41flash-fp4-gb300-tokenspeed-agentic-dspark:在 GB300 上用未经修改的 TokenSpeed 镜像 lightseekorg/tokenspeed:rc-377b78 运行 DeepSeek-V4.1-Flash FP4 AgentX,配置中按 digest 固定(sha256:c5e3c416…)。该镜像由上游 nightly Docker 工作流从 lightseekorg/tokenspeed@377b782 构建,内置 tokenspeed-smg 1.16.0.post20261115 网关。配置没有 setup 脚本,任务启动时不安装任何东西。

  • 推理配置: 每个节点一个 DEP4 服务(attention DP4,attention 为 TP1,EP4 MegaMoE 专家并行),使用 checkpoint 自带 draft 层的五步 DSpark 投机解码,FP8 Engram 表放在主机内存,1M 上下文,32K 预填充分块,预填充 CUDA graph 最大 16384 tokens,gpu-memory-utilization 0.85,不使用主机 KVStore。
  • 网关: SMG 使用 ts serve 的网关设置;--grouped-dp-sticky hash 先按 x-dynamo-parent-session-id、再按 x-correlation-id 路由,使同一会话树固定在一个 attention-DP rank 上。
  • 测点: 按并发度分别配置(agg-variants.yaml),并发度为 128、176、216、240、256、288、320、352、368、384、448、512、768 和 896,均在实测前沿上。每个测点设置运行请求上限、解码 CUDA graph 和预填充门控间隔。并发度 352 及以下加 --prefill-single-round,256 及以下捕获从 1 到上限 / 4 的每个解码 graph 尺寸。
  • 连接器: golden-AL 连接器支持 TokenSpeed DSPARK,并根据 speculative-num-steps 选择 DeepSeek-V4.1-Flash 的 DSpark 曲线。

提交前验证(在我们自己的 GB300 节点上使用固定镜像,非官方结果;详见下方):14 个测点全部完成。相对当前 GB300 最佳结果(#4844)的看板分档:P90 50 / 75 / 100 / 120 / 125 / 150 / 200 分别为 840.8k / 766.4k / 638.1k / 550.6k / 514.3k / 442.3k / 271.8k tok/s/GPU(+11.4% / +27.2% / +22.2% / +16.2% / +11.0% / +6.0% / +2.2%)。并发度 768 的 GSM8K 评估两次 strict-match 为 0.975 / 0.973(及格线 0.90)。并发度 512 的 68,554 个请求中有 1 个失败,其余测点全部成功,所有测点均通过提交有效性检查。

Validation details

Pre-submission runs on our own GB300 nodes, not an InferenceX sweep (Unofficial). Each row ran the pinned image by digest with the recipe's engine args and env, SMG with the recipe's gateway args, and the AgentX client from InferenceX cf2af9a for 3600 s.

Concurrency Max running requests tok/s/GPU P90 interactivity (tok/s/user) TTFT p50 / p90 (s) Requests
128 48 178.1k 221.4 2.80 / 12.64 18871 / 18871
176 64 244.6k 209.8 3.58 / 12.82 24257 / 24257
216 78 277.3k 198.5 6.39 / 19.53 27351 / 27351
240 84 287.4k 196.2 9.23 / 24.04 28919 / 28919
256 90 300.5k 193.5 8.44 / 24.89 30404 / 30404
288 192 410.0k 157.3 1.90 / 8.53 42149 / 42149
320 224 426.9k 153.6 2.50 / 12.65 44812 / 44812
352 256 487.3k 137.0 1.55 / 7.29 50502 / 50502
368 272 504.0k 126.2 1.34 / 5.90 52459 / 52459
384 288 523.3k 124.2 1.32 / 5.64 54147 / 54147
448 320 583.0k 112.0 1.66 / 7.74 61017 / 61017
512 384 650.3k 97.3 1.61 / 7.51 68553 / 68554
768 576 834.7k 55.7 1.11 / 5.37 91190 / 91190
896 640 846.2k 39.2 1.51 / 8.74 93327 / 93327
  • One request of 68,554 failed at concurrency 512; every other row completed all requests, and every row met the submission validity checks.

  • GSM8K at concurrency 768 with real DSpark verification and FlashInfer sampling: strict-match 0.9750 / 0.9727, flexible-extract 0.9742 / 0.9719 (two runs).

AI model disclosure
  • Model/version: claude-opus-5-5
  • Role: drafted the recipe bundle, master-config entry, README, changelog entry and this description, ran the pre-submission benchmarks and evals, and ran the repository's tests, recipe generation and changed-config checks.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@fanyang-real
fanyang-real force-pushed the tokenspeed-dsv41-gb300-agentx branch from a7bd40d to 9ee32ea Compare November 16, 2026 23:48
Add dsv41flash-fp4-gb300-tokenspeed-agentic-dspark on the unmodified
lightseekorg/tokenspeed:rc-377b78 image: one DEP4 TokenSpeed server per
GB300 node behind SMG with sticky grouped DP, per-concurrency rows, and
DSPARK support in the TokenSpeed golden-AL connector.

Signed-off-by: Fan Yang <250624800+fanyang-real@users.noreply.github.com>
@fanyang-real
fanyang-real force-pushed the tokenspeed-dsv41-gb300-agentx branch from 9ee32ea to bd807a1 Compare November 17, 2026 00:31
@fanyang-real
fanyang-real marked this pull request as ready for review November 17, 2026 00:31
@fanyang-real
fanyang-real requested a review from a team November 17, 2026 00:31
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +16 -7

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @fanyang-real's task in 29s —— View job


LGTM - no blocking issues found.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

…300-agentx

Signed-off-by: Fan Yang <250624800+fanyang-real@users.noreply.github.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@fanyang-real fanyang-real added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 17, 2026
…nimum

Keep the speculative-config error text, revert the reflowed README lines and
the existing test parameters, and reduce the DSPARK test to the curve lookup.

Signed-off-by: Fan Yang <250624800+fanyang-real@users.noreply.github.com>
…300-agentx

Signed-off-by: Fan Yang <250624800+fanyang-real@users.noreply.github.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant