Skip to content

[AMD][MI35X] 1111 DSV41 Flash sglang mtp agentic / [AMD][MI35X] 1111 DSV41 Flash sglang mtp agentic 配置更新 - #5029

Merged
Oseltamivir merged 6 commits into
mainfrom
amd/dsv41flash-mi355x-sglang-1111
Nov 17, 2026
Merged

Oseltamivir merged 6 commits into
mainfrom
amd/dsv41flash-mi355x-sglang-1111

Conversation

@zijiecode

Copy link
Copy Markdown
Collaborator

No description provided.

@zijiecode
zijiecode requested a review from a team November 12, 2026 04:17
@zijiecode zijiecode added AMD non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) agentx AgentX benchmarks, recipes, and infrastructure labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment on lines 63 to +68
# One variant per point; the saturation point gets a longer warmup drain. A longer
# prefill-decode interval trades TTFT for P90 ITL: 192 with lpm up to c8, 96 at c16
# and 32 at c32. amd-master.yaml runs TP2 through c32, TP4 through c16; DP2 covers
# c32 and c64.
# and 8 at c32. amd-master.yaml runs TP2 through c32, TP4 through c16; DP2 covers
# c32 and c64. TP4 c1 and TP2 c4, c16 and c32 cap running requests at 1, 1, 4 and 8:
# sessions past the cap queue instead of joining the decode batch, which keeps each
# token stream at small-batch speed and lifts P90 interactivity at the cost of TTFT.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): The section header comment now repeats the per-point values, so maintainers have one more copy to keep in sync with the overrides. AGENTS.md bans header comments on a recipe or section and comments that restate the code. Lines 65-66 ("8 at c32", "cap running requests at 1, 1, 4 and 8") just copy each override's values. Fix: drop the value lists from this block, and if a reason is needed, put one line next to the max-running-requests fields saying why the cap sits below CONC.

Why this was flagged

The diff adds to the header block above override_tp2_c1 (agentic.yaml:63-68). It lists max-running-requests for TP4 c1 and TP2 c4, c16 and c32 as "1, 1, 4 and 8", and the c32 prefill-decode interval as 8. Those numbers already appear at lines 114, 147, 161, 163 and 286. AGENTS.md:56 bans header comments describing a recipe, override or section, and AGENTS.md says comments must not restate the code. The next retune that edits one of those fields without editing this block will leave the comment stating the wrong caps. Nothing checks comments against values, so the drift would go unnoticed. The base had the same header but did not list these caps.

Verification: The diff adds to the section-level comment above override_tp2_c1 (agentic.yaml:63-68). It now says "and 8 at c32" and "TP4 c1 and TP2 c4, c16 and c32 cap running requests at 1, 1, 4 and 8". AGENTS.md:54-58 says: "No header comments describing a file, topology, config, recipe, override, or section" and "One short line is the norm; never restate the code." The value list breaks the restate rule directly.

wunhuang added 4 commits November 12, 2026 17:23
… P90 interactivity

TP4 c1 runs 1 request instead of 2, and TP2 c4, c16 and c32 run 1, 4 and 8
instead of 8, 32 and 64. Sessions past the cap queue instead of joining the
decode batch, so each token stream keeps small-batch decode speed.
Image, search space and all other flags are unchanged.
With max-running-requests 8 a decode step is at most 8 x 6 = 48 rows, which
the AMD monokernel covers, so TP2 c32 now runs it like c4, c8 and c16. The
monokernel refuses SGLANG_HIP_DSV41_INDEX_K_PAGE8, so that env is dropped.
prefill-decode-interval goes from 32 to 8.
Image, search space and all other flags are unchanged.
…P2 c32

--tokenizer-backend fastokens turns on the segmented DeepSeek-V4.1 prompt
encoder (sglang#48928). It needs an image that ships fastokens
(sglang#49724).
…r the TP2 c32 recipe

Bump the image to lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261111, which pins
sgl-amd-mono 7113024f (sglang#49428) and ships fastokens (sglang#47923), both
needed by the TP2 c32 monokernel + fastokens recipe. Fold the changelog entries
into one.
@zijiecode
zijiecode force-pushed the amd/dsv41flash-mi355x-sglang-1111 branch from 14fdfa6 to 7630bdc Compare November 13, 2026 01:23
@github-actions

Copy link
Copy Markdown
Contributor

@1am9trash

Copy link
Copy Markdown
Collaborator
image

@1am9trash

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 43344632854

@1am9trash 1am9trash left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Link: https://inferencex.semianalysis.com/inference?unofficialRun=43344632854
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Link: https://inferencex.semianalysis.com/evaluation?unofficialRun=43344632854
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Recipe:

Draft precision:

  • No draft-specific setting; draft is the in-checkpoint DSpark head of deepseek-ai/DeepSeek-V4.1-Flash.
  • Stored: mtp.* attn / shared experts / main_proj F8_E4M3, routed experts MXFP4, norms/gate BF16.
  • Pinned image converts wo_a FP8->BF16 by default.
  • That image includes sgl-project/sglang#45480: V4.1 moe, draft included, default to MXFP8 acts; AITER_BF16_FP8_MOE_BOUND is no longer set.
    --monokernel-backend amd (c<=32 only) also uses MXFP8 expert acts, same as the image default; no draft weight change.
  • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE unset (default false; gated to modelopt_fp4 + deep_gemm, not this checkpoint).
  • The image includes sgl-amd-mono b27d604 via sgl-project/sglang#45546; its DeepSeek-V4.1 DSpark draft spec uses MXFP8 expert activations and does not requantize draft weights.

Signed: @1am9trash

@1am9trash

1am9trash commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Check 2 is a wrong alarm.
Pass after rerun.

In c32 log, we can see following msgs, which prove the feature works well and has no precision regression.
CI: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43344632854/job/131799621541

Fast token path works.

[2026-11-13 11:46:45] fastokens backend enabled - transformers patched successfully

Mono path works.

monokernel deepseek_v41_hip on gfx950 runs target_verify steps of batch size [1, 2, 3, 4, 5, 6, 7, 8] x 6 tokens (8 of 8 captured decode graphs); every other step runs the standard path

image

@SemiAnalysisAI SemiAnalysisAI deleted a comment from github-actions Bot Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

❌❌❌ REJECTED ❌❌❌

The verifier did not produce a valid verdict. Retry the sign-off verification.

Assessed commit: 6b68cc09d051e10639df9fd8552830f5b5455acf.

@Oseltamivir
Oseltamivir merged commit 3253e65 into main Nov 17, 2026
37 checks passed
@Oseltamivir
Oseltamivir deleted the amd/dsv41flash-mi355x-sglang-1111 branch November 17, 2026 06:42
@github-actions

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 13 0 13 0 0
eval 2 2 2 0 0
trace 13 0 0 0 0
power 13 0 0 0 0

benchmarkFiles 14, rawBenchmarkRows 26, coveredByPerJobFiles 13, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 168388, powerPublicationPoints 13, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure AMD non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants