Skip to content

[Docs] Cookbook: refresh SemiAnalysis AgentX launches (GLM-5.2 B300, DeepSeek-V4.1 GB300) - #43799

Merged
nvpohanh merged 1 commit into
sgl-project:mainfrom
nvpohanh:pohanh/cookbook-agentx-refresh-1011
Oct 12, 2026
Merged

nvpohanh merged 1 commit into
sgl-project:mainfrom
nvpohanh:pohanh/cookbook-agentx-refresh-1011

Conversation

@nvpohanh

@nvpohanh nvpohanh commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

[by Claude Code]

Motivation

Refresh the cookbook's interactive Agentic Long-Context Deployment sections (#42726, #43083) on GLM-5.2, DeepSeek-V4, DeepSeek-V4.1, Qwen3.5 and Qwen3.8-Flash-Next against the latest SemiAnalysis InferenceX AgentX submissions. The configurations are mirrored from SemiAnalysis submissions, as raw launch commands, each linked to the run that measured it.

Source

InferenceX main (4f00011e), with these open InferenceX PRs overlaid:

PR Page Lane(s) Pinned to
#3890 GLM-5.2 NVFP4 B300 SGLang GLM-5.2 glm5.2-fp4-b300-sglang-agentic-mtp run 38021505485 (/use), verified
#3836 GLM-5.2 FP8 B300 SGLang GLM-5.2 glm5.2-fp8-b300-sglang-agentic-mtp run 37815475708 (/use), verified
#3817 DSV4 B300 Dynamo+SGLang DeepSeek-V4 dsv4-fp4-b300-dynamo-sglang-agentic-agg, -disagg run 37715955859 (/use), verified
#3701 DSV4 GB200 Dynamo+SGLang DeepSeek-V4 dsv4-fp4-gb200-dynamo-sglang-agentic-disagg run 37656460117 (/use), verified
#3606 DSV4 B300 SGLang DeepSeek-V4 dsv4-fp4-b300-sglang-agentic-hicache-mtp no /use yet: in progress, PR head 5b394d39
#3860 DSV4.1 GB300 Dynamo+SGLang DeepSeek-V4.1 dsv41flash-fp4-gb300-dynamo-sglang-agentic-disagg no /use yet: in progress, PR head b2061c77
#3859 DSV4.1 GB300 SGLang DeepSeek-V4.1 dsv41flash-fp4-gb300-sglang-agentic-dspark no /use yet: in progress, PR head e182134f
#2978 Qwen3.5 B300 Dynamo+SGLang Qwen3.5 qwen3.5-fp4-b300-dynamo-sglang-agentic-pareto run 34548026742 (/reuse-sweep-run), verified

A lane from a PR without an authorized /use run is rendered at the PR head and labeled "In progress in SA InferenceX AgentX". It switches to the measured run on the next sync after the /use comment or the merge. Four overlays from #43083 (InferenceX #3816, #3840, #3782, #3744) have since merged into InferenceX main and are now mirrored from there, at their dashboard runs.

What this PR does

Page Lanes Deployments Points Hardware
GLM-5.2 15 25 57 GB300, GB200, B300, B200, H200
DeepSeek-V4 12 26 75 GB300, GB200, B300, B200, H200
DeepSeek-V4.1 9 25 88 GB300, GB200, B300, B200, H200, H100
Qwen3.5 19 50 221 GB300, GB200, B300, B200, H200, H100
Qwen3.8-Flash-Next 4 4 22 GB300, B300, B200, H200

Only the generated data files and _agentx.jsx change; the page text, Deployment panels and Playgrounds are untouched. Rendered commands change only for these lanes:

  • GLM-5.2 NVFP4 B300 (InferenceX #3890). c1/c4/c8/c20 now use 5 EAGLE steps (6 draft tokens) and become their own deployment, "Agg TP8 · MTP 5-1-6". c12–c48 keep 3 steps ("Agg TP8 · MTP 3-1-4").
  • DeepSeek-V4.1 GB300 SGLang (InferenceX #3859). All of this lane's deployments move to the nightly-dev-cu13-20261012-e6072935 image. TP4 c1 and TP2 c8–c32 are retuned, and c64/c128 move from TP2 to a new "Agg DEP2" deployment behind the SGLang router. The new SGLANG_NUMA_BIND_V2=0 env var (TP2 c8/c16, DEP2) is mirrored as submitted.
  • DeepSeek-V4.1 GB300 Dynamo disaggregated (InferenceX #3860). Adds five shapes (1P3D, 2P2D, 2P3D, 2P4D, 2P5D). Every worker gets SGLANG_DSPARK_FOLDED_SAMPLING=2. 2P1D's router session-affinity TTL goes from 1 s to 3600 s and its --mem-fraction-static from 0.85 to 0.8. The lane shows as in progress until InferenceX #3860 has a measured run.

Status-only changes (the commands are byte-identical to main):

  • Verified: Qwen3.5 FP8 GB300 midcurve, Qwen3.8-Flash-Next GB300 and DeepSeek-V4.1 B300 (merged).
  • Verified at their /use runs: DeepSeek-V4 GB200 disaggregated (InferenceX #3701) and GLM-5.2 FP8 B300 c12 (InferenceX #3836).

HiCache donor for DeepSeek-V4.1. Every DeepSeek-V4.1 deployment's derived HiCache variant borrows its flags from dsv41flash-fp4-gb300-dynamo-sglang-agentic-disagg. Only measured lanes donate KV settings, and InferenceX #3860's revision of that lane has no measured run yet. So the lane's measured main version (InferenceX #3598's run) keeps donating, used only as a fallback where no displayed measured lane can donate. Every derived HiCache command on the page stays byte-identical to main; the per-deployment note now reads "...launch on InferenceX main".

Component (_agentx.jsx), three small changes:

  • A point's own measured KV tier is always selectable. Before, the KV row enabled only the tiers the deployment could derive, so a point measured with a tier that had no donor showed its "as measured" chip disabled. This sync hit that case before the donor fallback above was added; no point on the current pages is affected.
  • A disabled KV chip's sub-label reads "not available"; the tooltip still gives the reason. It used to say "not available for this image", which was wrong for HiCache disabled for lack of a verified donor (Qwen3.8-Flash-Next B300, B200 and H200).
  • The derived-HiCache note adds "on InferenceX main" when the donor is the measured main version of a lane that an open PR replaces.

How the commands are produced

The pipeline and rules are the same as in #42726 / #43083. Each lane is rendered at the exact commit of the run that produced its dashboard data or, for an open PR, the run named by its authorized /use comment (the PR head when there is none). SA-only settings are removed: the benchmark harness, cluster paths and caches, telemetry, and InferenceX's synthetic speculative acceptance (disclosed per deployment). Placeholders replace IPs, ports and NICs. Router and KV-offload variants that were not run are derived and labeled "Derived — not run".

  • Recipe-only overlays. InferenceX #3890 changes only the lane's srt recipe; its config entry is unchanged apart from comments. An open PR now also claims a lane whose recipe file it changes.
  • Not mirrored. qwen3.5-fp8-h100-sglang-agentic (H100 without MTP) has never produced dashboard data, so there is no run to pin it to.
  • Images. InferenceX #3859 pins lmsysorg/sglang:nightly-dev-cu13-20261012-e6072935@sha256:b5c48869… (multi-arch). Pruned nightly tags are flagged on the page as before.

The data files are generated; please review the component changes rather than the JSON.

Validation

  • mint validate (strict build) and mint broken-links --check-anchors --check-redirects pass; node docs/scripts/check_cookbook_configs.mjs passes; pre-commit run --all-files passes.
  • Headless-browser click-through under mint dev of every hardware / checkpoint / deployment / concurrency / KV tier / router combination on all five pages (2,358 command sets: GLM-5.2 282, DeepSeek-V4 424, DeepSeek-V4.1 528, Qwen3.5 1,048, Qwen3.8-Flash-Next 76), opening every tab. The rendered commands match the expected text exactly. For every deployment, a non-default selection's "Copy link" URL was reopened in a fresh tab and restored the same selection and tab (130 links).
  • The Deployment-panel footer link still appears on GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 and Qwen3.8-Flash-Next and scrolls to the section; Qwen3.5's anchor sits directly above the explorer; pages that use the Deployment panel without the prop still show no link.
  • The component's decoder and KV-tier transform, run through Node, reproduce the Python-generated commands for all 2,358 combinations, including every point's measured tier. Every KV-cache flag in a shown command exists in the SGLang commit of that deployment's image (188 image/flag pairs).
  • Compared with main, every deployment outside the three changed lanes above renders byte-identical commands on all five pages, including the derived router and KV-offload variants.
  • Single-node commands from bash launchers were accepted only where the dry run of the pinned launcher reproduced the server command in the run's job log (237 points).
  • Every submitted and derived sglang.launch_server, dynamo.sglang, sglang_router.launch_router and dynamo.frontend command (1,957 distinct, across 28 images) parses with the argument parsers of its own submission image (CPU only). The 133 commands new in this PR were parsed now, each in its own image; the rest are identical to commands parsed for [Docs] Cookbook: interactive SemiAnalysis InferenceX AgentX launches for GLM-5.2, DeepSeek-V4 and DeepSeek-V4.1 #42726 / [Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 #43083.

No serving or benchmark runs were performed for this PR; the numbers live in the linked SemiAnalysis runs. SGLang CI was not triggered.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #38220411660
Latest PR Test (Extra): ❌ Run #38220411245
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

…DeepSeek-V4.1 GB300)

Mirror SemiAnalysis InferenceX main (4f00011e), overlaid with InferenceX
PRs sgl-project#3890, sgl-project#3836, sgl-project#3817, sgl-project#3701, sgl-project#3606, sgl-project#3860, sgl-project#3859 and sgl-project#2978, into the
Agentic Long-Context Deployment sections of GLM-5.2, DeepSeek-V4,
DeepSeek-V4.1, Qwen3.5 and Qwen3.8-Flash-Next.

- GLM-5.2 NVFP4 B300: EAGLE 5-1-6 at c1/c4/c8/c20 as its own deployment (sgl-project#3890).
- DeepSeek-V4.1 GB300: new nightly, retuned TP2/TP4 points and DEP2 at
  c64/c128 (sgl-project#3859); five more disaggregated shapes (sgl-project#3860).
- DeepSeek-V4.1's derived HiCache flags keep coming from the measured main
  version of the GB300 disaggregated lane while sgl-project#3860's revision is unmeasured.
- _agentx.jsx: a point's measured KV tier is always selectable; disabled KV
  chips read "not available" (the tooltip has the reason); the derived-HiCache
  note says when its donor comes from InferenceX main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator Author

Verified locally that UI looks fine. Force-merge this to unblock SA AgentX submissions.

@nvpohanh
nvpohanh merged commit b0c84d4 into sgl-project:main Oct 12, 2026
104 of 108 checks passed
Liwansi added a commit to Liwansi/sglang_vans that referenced this pull request Oct 13, 2026
…n_allgather

* 'main' of https://github.com/sgl-project/sglang: (236 commits)
  [NPU] Stabilize HiCache and registered CI tests (sgl-project#43308)
  [Docs] Cookbook: refresh SemiAnalysis AgentX launches (GLM-5.2 B300, DeepSeek-V4.1 GB300) (sgl-project#43799)
  [Docs] Update GLM-5.3-Flash playground: B300 default, CP8, EP and PD (sgl-project#43802)
  [DSV4.1][PD][1/N] Support prefill node use cp parallel strategy (sgl-project#40323)
  [BCG][DSV4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs (sgl-project#38603)
  [XPU] Support decode context parallelism (DCP) on Intel XPU (sgl-project#34355)
  Keep NIXL Mamba checkpoint keys separate by TP rank (sgl-project#43594)
  Capture KV host transfer buffers (sgl-project#43711)
  [CI] Move tests that do not need GB300 off the per-PR GB300 suite (sgl-project#43784)
  [HiCache] Preserve all MTP KV and recurrent state (sgl-project#40223)
  [HiCache] Support Mamba in buffer-only host memory mode (sgl-project#36345)
  [PD] Poison the deferred abort ack when a Mooncake send fails (sgl-project#43719)
  [Perf] Route BF16 mixed-batch decode tails through decode attention (sgl-project#43238)
  [Fix] Link cuda_ipc JIT against CUDA driver stubs dir (sgl-project#43764)
  docs: clarify prefill CP terminology and simplify wording (sgl-project#43787)
  [diffusion][xpu] FLUX.2: enable the bit-exact Triton SwiGLU on XPU (sgl-project#41940)
  [Fix] Adapt to transformers 5.18 / 5.19 changes (sgl-project#43688)
  [XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast (sgl-project#42048)
  [Deps] Bump FlashInfer to 0.7.1 (sgl-project#43494)
  docs: sync LMSYS SGLang blog cards (sgl-project#43462)
  ...
Jianhong-Zhang added a commit to Jianhong-Zhang/sglang that referenced this pull request Oct 21, 2026
* [diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against cuDNN SDPA on Blackwell (sgl-project#43373)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] fix: fail MiniMax-H3 quality="high" closed on substituted DiT weights (sgl-project#34978)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* [diffusion] Z-Image: autotune the shared transformer block at both token scales (sgl-project#35755)

* [DSv4.1] Initialize the full-to-SWA mapping for request-window pools (sgl-project#43165)

* [Dsv4.1] Add DeepSeek V4.1 DSPARK TP and DP EP accuracy coverage (sgl-project#43478)

* [Refactor][TCPCG] Retire backend selection and configuration (3/9) (sgl-project#42287)

* [CI] Remove the GLM-5.3-Flash PTX KDA prefill nightly test (sgl-project#43470)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* Clamp default decode CUDA graph max_bs to max_running_requests (sgl-project#43172)

* [Model Loader] Add InstantTensor loader support (sgl-project#40453)

* [CI] Fix Python 3.10 failure in prefill complete integration test (sgl-project#43505)

* [diffusion] fix: keep a LoRA unmerged under merge_mode auto when merging would round its update away (sgl-project#43385)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [AMD] Resolving bug in per batch size graph capture due to multiple capture runs per batch size (sgl-project#39840)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: HAI <hixiao@gmail.com>

* [AMD][CI] Drop removed tp_rank/tp_size kwargs from small-M FP8 proj test (sgl-project#43318)

* [AMD] Add an opt-in AITER CK blockscale GEMM for gfx942 block-FP8 linears (sgl-project#41071)

Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com

* [CI] Drop CPU tests with no bug-catching signal and run Rust workspace tests in their own workflow (sgl-project#43503)

* [rust-server] preserve PD routing in chat and completions (sgl-project#42989)

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

* [AMD][CI] Cap pyarrow below 26 for lmms-eval (sgl-project#43547)

* [Rust] Upgrade sglang-radix-tree to PyO3 0.29 (sgl-project#42807)

Co-authored-by: colleen-tml <296620403+colleen-tml@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [CI] Fix EAGLE3 CUDA parity prefill chunk alignment (sgl-project#43538)

* [diffusion] perf: pipeline every USPAttention Ulysses path's exchange over the copy engine (sgl-project#43521)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] perf: run head_dim-128 QK-norm + RoPE two rows per warp for every RoPE layout (sgl-project#43508)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Responses] Fix max_output_tokens overestimate on multimodal requests (sgl-project#35524)

* [HiSparse] Avoid host synchronization in eager backup (sgl-project#41446)

* [MemCache] refactor: require an anchored lock receipt for non-root releases (sgl-project#41452)

* [HiCache] Add staged write-back for the page-unified KV cache layout (sgl-project#39606)

* [kv-shard 3/4] Enable Control Plane C (sgl-project#40102)

* [Docs] Document Qwen3-4B/8B H200 mixed-chunk latency tradeoffs (sgl-project#43551)

* [Docs] Let the Qwen3 B300 recipe use the default attention backend (sgl-project#43554)

* [HiCache] Keep protected cgroup page cache charged in host budgets (sgl-project#42788)

Co-authored-by: Ke Bao <ispobaoke@gmail.com>

* [Scheduler] Use committed length in spec mamba track boundary check (sgl-project#43029)

Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>

* [Refactor] Record what a bound stage path does instead of reading it back (sgl-project#43453)

* [Fix] Keep residual rows and sums consistent across appends and pipeline handoffs (sgl-project#43454)

* [Refactor] Make residual read and update capabilities explicit and checked (sgl-project#43455)

* [Refactor] Decide input-scattered eligibility and stage rejections in one place (sgl-project#43456)

* [Refactor] Bind a layer stack in one pass over each stage's data flow (sgl-project#43457)

* [Refactor] Decide an exit's fixed facts when the stage is built (sgl-project#43458)

* [Refactor] Declare pipeline neighbours from a shared declaration function (sgl-project#43459)

* [Refactor] Declare every model's stages from a shared function and drop the meta-device neighbour build (sgl-project#43460)

* [CI] Fix Kimi-Linear DCP PD test startup (sgl-project#43598)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* [Model] Keep gpt-oss router logits batch-invariant under deterministic inference (sgl-project#43624)

* [Dsv4.1] Mega-mhc integration (sgl-project#42808)

Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>

* [diffusion] fix: accept request-level profiling on the image API (sgl-project#43573)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] feat: declare regional compile regions for flux (sgl-project#43575)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] feat: bind each GPU worker to its GPU's NUMA node (sgl-project#43565)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] feat: warn instead of disabling on unvalidated models for BCG(sgl-project#43568)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] feat: keep the TP all-reduce inside the graph (sgl-project#43602)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] fix: keep exported buffers out of expandable segments for IPC all-to-all(sgl-project#43584)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] fix: skip QKV packing while layers are offloaded (sgl-project#43576)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] perf: add the dynamic LoRA delta inside the second GEMM (sgl-project#43501)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] CI: skip the LTX-2 RMSNorm-modulation guard test off the CUDA platform (sgl-project#43574)

Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] fix: skip inductor's saved multi-kernel pick for torch.compile (Z-Image warm starts crash or hang) (sgl-project#43601)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] comfyui: fail early instead of silently dropping features in SGLD integrated mode (sgl-project#43495)

Co-authored-by: OnePunchMonk <aggarwal.avaya@yahoo.com>

* [Fix] Reserve remaining tokens for ignore_eos requests during prefill admission (sgl-project#43564)

* [DSA] Support TileLang FP8 KV cache on CUDA (sgl-project#42957)

Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>

* fix: wait for HiCache load-back before hybrid MLA KV reads (sgl-project#38217)

Co-authored-by: zhouzechun <zhouzechun@kuaishou.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>

* [diffusion] fix: keep the overlay probe from stalling cached startup (sgl-project#43590)

* [diffusion] fix: serve whole graphs after offload-during-compile (sgl-project#43631)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] Support LLaDA-Image BF16 generation and editing (sgl-project#37907)

* [PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll (sgl-project#43562)

Co-authored-by: Kevin Flansburg <kflansburg@cloudflare.com>

* [diffusion] perf: keep Qwen-Image-2.1 and Ulysses graphs whole (sgl-project#43586)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] perf: keep Cosmos3's fused kernels in the graph (sgl-project#43577)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] fix: Qwen-Image BCG prompt padding misses every graph and attends the pad (sgl-project#43632)

Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] fix: replay Qwen-Image-2.1 BCG graphs across prompt lengths (sgl-project#43633)

Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] perf: run the attention section eager under torch.compile for MiniMax-H3 (sgl-project#43634)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [1/N][DSv41] Enable DP attention with MegaMoE (sgl-project#43228)

* [diffusion] comfyui: consolidate FluxAdapter correctness fixes (sgl-project#43490)

Co-authored-by: niehen6174 <1639206518@qq.com>

* [diffusion] comfyui: consolidate executor conditioning-cache fixes (sgl-project#43492)

Co-authored-by: niehen6174 <1639206518@qq.com>

* [Scheduler] Keep waiting-prefix LRU refresh consistent across TP ranks (sgl-project#43479)

* [Perf] Fuse BF16 writes to native HND KV cache storage (sgl-project#43237)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [diffusion] Don't report video jobs completed after their outputs are deleted (sgl-project#43084)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>

* [GDN] Fix CuTe DSL decode to read the [N, HV, V, K] state layout (sgl-project#43332)

Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>

* [KDA] Support Helion MTP verification and harden prefill, decode (sgl-project#42857)

* [AMD]Qwen add Sparse attention to capturing hip graphs (sgl-project#43383)

* [Spec] Allow Qwen3.5 backends to opt into packed target verification (sgl-project#43037)

* [qwen 3.8 next] Fuse QSA prefill KV gathering and dtype conversion (sgl-project#41933)

* [Scheduler] Requeue preempted requests when admission admits nothing (sgl-project#43298)

* [Test] Do not hang scripted-runtime teardown when the server died (sgl-project#43285)

* [KV canary] Expect sweep counters on the target runner only (sgl-project#43286)

* [Scheduler] Hold back decode rows on a mixed step when they do not fit the KV pool (sgl-project#43290)

* [Sampler] Sort only the max(top_k) leading columns in the torch sampler (sgl-project#43346)

* [Scheduler] Keep a DP-converted decode batch as the running batch (sgl-project#43283)

* [kvevents] fix missing topics in replayed kv events (sgl-project#43386)

Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>

* [DeepSeek-V4.1] Use num_stages=1 for the hc_mix stats partial kernel (sgl-project#43070)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yuan Luo <4908075+yuan-luo@users.noreply.github.com>

* [VLM] fix: repair multimodal VMM lease cleanup and source-stream ordering (sgl-project#43475)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* [Diffusion] Add ModelOpt FP8 checkpoint quantization for Pi0.5 (sgl-project#43062)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* fix: default Inkling to fa4 on Hopper (sgl-project#35292)

* [BugFix][DP] Prevent health checks from perturbing user request routing (sgl-project#35721)

Signed-off-by: Phoenix3334 <Phoenix3334@users.noreply.github.com>

* [Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking (sgl-project#43567)

* [diffusion] perf: encode LTX video/audio through the shared bounded MP4 path (sgl-project#43656)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* [HiCache] Keep storage workers alive on backend exceptions (sgl-project#43550)

* Build Mamba host pools from device buffer infos (sgl-project#43596)

* Share MLA host capacity initialization (sgl-project#43595)

* Preserve serialized page dtype in NIXL copy fallback (sgl-project#43593)

* [diffusion] fix: select VSA-H3 for FastH3 when no transformer backend is given (sgl-project#43690)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [KV Events] Publish LoRA adapter names and namespaced block hashes (sgl-project#42455)

Signed-off-by: Banruo Liu <lichangxinlivingshade@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>

* [DSpark] Run every markov head through one step interface (sgl-project#43620)

* [DSpark] Add `MarkovSampleStep`, one op for a markov draft step's add and token pick (sgl-project#43621)

* [DSpark] Draft the eager markov block through `MarkovSampleStep` (sgl-project#43622)

* [DSpark] Run the same markov block in the draft graph tail; drop `SGLANG_DSPARK_FOLDED_SAMPLING` (sgl-project#43668)

* [DSpark] Merge the draft TP sync sites into `dspark-draft` (sgl-project#43669)

* [DSpark] Add `SGLANG_DSPARK_USE_GREEDY_DRAFT`, an argmax draft accepted as one-hot (sgl-project#43670)

* [DSpark] Apply `SGLANG_SIMULATE_ACC_LEN` inside the accept instead of switching paths (sgl-project#43623)

* [diffusion] fix: shard Qwen-Image adaLN modulation and text FFN under TP (sgl-project#43691)

* [diffusion] perf: layerwise offload for the LTX-2.5 Gemma-4 text encoder (sgl-project#43683)

* [diffusion] perf: split and batch LTX-2.3 one-stage guidance passes under CFG parallel (sgl-project#43684)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* [Fix] Initialize BF16 GEMM dispatch in the one-batch benchmark (sgl-project#43702)

* Fix Mistral-native format detection when the Hub is unreachable (sgl-project#43446)

* [MoE] Skip block-padding rows in the TMA down activation and quant (sgl-project#42212)

* [MemCache] Split the SWA mapping move out of locked-full recovery as `RecoverEvictedSWA` (sgl-project#43721)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>

* [Rust] Lock hyper 1.12.0 and want 0.3.2 in the rust workspace (sgl-project#43724)

Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>

* [MemCache] Add shared component size and path APIs to the tree core interface (sgl-project#43722)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>

* [CI] Fix TestQwenImageModulationCache after Qwen-Image modulation sharding (sgl-project#43747)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Bugfix] Preserve available host memory for integrated MUSA devices (sgl-project#43398)

Signed-off-by: Xucheng Zhou <aden1350@outlook.com>

* [Tokenizer] Require fastokens>=0.3.2 and include it in the all extra (sgl-project#43104)

* [AMD][V4.1][*/N] Fix DP attention + TP MoE decode NaN and support bounded replay (sgl-project#43077)

Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>

* Support kv cache sharding for DSA indexer (sgl-project#40925)

* [AMD][MORI] Add MORI EP V2 support (sgl-project#40666)

* [PD] Retract transferred decode requests on a retract pause (sgl-project#43757)

* [DSV4.1] Reduce eager Engram extend preparation (sgl-project#42159)

Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Test] Re-anchor the VLM flashinfer throughput floor (sgl-project#43762)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* [Fix] scheduler: stop latching batch_is_full when nothing can run (sgl-project#41872)

Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>

* [RL] Return a failed weight update when the cache flush is refused (sgl-project#43758)

* [CI] Skip eval retries in the PD transfer-failure test (sgl-project#43769)

* docs: sync LMSYS SGLang blog cards (sgl-project#43462)

Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>

* [Deps] Bump FlashInfer to 0.7.1 (sgl-project#43494)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* [XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast (sgl-project#42048)

* [Fix] Adapt to transformers 5.18 / 5.19 changes (sgl-project#43688)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* [diffusion][xpu] FLUX.2: enable the bit-exact Triton SwiGLU on XPU (sgl-project#41940)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* docs: clarify prefill CP terminology and simplify wording (sgl-project#43787)

* [Fix] Link cuda_ipc JIT against CUDA driver stubs dir (sgl-project#43764)

* [Perf] Route BF16 mixed-batch decode tails through decode attention (sgl-project#43238)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>

* [PD] Poison the deferred abort ack when a Mooncake send fails (sgl-project#43719)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* [HiCache] Support Mamba in buffer-only host memory mode (sgl-project#36345)

* [HiCache] Preserve all MTP KV and recurrent state (sgl-project#40223)

Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>

* [CI] Move tests that do not need GB300 off the per-PR GB300 suite (sgl-project#43784)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* Capture KV host transfer buffers (sgl-project#43711)

* Keep NIXL Mamba checkpoint keys separate by TP rank (sgl-project#43594)

* [XPU] Support decode context parallelism (DCP) on Intel XPU (sgl-project#34355)

Co-authored-by: Siju Samuel <siju.samuel@intel.com>

* [BCG][DSV4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs (sgl-project#38603)

* [DSV4.1][PD][1/N] Support prefill node use cp parallel strategy (sgl-project#40323)

* [Docs] Update GLM-5.3-Flash playground: B300 default, CP8, EP and PD (sgl-project#43802)

* [Docs] Cookbook: refresh SemiAnalysis AgentX launches (GLM-5.2 B300, DeepSeek-V4.1 GB300) (sgl-project#43799)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [NPU] Stabilize HiCache and registered CI tests (sgl-project#43308)

* [sgl-router] Serve the Anthropic Messages API on /v1/messages (sgl-project#42316)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>

* [Fix] Make tokenizer request-state initialization atomic (sgl-project#41911)

Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>

* [XPU] Upgrade XPU backend to PyTorch 2.14 (sgl-project#42003)

Co-authored-by: Ashwini Rathi <arathi@habana.ai>
Co-authored-by: Y <ashwini.rathi@intel.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* [PD][Spec] Give a PD decode's first EAGLE draft step a proposal distribution (sgl-project#41783)

Co-authored-by: Shangming Cai <csmthu@gmail.com>

* [Fix] Adapt NPU Qwen-VL processor patch to transformers 5.19 resize() signature change (sgl-project#43309)

* [NPU] Diffusion CI fixes (sgl-project#43210)

* [diffusion] chore: bind worker CPUs to the GPU's NUMA node when memory policy is not permitted (sgl-project#43756)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [CI][Diffusion] Add caching for npu GT images in CI (sgl-project#37426)

Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>

* [QSA] Bound the decode indexer's work by the context length (sgl-project#43463)

Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>

* Prebuild fixup_zero_kv JIT module at TRTLLM-MLA backend init (sgl-project#37525)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* [Unified Cache] Abort requests whose external linker load fails instead of crashing the scheduler (sgl-project#40896)

* [DSpark] Share draft params, logits dtype, confidence and the accept tail between graph and eager (sgl-project#43790)

* [DFlash] Decide the Domino candidate pool in one place and apply simulated acceptance inside the accept (sgl-project#43791)

* Fix autograd retention in CUDA graph capture tail hooks (sgl-project#43854)

* [ROCm] Eliminate HiCache migration stalls and use exact-size host allocations (sgl-project#42553)

Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>
Co-authored-by: JohnQinAMD <yanyuan.qin@amd.com>

* [AMD] Keep the FP4 varctx schedule test off AITER's 2048-row compile (sgl-project#41793)

* [AMD] Add flydsl mxfp8 dense gemm kernel for dpsk v4.1 prefill (sgl-project#43803)

* [AMD] Fix test_compact_attention_hip.py CI (sgl-project#43836)

* [AMD] [Docker] Update MoRI to v1.2.4 (sgl-project#43834)

* [Kernel] Expose bundled FlashMLA MegaAttention kernels and APIs (sgl-project#43918)

* [DeepSeek-V4.1] Admit the NIXL transfer backend for DSpark PD (sgl-project#43734)

* [AMD] Recalibrate test_quark_mxfp4 GSM8K thresholds to observed MI355X noise (sgl-project#41980)

* [DSV4][CP] Write low-ratio page indices when CP prefill runs paged attention (sgl-project#43935)

* [Build][CI] Drop Python 3.10: require Python 3.11+ and run CUDA CI on Python 3.12 (sgl-project#43021)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>

* [DeepSeek-V4.1] mHC: drop seam_open, narrow precomputed stats in take_rows (sgl-project#43833)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [DSv4.1] Reduce sparse-prefill workspace padding (sgl-project#43921)

* [HiCache] Avoid full-store scans for uncached file prefix lookups (sgl-project#42077)

Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>

* [Fix] Preserve HTTP errors for non-streaming /v1/messages (sgl-project#43796)

Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>

* [DSv4.1] Temporarily disable B300 PD/CP accuracy test (sgl-project#43948)

* [PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode (sgl-project#43936)

* [Fix] Account for captured inputs in DeepGEMM masked workspace budget (sgl-project#43563)

* [AMD] Update v4 amd cookbook 1012 (sgl-project#43962)

* [CI] Remove Mistral Small 4 e2e test (sgl-project#43973)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* Reuse Hugging Face LFM2-MoE configuration (sgl-project#43939)

Co-authored-by: Codex <codex@openai.com>

* fix(openai): honor tool_choice=none in DeepSeek encoders (sgl-project#43006)

Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>

* [diffusion] comfyui: run MiniMax H3, FastH3 and VDN in SGLD integrated mode (sgl-project#43504)

* [diffusion] fix: keep MiniMax-H3 VAE fused norms off ROCm (sgl-project#43350)

* [diffusion] fix: avoid redundant LoRA scaling while preserving exact numerics (sgl-project#43028)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* tokenizer perf optimization (sgl-project#39771)

Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>

* [diffusion] perf: implement paint_use_remesh (decimate to 40k faces before UV unwrap) for Hunyuan3D (sgl-project#42770)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* [AMD][Kimi-K3] Use public FlyDSL APIs in KDA decode kernels (sgl-project#43968)

Co-authored-by: wunhuang <wunhuang@amd.com>

* [AMD] Fix Flux2 kv-gather abort on non-contiguous K/V (sgl-project#43793)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* [AMD] Fuse prefill index-Q RoPE fake-quant and fp4 quantize kernels in DSV4.1 (sgl-project#42911)

* [AMD][DI][CI] dsv4pro-fp8 2p1d-ep16-mtp: raise prefill mem fraction for MORI EP v2 memory (sgl-project#43944)

* [router] Fix power_of_two with --dp-aware (sgl-project#42027)

* [ROCm][Spec] Route NGRAM tree reconstruction to the JIT kernel on HIP (sgl-project#43328)

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>

* [PD] Support heterogeneous TP for hybrid SWA KV transfer (sgl-project#40909)

* [Diffusion] Speed up LTX-2 guidance and video output (sgl-project#43667)

* [AMD][PD] enable mori transfer for PD decode DCP (sgl-project#37358)

Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>

* [Docs] Cookbook: refresh SemiAnalysis AgentX launches (DeepSeek-V4.1 GB200/GB300, GLM-5.2 FP8 B300) (sgl-project#43983)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids (sgl-project#43723)

Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>

* [router] power_of_two: in-flight request load for --dp-aware decode (sgl-project#43483)

* [Perf] Enable fast-math for JIT activations on Blackwell (sgl-project#43236)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe (sgl-project#40772)

Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>

* [HiCache][PD] Allow DSA index-K elision under HiCache and disaggregation (sgl-project#41769)

Co-authored-by: Anish Shanbhag <ashanbhag@nvidia.com>

* [CI] Use sgl-eval for GLM-5.3-Flash GSM8K tests (sgl-project#43975)

Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* [FlashInfer] Add startup autotuning for MegaMoE (sgl-project#43789)

* [Perf] Compact GDN causal-conv grids for skewed prefill batches (sgl-project#43570)

* [AMD][CI] Run test_tracing.py in the extra tier, matching CUDA (sgl-project#43928)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Fix EVS for NemotronH_Omni_Reasoning_V3 (sgl-project#42933)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>

* fix(multimodal): convert grayscale tensors to RGB in PIL backend (sgl-project#42127)

Co-authored-by: 周云昊 <zhouyunhao@zhouyunhaodeMacBook-Pro.local>

* [diffusion] comfyui: run Qwen-Image 2.1 in SGLD integrated mode (sgl-project#43581)

* [diffusion] feat: add configurable reference image resizing for MiniMax H3 Ref2VA (sgl-project#35839)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>

* Fix/compressed tensors nvfp4 hopper (sgl-project#40022)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* [NPU] [DOC] update kimi k3 cookbook and supported features on NPU (sgl-project#44052)

* [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.3-Flash with BF16 KV Cache (sgl-project#41790)

Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>

* [CI] Run test_pi05_modelopt_fp8_cuda in the kernel-unit lane (sgl-project#44055)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(openai): omit absent tool_calls from chat responses (sgl-project#43027)

Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>

* Skip fully-masked KV tiles in sliding-window extend attention (sgl-project#39769)

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(openai): keep chat stream created timestamp stable (sgl-project#43026)

Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>

* fix(openai): preserve zero cached tokens when cache reporting is enabled (sgl-project#43005)

Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>

* [AMD][DI][CI] glm52 2p1d-ep16: capture CUDA graph on decode (eager decode is too slow) (sgl-project#43977)

* [AMD][DI][CI] kimik26 2p1d-ep16: capture CUDA graph on decode (eager decode is slow) (sgl-project#43985)

* [perf] [multi-lora] remove empty CTAs from fused LoRA expand and add split-K to shrink (sgl-project#41951)

Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>

* Rainj me/rust bench (sgl-project#43950)

* [Observability] Report per-request storage prefetch and staging waits in meta_info (sgl-project#43946)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>

* [lora] Support LoRA for the NemotronH_Nano_VL_V2 model family (sgl-project#39798)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>

* [Rust] Refuse an unsupported multimodal model before the Rust listener binds (sgl-project#43925)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>

* [Core] Keep request priority within the signed 64-bit range in radix caches (sgl-project#43920)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: qandrew <qandrew@users.noreply.github.com>

* [DCP] Decode context parallelism for DSA (GLM-5.2 / GLM-5.3) with MTP (sgl-project#43735)

Co-authored-by: vincentzed <207368749+vincentzed@users.noreply.github.com>

* [HiCache] Block-group pool hits, caller-owned load destinations, allocation-target FULL eviction (sgl-project#43868)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>

* [Rust] Let an embedding package wrap the frontend HTTP app (sgl-project#43924)

Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>

* [Core] Add request-level customized_info keys and mark Full prefill capture batches (sgl-project#43772)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>

* [HiCache] Extract the buffer-mode prefetch and load-back lifecycle into pipeline operations (sgl-project#43870)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>

* [unified memory pool] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode (sgl-project#43919)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>

* [Rust] Serve any `model` name on the OpenAI chat and completions routes (sgl-project#43923)

Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>

* docs: add MiniCPM-V 4.7 cookbook (sgl-project#43829)

* [Feature] Add Aaronson-Gumbel text watermarking (sgl-project#37577)

Co-authored-by: Nihar Patel <nihpat95@gmail.com>

* [HiSparse] Add speculative decoding support with multi-step KV cache swapping (sgl-project#37771)

* [gRPC] Move runtime.v1 bindings to tonic 0.14, report OpenAI status codes, return text on request (sgl-project#42673)

Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [DeepSeek-V4.1] Run decoder SWA bounded replay inside the breakable prefill CUDA graph (sgl-project#43736)

* fix(benchmark): request token IDs when counting OpenAI prompts (sgl-project#34511)

Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>

* Mark FP32 redux asm volatile (sgl-project#44137)

* [Score] Send stacked GPU embedding overrides through a CUDA IPC pool (sgl-project#39730)

Co-authored-by: Sundara Raman Ramachandran <sundar24295@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>

* [Feature] Return output top logprobs as flat base64 arrays on request (sgl-project#42890)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [DSV4.1] Support decoder SWA bounded replay with CUDA DP attention (sgl-project#43812)

* GLM5.2 DSA Indexer Fallback (sgl-project#32947)

Co-authored-by: Shu Wang <shuw@login-hecate03.hecate.clusters.nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>

* [Fix] Work around a CPython fork deadlock in the CPU preprocess pool (sgl-project#44143)

* [DSV4.1] Skip Mega mHC fusion on 0-row (DP-idle) batches (sgl-project#43993)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [Chore] release: Cherry-pick [Disagg] Return prompt routed experts from the PD prefill server to release/v0.5.22 (sgl-project#44002) (sgl-project#44212)

Co-authored-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>

* [Chore] release: Cherry-pick [CI] glm5.2: Temporarily disable PD HiSparse+MTP B300 test to release/v0.5.22 (sgl-project#44192) (sgl-project#44214)

Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>

* [XPU] Fix FP8 KV cache support for ModelOpt quantized models (e.g. Llama-3.3-70B-FP8)

* [XPU] Add MXFP4 W4A16 MoE inference support for Intel Xe2 (GPT-OSS)

Enable MXFP4 quantized inference on XPU using the native Xe2 SYCL
fused_experts kernel (sgl-kernel-xpu PR#184).

- quantization/__init__.py: register mxfp4 quantization method for XPU
- quantization/fp4_utils.py: broaden flashinfer import exception to avoid
  crash on XPU where flashinfer is unavailable
- quantization/mxfp4.py:
  - align intermediate size to 64 for XPU (create_weights)
  - skip CPU dequant on XPU; leave weights as packed uint8 in VRAM
    (process_weights_after_loading)
  - add XPU dispatch in apply(): call sgl_kernel.fused_experts with
    use_mxfp4_w4a16=True; convert UE8M0 uint8 scales to float32 via
    exp2(uint8 - 127); default gemm1_alpha=1.702 / gemm1_limit=7.0 to
    select activation_type=2 (ACT_SWIGLU_GPT_OSS, interleaved weight
    layout required by GPT-OSS MXFP4 checkpoints)
- models/gpt_oss.py: replace hardcoded .cuda() with .to(get_device())
  in weight loading to support XPU

* [XPU] Improvement for fp4 and fp8

 Optimize FP8 KV cache: lazy mirror buffer to avoid full cast per step
 Fix MXFP4 MoE: read activation/gemm1 params from model config instead of GPT-OSS hardcoded values

* Remove WA for FP8 KV cache

* Add analyze support for XPU

* [Intel][XPU] Allow mamba extra_buffer radix cache strategy on XPU (#9)

Co-authored-by: root <root@gnrsp-bmg3.iind.intel.com>

---------

Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Signed-off-by: Phoenix3334 <Phoenix3334@users.noreply.github.com>
Signed-off-by: Banruo Liu <lichangxinlivingshade@gmail.com>
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com>
Signed-off-by: apex-mochen <2756823972@qq.com>
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Raymond Huang <103474329+Raymondlol@users.noreply.github.com>
Co-authored-by: Frederik Gossen <frgossen@meta.com>
Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: cctry <csycfl@gmail.com>
Co-authored-by: tao <yuanyitao@pku.edu.cn>
Co-authored-by: Richard Wang <wangrichard08@gmail.com>
Co-authored-by: mohbasit <mohbasit@amd.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: chuyeh <chuyeh@amd.com>
Co-authored-by: siliangchen-amd <SiLiang.Chen@amd.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: Sage <80211083+sagearc@users.noreply.github.com>
Co-authored-by: Bingxu Chen <bingxche@amd.com>
Co-authored-by: colleen-tml <colleend@thinkingmachines.ai>
Co-authored-by: colleen-tml <296620403+colleen-tml@users.noreply.github.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
Co-authored-by: hhy-seven <33918672+hhy-seven@users.noreply.github.com>
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com>
Co-authored-by: Shunkangz <182541032+Shunkangz@users.noreply.github.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local>
Co-authored-by: Mick <mickqian@radixark.ai>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
Co-authored-by: OnePunchMonk <aggarwal.avaya@yahoo.com>
Co-authored-by: Jzz1943 <jiangzhongze.jzz@ant-intl.com>
Co-authored-by: LoCinder <61399298+LoCinder@users.noreply.github.com>
Co-authored-by: zhouzechun <zhouzechun@kuaishou.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Kevin Flansburg <kflansburg@cloudflare.com>
Co-authored-by: Avaya Aggarwal <119044997+OnePunchMonk@users.noreply.github.com>
Co-authored-by: niehen6174 <1639206518@qq.com>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: SunskyXH <SunskyXH@gmail.com>
Co-authored-by: Nikolai <nprotaso@amd.com>
Co-authored-by: paulzhang-tm <paulzhang@thinkingmachines.ai>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: jiaqiang-dot-liu <jiaqiang.liu@amd.com>
Co-authored-by: Ziyi Xu <76582120+DarkSharpness@users.noreply.github.com>
Co-authored-by: Yuan Luo <4908075+yuan-luo@users.noreply.github.com>
Co-authored-by: trebladev <54441939+trebladev@users.noreply.github.com>
Co-authored-by: FeiHong Guo <1187883824@qq.com>
Co-authored-by: Banruo Liu <lichangxinlivingshade@gmail.com>
Co-authored-by: Alison Shao <54658187+alisonshao@users.noreply.github.com>
Co-authored-by: WangHemeng <47976301+wanghemeng@users.noreply.github.com>
Co-authored-by: metamergebot <metamergebot@gmail.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: adenzhou1350 <aden1350@outlook.com>
Co-authored-by: weireweire <weiliangl@nvidia.com>
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com>
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: ChengYM <Yueming.Cheng@amd.com>
Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: sglang-bot <sglangbot@gmail.com>
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
Co-authored-by: ashwini rathi <ashwini.rathi@intel.com>
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Anupa Sajikumar <anupa.sajikumar@intel.com>
Co-authored-by: Siju Samuel <siju.samuel@intel.com>
Co-authored-by: BingjiaWang <wangbingjia.wbj@ant-intl.com>
Co-authored-by: Jinyan Yi <yijinyan1@huawei.com>
Co-authored-by: ybyang <10629930+whybeyoung@users.noreply.github.com>
Co-authored-by: C <3121985758@qq.com>
Co-authored-by: Pavan Sivaram Girijala <pavan.sivaram.girijala@intel.com>
Co-authored-by: Ashwini Rathi <arathi@habana.ai>
Co-authored-by: Sasha Sidorov <asidorov566@gmail.com>
Co-authored-by: ming_wang <68357922+sigama-w@users.noreply.github.com>
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>
Co-authored-by: Alexandr <117110413+Allor-maker@users.noreply.github.com>
Co-authored-by: Yuan Luo <yuan.luo@hotmail.com>
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
Co-authored-by: ix-48 <90045377+ix-48@users.noreply.github.com>
Co-authored-by: Zhang, Jiejing <280536569+jiejingzhangamd@users.noreply.github.com>
Co-authored-by: JohnQinAMD <yanyuan.qin@amd.com>
Co-authored-by: Michael <13900043+michaelzhang-ai@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
Co-authored-by: Yizheng Jiao <jyizheng@gmail.com>
Co-authored-by: dingdangmao <2756823972@qq.com>
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
Co-authored-by: Vegetog <110553275+Vegetog@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Piotr Mazurek <piotr635@gmail.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: ai-jz <156989844+ai-jz@users.noreply.github.com>
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com>
Co-authored-by: Bingoo <33573610+BingooYang@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
Co-authored-by: Xinyi Song <xinyisong0111@gmail.com>
Co-authored-by: Zhaoyi Li <36555117+Lzy17@users.noreply.github.com>
Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Michal Shalev <mshalev@nvidia.com>
Co-authored-by: AMD-yanfeiwang <yanfwang@amd.com>
Co-authored-by: Shu Wang <shuw@nvidia.com>
Co-authored-by: Anish Shanbhag <ashanbhag@nvidia.com>
Co-authored-by: Ziang Li <ziangli@umich.edu>
Co-authored-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Yunhao Zhou <131600287+yyh-sjtu@users.noreply.github.com>
Co-authored-by: 周云昊 <zhouyunhao@zhouyunhaodeMacBook-Pro.local>
Co-authored-by: LittleNyima <62497818+LittleNyima@users.noreply.github.com>
Co-authored-by: jlqibm <87678480+jlqibm@users.noreply.github.com>
Co-authored-by: amote-i <49533125+amote-i@users.noreply.github.com>
Co-authored-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
Co-authored-by: Kailash Ranganathan <kailash@modal.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
Co-authored-by: Rain Jiang <96632942+rainj-me@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: qandrew <qandrew@users.noreply.github.com>
Co-authored-by: vincentzed <207368749+vincentzed@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
Co-authored-by: Nihar Patel <nihpat95@gmail.com>
Co-authored-by: Kan Wu <wukanustc@gmail.com>
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Jiaqi Li <ljq.lijiaqi@outlook.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: Andrew Briand <atb8888@comcast.net>
Co-authored-by: Chuanrui Zhu <47195419+chuanrui@users.noreply.github.com>
Co-authored-by: Sundara Raman Ramachandran <sundar24295@gmail.com>
Co-authored-by: HEJIAN SANG <sanghj0923@gmail.com>
Co-authored-by: Shu Wang <shuw@login-hecate03.hecate.clusters.nvidia.com>
Co-authored-by: shyeh25 <206795756+shyeh25@users.noreply.github.com>
Co-authored-by: Kangyan-Zhou <zky314343421@gmail.com>
Co-authored-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Spurthi Lokeshappa <spurthi.lokeshappa@intel.com>
Co-authored-by: root <root@gnrsp-bmg3.iind.intel.com>
"args": {
"--cuda-graph-max-bs-decode": ["64"],
"--data-parallel-size": ["2"],
"--enable-dp-attention": [],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--enable-dp-attention is being deprecated. Please use --attn-dp-size instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants