Repository navigation
[AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 - #39273
Merged
HaiShaw merged 108 commits intoOct 1, 2026
Merged
Conversation
hdt98
force-pushed
the
feature/optimize-glm53-flash-frontier
branch
from
September 13, 2026 07:26
5fd4387 to
0ea3a69
Compare
hdt98
force-pushed
the
feature/optimize-glm53-flash-frontier
branch
from
September 13, 2026 10:36
0ea3a69 to
3626f2d
Compare
hdt98
marked this pull request as ready for review
September 13, 2026 10:55
hdt98
requested review from
Fridge003,
Qiaolin-Yu,
Ying1123,
alphabetc1,
ch-wan,
fzyzcjy,
hanming-lu,
hnyls2002,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
September 13, 2026 10:55
3 of 5 tasks
3 of 5 tasks
This was referenced Oct 7, 2026
| tool.function.strict for tool in effective_tools | ||
| # Only tool-bearing requests get the full-assistant EBNF: its terminal | ||
| # state finishes a request even under ignore_eos. | ||
| glm_constraint = ( |
Contributor
There was a problem hiding this comment.
this fix the ignore eos issue.
2 tasks done
This was referenced Oct 8, 2026
7 of 12 tasks
4 of 5 tasks
This was referenced Oct 14, 2026
bvolpato
added a commit
to bvolpato/sglang
that referenced
this pull request
Oct 15, 2026
On a multimodal model, the Jinja chat path decodes the whole prompt ids back to text and keeps the result in the chat template cache, for every request. Since sgl-project#39273, a text-only chat completion on these models is sent as token ids (`_can_reuse_text_only_prompt_ids`), so the decoded text is not used. An agent sends a new conversation on every turn, so the cache does not hit, and each entry keeps the whole prompt. Use the cache and the eager decode only for requests with media. A text-only request that is still sent as text (Responses API, Moss-VL, tokenizers that add special tokens on encode) is decoded in `_engine_prompt`, which gives the same text as before.
3 of 4 tasks
dzorlu
pushed a commit
to dzorlu/sglang
that referenced
this pull request
Oct 16, 2026
…requests (sgl-project#39273) The serving_chat.py hunk of upstream sgl-project#39273 (f17f770, 2026-10-01): a glm47 request without tools no longer gets the full-assistant EBNF, whose terminal state finishes a request even under ignore_eos. The rest of that PR (AMD FP8/MXFP4 kernels, docs) is not taken. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 05840db)
dzorlu
pushed a commit
to dzorlu/sglang
that referenced
this pull request
Oct 17, 2026
…requests (sgl-project#39273) The serving_chat.py hunk of upstream sgl-project#39273 (f17f770, 2026-10-01): a glm47 request without tools no longer gets the full-assistant EBNF, whose terminal state finishes a request even under ignore_eos. The rest of that PR (AMD FP8/MXFP4 kernels, docs) is not taken. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 05840db) (cherry picked from commit fa0ea2c)
bvolpato
added a commit
to bvolpato/sglang
that referenced
this pull request
Oct 18, 2026
On a multimodal model, the Jinja chat path decodes the whole prompt ids back to text and keeps the result in the chat template cache, for every request. Since sgl-project#39273, a text-only chat completion on these models is sent as token ids (`_can_reuse_text_only_prompt_ids`), so the decoded text is not used. An agent sends a new conversation on every turn, so the cache does not hit, and each entry keeps the whole prompt. Use the cache and the eager decode only for requests with media. A text-only request that is still sent as text (Responses API, Moss-VL, tokenizers that add special tokens on encode) is decoded in `_engine_prompt`, which gives the same text as before.
fy1214
added a commit
to fy1214/sglang
that referenced
this pull request
Oct 19, 2026
Backport the sgl-project#39273 kpool overflow path: when the threshold bin has more than SMEM_INPUT_SIZE candidates, rescan with unique 64-bit score+index keys instead of clipping into shared memory and selecting the wrong set. Normal rows keep the existing fast path with no extra work.
3 of 4 tasks
weireweire
added a commit
to weireweire/sglang
that referenced
this pull request
Nov 9, 2026
Conflict resolutions: - base_attn_backend.py, eagle_draft_extend_cuda_graph_runner.py: take main. Main's draft-extend runner already gates the shared-read-done publish on the backend's shared_read_ends (sgl-project#39273). DSV4 declares UNKNOWN for draft extend, so no pre-replay event is published and the scheduler falls back to the whole-forward fence; the WAR ordering part of this PR is obsolete. - deepseek_v4_backend.py: take main, then re-apply the padding fix in expand_extend_with_same_length (clamp causal lengths to >= 1). Drop the trtllm-only clamp in init_forward_metadata_draft_extend, now redundant. - test_dsv4_draft_extend_cuda_graph.py: keep only the padding regression case; the event-order case targeted the removed runner helper. Validation (GB300, sglang nightly cu13 20261106 image): - test_dsv4_draft_extend_cuda_graph.py: passes on this merge, fails on plain main with [4, 5, 6, 7, -2, -1, 0, 1]. - Whole-file unit tests pass, same as plain main: test_eagle_draft_extend_cuda_graph_runner, test_eagle_draft_cuda_graph_runner, test_dsv41_draft_metadata_glue, test_dsv41_target_verify_metadata_glue, test_decode_cuda_graph_shared_read_fence, test_early_verify_metadata, test_forked_draft_metadata_prep, test_prefill_shared_read_done, test_encoder_swa_replay, test_dsv4_cutedsl_h16_dispatch, test_kv_translate_ownership, test_dsv41_late_layer_tail_readers, test_ragged_verify_backend_capability. - test_deepseek_v4.py: 24 passed, 2 failed; the same 2 fail on plain main (ImportError: get_eager_max_batch_size from sglang.srt.utils.common in deepseek_v4_mega_backend.py). - pre-commit on changed files: passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Serve the official GLM-5.3-Flash FP8 checkpoint and
OneNexus/GLM-5.3-Flash-MXFP4on MI355X (gfx950) at full speed and correct output. The target stack is zero-RoPE sparse attention, FP8 KV cache, and graph-enabled EAGLE 5/1/6.mainnow includes all ten Day-0 enablement PRs (#38545, #38546, #38547, #39317, #39338, #39339, #39340, #39341, #39778, #39779). This PR adds the remaining correctness fixes and optimizations for serving Flash on gfx950. On the same four MI355X GPUs and AgentX replay, it measures +14.3% geomean output throughput overmainat c1–c32, with higher P90 interactivity at every rung. At c48,mainfalls off the frontier, while this PR holds it.The four CI and test-hygiene fixes that used to be here now live in #40721.
What changed
Runtime paths are relative to
python/sglang/; tests are excluded.kernels/jit/csrc/dsa/kpool_topk_transform.cuhsrt/layers/attention/dsa/dsa_indexer_kpool.pysrt/layers/attention/dsa/kpool_fp8_index.py_TOPK_ROW_CHUNK). Each row's top-k depends only on that row, so results are identical on every backend; only the scratch peak changes.srt/layers/attention/dsa/dsa_topk_backend.pysrt/layers/attention/graph_variants.pymainthe FP8 MHA prefix read takes the outer backend and fails withforward_metadata is Noneon the first image request (reproduced on MI355X).resolve_attn_backendnow unwraps to the full-attention child. AITER extend also prefers the per-batchqo_indptr.srt/models/deepseek_common/attention_forward_methods/forward_mha.pysrt/layers/attention/aiter_backend.pysrt/managers/schedule_batch.pysrt/managers/scheduler_components/batch_result_processor.pysrt/mem_cache/{base_prefix_cache,common,memory_pool}.pysrt/mem_cache/unified_cache/components/mamba.pysrt/managers/schedule_policy.pysrt/model_executor/runner/decode_cuda_graph_runner.pysrt/speculative/eagle_draft_extend_cuda_graph_runner.pysrt/speculative/eagle_worker_v2.pysrt/speculative/eagle_draft_cuda_graph_runner.pykernels/ops/attention/fla/fused_kda_conv_recurrent_verify.pysrt/layers/attention/linear/kda_backend.pyignore_eos.srt/entrypoints/openai/serving_chat.pyignoreis materialized once. Quarkweight_scalemaps toweight_scale_invonly when the name is missing.srt/layers/quantization/quark/{quark,utils}.pysrt/models/glm5_next.pyConv2dLayer's unfold + linear path, which avoids the MIOpen search.srt/models/glm5_next.pytree_speculative_sampling_target_onlyaccepts any GPU device (kDLGPU), and a comment in the AITER MoE runner is corrected.kernels/jit/csrc/speculative/sampling.cuhsrt/layers/moe/moe_runner/aiter.pydocs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdxdocs/src/snippets/configs/zai-org/glm-5.3-flash.jsxdocs/src/snippets/_playground.jsxFootprint against
main: 26 production files (+361 / −80), 14 test files (+563 / −21), 3 docs files (+179 / −26).Serving performance
Setup: four MI355X VFs, TP4/EP4, FP8 E4M3 KV, EAGLE 5/1/6, 393-entry AgentX corpus, 900 s per rung, image
lmsysorg/sglang-rocm:v0.5.20-rocm724-mi35x-20260918. The client and server are pinned to the same NUMA placement, andmainand this PR ran back to back on the same GPUs.Serving command (identical for both arms; only the source differs)
Transformers is pinned to
e4052f55for the GLM-5.3-Flash configuration classes.maintok/smainmainP90 tok/s/userGeomean output throughput: +14.3% over
mainat c1–c32. P90 tok/s/user is the 90th percentile of each request's reciprocal mean inter-token latency.Checkpoint and hardware context
These replays were measured at
5fd4387a75, before the Day-0 merges; this PR's head matches that source within run-to-run noise on the same AgentX ladder.Against the official
zai-org/GLM-5.3-FlashFP8 checkpoint on the same four-GPU MI355X serving policy, OneNexus MXFP4 measures +12.4% geomean output throughput and +18.2% geomean P90 per-user rate over c1–c48, with zero request errors in both runs. This compares complete checkpoints, not an isolated quantization kernel.Against
amd/GLM-5.3-Flash-Quark-MXFP4under the identical serving setup, OneNexus is within 0.1% geomean output throughput and +1.0% geomean P90 per-user rate over c1–c48 (9,223 vs 9,264 requests, zero errors in both).OneNexus vs AMD MXFP4, per rung
The next plot places the measured MI355X MXFP4 frontier next to supplied four-GPU B300 estimates. The B300 runs were not reproduced here, and the DSA backend and serving installation differ, so this is context rather than a controlled cross-hardware claim.
The matched serving protocol has per-rung cache, capacity and acceptance data.
Measured operator optimizations
KDA fused verify. At batch 8 with eight verification tokens, the reference graph launches convolution and recurrence separately (38.9 μs). The single fused kernel with BV4 tiles is slower (41.7 μs), because it repeats the Q/K convolution across 32 value tiles. BV16 cuts the grid to 8 tiles and the paired time to 28.4 μs, 27.0% less operator time. For the six-token EAGLE 5/1/6 shape: 31.50 → 23.02 μs at batch 8 and 37.66 → 33.03 μs at batch 16. On this PR's head, the tuning still measures about 26% faster at batch 8 on top of #39524, with bit-exact output.
This is an isolated MI350X VF (gfx950) study with synthetic GLM TP4-shaped tensors, not model throughput. See the study and full data.
Vision merge. The 2×2 downsample has a non-overlapping kernel and stride, no padding and one group, so each patch is a linear projection. Reusing
Conv2dLayer's unfold + linear path skips the MIOpen search. On the model's actual[256, 1024, 2, 2]merge layout, first-call time falls 4.50 → 3.31 s (−26.5%) and warmed graph time 63.28 → 25.62 μs (−59.5%), with identical output. See the operator study.The AITER mHC dispatch and its caller-applied RMSNorm fix, measured in the same study, landed in
mainvia #38545.Accuracy
On the serving configuration above (sgl-eval 0.1.2, 500 examples, reasoning effort max, temperature 0): GSM8K 0.978, MMLU 0.870.
Checkpoint quality against
amd/GLM-5.3-Flash-Quark-MXFP4, on identical pinned samples with one frozen runtime, graphs and speculation off, BF16 KV, greedy decoding, and truncations kept in the denominator:OneNexus/GLM-5.3-Flash-MXFP4amd/GLM-5.3-Flash-Quark-MXFP4The MMLU (+1.0 point) and GPQA Diamond (+3.5 points) differences do not reach
p < 0.05; see the quality protocol and paired statistics.Validation
main's Mamba checkpoint and radix lock-ref tests pass (1,626 passed, 0 failed), including the native k-pool top-k and KDA bit-exactness kernels.Checklist
main, plus accuracy on the serving configuration.CI States
Latest PR Test (Base): ❌ Run #36822051443
Latest PR Test (Extra): ✅ Run #36822051500
Latest PR Test (AMD ROCm 10): ❌ Run #36822051511