Skip to content

[AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 - #39273

Merged
HaiShaw merged 108 commits into
sgl-project:mainfrom
hdt98:feature/optimize-glm53-flash-frontier
Oct 1, 2026
Merged

HaiShaw merged 108 commits into
sgl-project:mainfrom
hdt98:feature/optimize-glm53-flash-frontier

Conversation

@hdt98

@hdt98 hdt98 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Serve the official GLM-5.3-Flash FP8 checkpoint and OneNexus/GLM-5.3-Flash-MXFP4 on MI355X (gfx950) at full speed and correct output. The target stack is zero-RoPE sparse attention, FP8 KV cache, and graph-enabled EAGLE 5/1/6.

main now includes all ten Day-0 enablement PRs (#38545, #38546, #38547, #39317, #39338, #39339, #39340, #39341, #39778, #39779). This PR adds the remaining correctness fixes and optimizations for serving Flash on gfx950. On the same four MI355X GPUs and AgentX replay, it measures +14.3% geomean output throughput over main at c1–c32, with higher P90 interactivity at every rung. At c48, main falls off the frontier, while this PR holds it.

The four CI and test-hygiene fixes that used to be here now live in #40721.

What changed

Runtime paths are relative to python/sglang/; tests are excluded.

Area Kind Change Files
K-pool top-k Correctness (CUDA and HIP) A threshold bin holding more than 4,096 candidates is rescanned with unique score/index keys, so ties or late higher scores are never clipped. The ordinary fast path is unchanged. HIP sets the dynamic shared-memory attribute. kernels/jit/csrc/dsa/kpool_topk_transform.cuh
K-pool scoring memory Correctness · HIP The logits row budget counts AITER's 256-column row padding, so a chunk cannot exceed AITER's 2 GiB limit, and page indices are promoted to int64 before offsetting. srt/layers/attention/dsa/dsa_indexer_kpool.py
srt/layers/attention/dsa/kpool_fp8_index.py
Unfused top-k memory Correctness · all backends Ragged batches above 4,096 rows run in 4,096-row chunks (_TOPK_ROW_CHUNK). Each row's top-k depends only on that row, so results are identical on every backend; only the scratch peak changes. srt/layers/attention/dsa/dsa_topk_backend.py
Graph variants Correctness / capture cost K-pool has no dense-skip path, so it is captured once instead of as a dense/sparse pair. Without a host length mirror, sparse is chosen without a device sync. srt/layers/attention/graph_variants.py
Attention backend lookup Correctness GLM-5.3-Flash wraps DSA inside the hybrid KDA backend, so on main the FP8 MHA prefix read takes the outer backend and fails with forward_metadata is None on the first image request (reproduced on MI355X). resolve_attn_backend now unwraps to the full-attention child. AITER extend also prefers the per-batch qo_indptr. srt/models/deepseek_common/attention_forward_methods/forward_mha.py
srt/layers/attention/aiter_backend.py
Visible-prefix / Mamba ownership Correctness A verify step can commit past the stop token; only client-visible tokens are cached. The other ping-pong slot's length is tracked, so a shorter key donates the state that actually matches it. Freed rows can no longer scatter into a reused slot. srt/managers/schedule_batch.py
srt/managers/scheduler_components/batch_result_processor.py
srt/mem_cache/{base_prefix_cache,common,memory_pool}.py
srt/mem_cache/unified_cache/components/mamba.py
Mamba chunk fill Correctness Exact chunk fill is disabled for Mamba caches, because checkpoints only land on page-aligned chunk ends. srt/managers/schedule_policy.py
Graph read fences Correctness Without an in-graph marker (HIP), the read-done event is published after replay rather than before. Draft extend follows the backend's declared read end. srt/model_executor/runner/decode_cuda_graph_runner.py
srt/speculative/eagle_draft_extend_cuda_graph_runner.py
EAGLE draft on ROCm Optimization · measured ROCm enables rejection sampling (#37134), but verify takes argmax for all-greedy batches and never reads the draft distribution. The draft now skips the full-vocab softmax, Gumbel sample and probs stack for those batches and captures a matching greedy graph variant. srt/speculative/eagle_worker_v2.py
srt/speculative/eagle_draft_cuda_graph_runner.py
KDA fused verify Correctness + optimization · HIP only On top of #39524: separately rounded conv products (bit-exact with the reference on gfx950), a one-wave default, and BV16 for measured GLM shapes. The HIP gate keys on gfx950, bf16 QKV, fp32 conv weights, T ∈ {6, 8}, and batch 1–16. kernels/ops/attention/fla/fused_kda_conv_recurrent_verify.py
srt/layers/attention/linear/kda_backend.py
Chat serving Optimization / correctness Send the token ids already rendered instead of re-tokenizing long text-only prompts. Apply the GLM47 full-assistant grammar only when tools are present, since its terminal state ends requests even under ignore_eos. srt/entrypoints/openai/serving_chat.py
Quark loading Correctness Block-FP8 is claimed only when the activation grouping matches the weight block. ignore is materialized once. Quark weight_scale maps to weight_scale_inv only when the name is missing. srt/layers/quantization/quark/{quark,utils}.py
srt/models/glm5_next.py
Vision downsample Optimization · measured The non-overlapping 2×2 merge uses Conv2dLayer's unfold + linear path, which avoids the MIOpen search. srt/models/glm5_next.py
Small ROCm fixes Correctness tree_speculative_sampling_target_only accepts any GPU device (kDLGPU), and a comment in the AITER MoE runner is corrected. kernels/jit/csrc/speculative/sampling.cuh
srt/layers/moe/moe_runner/aiter.py
MI355X cookbook Docs FP8 and MXFP4 on the GLM-5.3-Flash selector, with TP4/EP4 commands and the ROCm runtime environment used for the measurements below. docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx
docs/src/snippets/configs/zai-org/glm-5.3-flash.jsx
docs/src/snippets/_playground.jsx

Footprint against main: 26 production files (+361 / −80), 14 test files (+563 / −21), 3 docs files (+179 / −26).

Serving performance

Setup: four MI355X VFs, TP4/EP4, FP8 E4M3 KV, EAGLE 5/1/6, 393-entry AgentX corpus, 900 s per rung, image lmsysorg/sglang-rocm:v0.5.20-rocm724-mi35x-20260918. The client and server are pinned to the same NUMA placement, and main and this PR ran back to back on the same GPUs.

Serving command (identical for both arms; only the source differs)
SGLANG_USE_AITER=1 AITER_ONLINE_TUNE=0 \
SGLANG_OPT_FUSED_KDA_VERIFY=1 SGLANG_DSA_FUSE_TOPK=0 SGLANG_OPT_USE_TOPK_V2=false \
SGLANG_ENABLE_WAR_BARRIER=1 SGLANG_FORCE_COARSE_WAR_BARRIER=1 \
HIP_FORCE_DEV_KERNARG=1 ROCM_QUICK_REDUCE_QUANTIZATION=NONE \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python3 -m sglang.launch_server --model-path OneNexus/GLM-5.3-Flash-MXFP4 \
  --tp-size 4 --ep-size 4 --quantization quark --trust-remote-code \
  --context-length 262144 --page-size 64 --mem-fraction-static 0.82 \
  --max-running-requests 96 --cuda-graph-max-bs-decode 96 --min-free-slots-delay 1 \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 --prefill-decode-interval 0 \
  --dsa-prefill-backend tilelang --dsa-decode-backend tilelang \
  --linear-attn-verify-backend triton --kv-cache-dtype fp8_e4m3 --moe-runner-backend aiter \
  --speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 6 --speculative-attention-mode prefill \
  --speculative-accept-threshold-single 1.0 --speculative-accept-threshold-acc 1.0 \
  --speculative-draft-model-quantization unquant --speculative-moe-a2a-backend none \
  --enable-hierarchical-cache --hicache-ratio 0.50 --hicache-write-policy write_through \
  --hicache-io-backend direct --hicache-mem-layout page_first_direct \
  --reasoning-parser glm45 --tool-call-parser glm47 --mm-feature-transport cpu

Transformers is pinned to e4052f55 for the GLM-5.3-Flash configuration classes.

Concurrency This PR tok/s main tok/s vs main This PR P90 tok/s/user main P90 tok/s/user
1 252.9 237.9 +6.3% 374.4 365.0
2 443.2 394.6 +12.3% 337.5 326.1
4 768.7 657.2 +17.0% 293.1 287.5
8 1,253.1 1,137.4 +10.2% 237.1 226.8
16 1,849.9 1,608.5 +15.0% 172.4 157.8
32 2,304.8 1,829.2 +26.0% 111.0 94.7
48 2,411.3 off frontier — 78.1 off frontier

Geomean output throughput: +14.3% over main at c1–c32. P90 tok/s/user is the 90th percentile of each request's reciprocal mean inter-token latency.

AgentX frontier on four MI355X GPUs: this PR vs main

Checkpoint and hardware context

These replays were measured at 5fd4387a75, before the Day-0 merges; this PR's head matches that source within run-to-run noise on the same AgentX ladder.

Against the official zai-org/GLM-5.3-Flash FP8 checkpoint on the same four-GPU MI355X serving policy, OneNexus MXFP4 measures +12.4% geomean output throughput and +18.2% geomean P90 per-user rate over c1–c48, with zero request errors in both runs. This compares complete checkpoints, not an isolated quantization kernel.

Measured OneNexus MXFP4 versus official FP8 AgentX frontiers on four MI355X VFs

Against amd/GLM-5.3-Flash-Quark-MXFP4 under the identical serving setup, OneNexus is within 0.1% geomean output throughput and +1.0% geomean P90 per-user rate over c1–c48 (9,223 vs 9,264 requests, zero errors in both).

OneNexus vs AMD MXFP4, per rung
Concurrency OneNexus tok/s AMD tok/s Output delta OneNexus P90 tok/s/user AMD P90 tok/s/user P90 delta
1 261.76 258.89 +1.1% 379.08 364.96 +3.9%
2 444.88 444.09 +0.2% 333.60 330.21 +1.0%
4 764.52 769.09 −0.6% 295.25 297.03 −0.6%
8 1,283.08 1,278.38 +0.4% 241.23 237.75 +1.5%
16 1,845.35 1,857.38 −0.6% 178.88 179.39 −0.3%
32 2,279.72 2,290.16 −0.5% 110.35 110.00 +0.3%
48 2,331.22 2,340.38 −0.4% 74.85 73.89 +1.3%

The next plot places the measured MI355X MXFP4 frontier next to supplied four-GPU B300 estimates. The B300 runs were not reproduced here, and the DSA backend and serving installation differ, so this is context rather than a controlled cross-hardware claim.

Measured OneNexus MXFP4 MI355X frontier and supplied B300 estimate

The matched serving protocol has per-rung cache, capacity and acceptance data.

Measured operator optimizations

KDA fused verify. At batch 8 with eight verification tokens, the reference graph launches convolution and recurrence separately (38.9 μs). The single fused kernel with BV4 tiles is slower (41.7 μs), because it repeats the Q/K convolution across 32 value tiles. BV16 cuts the grid to 8 tiles and the paired time to 28.4 μs, 27.0% less operator time. For the six-token EAGLE 5/1/6 shape: 31.50 → 23.02 μs at batch 8 and 37.66 → 33.03 μs at batch 16. On this PR's head, the tuning still measures about 26% faster at batch 8 on top of #39524, with bit-exact output.

KDA before/after GPU traces and paired graph timings: reference, initial fusion, and tuned fusion

This is an isolated MI350X VF (gfx950) study with synthetic GLM TP4-shaped tensors, not model throughput. See the study and full data.

Vision merge. The 2×2 downsample has a non-overlapping kernel and stride, no padding and one group, so each patch is a linear projection. Reusing Conv2dLayer's unfold + linear path skips the MIOpen search. On the model's actual [256, 1024, 2, 2] merge layout, first-call time falls 4.50 → 3.31 s (−26.5%) and warmed graph time 63.28 → 25.62 μs (−59.5%), with identical output. See the operator study.

The AITER mHC dispatch and its caller-applied RMSNorm fix, measured in the same study, landed in main via #38545.

Accuracy

On the serving configuration above (sgl-eval 0.1.2, 500 examples, reasoning effort max, temperature 0): GSM8K 0.978, MMLU 0.870.

Checkpoint quality against amd/GLM-5.3-Flash-Quark-MXFP4, on identical pinned samples with one frozen runtime, graphs and speculation off, BF16 KV, greedy decoding, and truncations kept in the denominator:

Checkpoint GSM8K MMLU GPQA Diamond
OneNexus/GLM-5.3-Flash-MXFP4 490/500 428/500 139/198
amd/GLM-5.3-Flash-Quark-MXFP4 491/500 423/500 132/198

The MMLU (+1.0 point) and GPQA Diamond (+3.5 points) differences do not reach p < 0.05; see the quality protocol and paired statistics.

Validation

  • On gfx950, the 14 changed test files and main's Mamba checkpoint and radix lock-ref tests pass (1,626 passed, 0 failed), including the native k-pool top-k and KDA bit-exactness kernels.
  • Every serving boot passes the basic, 30,031-token long-context, non-greedy and image-cached follow-up probes (320 cached tokens).
  • The cookbook-generated MI355X MXFP4 command boots on this PR and passes the same probes, with GSM8K 0.980 and MMLU 0.866.
  • pre-commit passes on all changed files.

Checklist

  • Format and lint all changed files.
  • Keep tests to behavior and numerical regressions, with native gfx950 coverage.
  • Measure AgentX throughput against main, plus accuracy on the serving configuration.
  • Pass the upstream CI matrix.

CI States

Latest PR Test (Base): ❌ Run #36822051443
Latest PR Test (Extra): ✅ Run #36822051500
Latest PR Test (AMD ROCm 10): ❌ Run #36822051511

tool.function.strict for tool in effective_tools
# Only tool-bearing requests get the full-assistant EBNF: its terminal
# state finishes a request even under ignore_eos.
glm_constraint = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this fix the ignore eos issue.

@ormandj ormandj mentioned this pull request Oct 13, 2026
7 of 12 tasks
bvolpato added a commit to bvolpato/sglang that referenced this pull request Oct 15, 2026
On a multimodal model, the Jinja chat path decodes the whole prompt
ids back to text and keeps the result in the chat template cache, for
every request. Since sgl-project#39273, a text-only chat completion on these
models is sent as token ids (`_can_reuse_text_only_prompt_ids`), so
the decoded text is not used. An agent sends a new conversation on
every turn, so the cache does not hit, and each entry keeps the whole
prompt.

Use the cache and the eager decode only for requests with media. A
text-only request that is still sent as text (Responses API, Moss-VL,
tokenizers that add special tokens on encode) is decoded in
`_engine_prompt`, which gives the same text as before.
dzorlu pushed a commit to dzorlu/sglang that referenced this pull request Oct 16, 2026
…requests (sgl-project#39273)

The serving_chat.py hunk of upstream sgl-project#39273 (f17f770, 2026-10-01): a
glm47 request without tools no longer gets the full-assistant EBNF, whose
terminal state finishes a request even under ignore_eos. The rest of that
PR (AMD FP8/MXFP4 kernels, docs) is not taken.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 05840db)
dzorlu pushed a commit to dzorlu/sglang that referenced this pull request Oct 17, 2026
…requests (sgl-project#39273)

The serving_chat.py hunk of upstream sgl-project#39273 (f17f770, 2026-10-01): a
glm47 request without tools no longer gets the full-assistant EBNF, whose
terminal state finishes a request even under ignore_eos. The rest of that
PR (AMD FP8/MXFP4 kernels, docs) is not taken.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 05840db)
(cherry picked from commit fa0ea2c)
bvolpato added a commit to bvolpato/sglang that referenced this pull request Oct 18, 2026
On a multimodal model, the Jinja chat path decodes the whole prompt
ids back to text and keeps the result in the chat template cache, for
every request. Since sgl-project#39273, a text-only chat completion on these
models is sent as token ids (`_can_reuse_text_only_prompt_ids`), so
the decoded text is not used. An agent sends a new conversation on
every turn, so the cache does not hit, and each entry keeps the whole
prompt.

Use the cache and the eager decode only for requests with media. A
text-only request that is still sent as text (Responses API, Moss-VL,
tokenizers that add special tokens on encode) is decoded in
`_engine_prompt`, which gives the same text as before.
fy1214 added a commit to fy1214/sglang that referenced this pull request Oct 19, 2026
Backport the sgl-project#39273 kpool overflow path: when the threshold bin has more
than SMEM_INPUT_SIZE candidates, rescan with unique 64-bit score+index
keys instead of clipping into shared memory and selecting the wrong set.
Normal rows keep the existing fast path with no extra work.
weireweire added a commit to weireweire/sglang that referenced this pull request Nov 9, 2026
Conflict resolutions:
- base_attn_backend.py, eagle_draft_extend_cuda_graph_runner.py: take main.
  Main's draft-extend runner already gates the shared-read-done publish on
  the backend's shared_read_ends (sgl-project#39273). DSV4 declares UNKNOWN for draft
  extend, so no pre-replay event is published and the scheduler falls back
  to the whole-forward fence; the WAR ordering part of this PR is obsolete.
- deepseek_v4_backend.py: take main, then re-apply the padding fix in
  expand_extend_with_same_length (clamp causal lengths to >= 1). Drop the
  trtllm-only clamp in init_forward_metadata_draft_extend, now redundant.
- test_dsv4_draft_extend_cuda_graph.py: keep only the padding regression
  case; the event-order case targeted the removed runner helper.

Validation (GB300, sglang nightly cu13 20261106 image):
- test_dsv4_draft_extend_cuda_graph.py: passes on this merge, fails on plain
  main with [4, 5, 6, 7, -2, -1, 0, 1].
- Whole-file unit tests pass, same as plain main:
  test_eagle_draft_extend_cuda_graph_runner, test_eagle_draft_cuda_graph_runner,
  test_dsv41_draft_metadata_glue, test_dsv41_target_verify_metadata_glue,
  test_decode_cuda_graph_shared_read_fence, test_early_verify_metadata,
  test_forked_draft_metadata_prep, test_prefill_shared_read_done,
  test_encoder_swa_replay, test_dsv4_cutedsl_h16_dispatch,
  test_kv_translate_ownership, test_dsv41_late_layer_tail_readers,
  test_ragged_verify_backend_capability.
- test_deepseek_v4.py: 24 passed, 2 failed; the same 2 fail on plain main
  (ImportError: get_eager_max_batch_size from sglang.srt.utils.common in
  deepseek_v4_mega_backend.py).
- pre-commit on changed files: passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd apple-silicon deepseek documentation Improvements or additions to documentation jit-kernel memory-pool quant LLM Quantization run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) speculative-decoding unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants