Skip to content

[Perf] [AMD] GLM5.3: Run KDA decode and HIP prefill on a BF16 state pool on gfx950 - #49036

Open
Raiden-Makoto wants to merge 2 commits into
sgl-project:mainfrom
Raiden-Makoto:RM/glm53-kda-bf16-state-pr
Open

Raiden-Makoto wants to merge 2 commits into
sgl-project:mainfrom
Raiden-Makoto:RM/glm53-kda-bf16-state-pr

Conversation

@Raiden-Makoto

@Raiden-Makoto Raiden-Makoto commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

GLM-5.3-Flash keeps the recurrent state of its 34 KDA layers in FP32 (mamba_ssm_dtype resolves to float32). Decode and MTP verify are bound by reading and rewriting that state, and the state pool takes GPU memory that the KV cache and prefix cache could use.

--mamba-ssm-dtype bfloat16 already exists, but on gfx950 both fast paths reject a BF16 pool: the row-streaming packed decode (#44659) and the HIP chunked prefill fall back to the generic Triton kernels. This change runs both on a BF16 state pool. All recurrence math stays FP32; only the stored state is BF16.

Depends on #44659, which adds the row-streaming decode kernel. This branch carries that commit; the change under review is a06475153e.

Modifications

  • Accept a BF16 state pool in the row-streaming KDA packed decode. The kernel already loads the state into FP32 and stores it in the pool dtype.
  • Add per-head-count BF16 launch tables, measured on MI355X for 16 local heads (TP4) and 8 local heads (TP8). TP4 batches 23-31 keep the generic kernel, where it measured faster. Other head counts with a BF16 pool keep the generic kernel. The FP32 path is unchanged.
  • Run the HIP KDA chunked prefill on a BF16 pool: the recurrence kernels load the BF16 state into FP32 and store the final state as BF16 (round to nearest even). Outputs, final states and tracking snapshots are bit-identical to the FP32 kernel run on the same values and rounded to BF16, with no extra copies or allocations.
  • Extend the gfx950 tests: every BF16 decode config bucket with two seeds, padded slots and the untouched pool, 16 decode steps, the generic-kernel cells, determinism and CUDA-graph replay; BF16 prefill against the rounded FP32 kernel and against Triton chunk_kda across every recurrence path, tracking snapshots, and no Triton fallback from TritonKDAKernel.extend.

Enable with:

--mamba-ssm-dtype bfloat16

Accuracy Tests

Focused validation passed 15 gfx950 tests and 105 subtests on upstream main 56e38dfe0f with #44659 and this change.

Full GLM-5.3-Flash GSM8K on snapshot b5688f25491202978c19c4d036eef579f61bbe07: 1,319 examples, thinking enabled, temperature 1.0, top-p 0.95, sampling seed unset.

Configuration Score Stop rate Truncated rate Error rate
TP4 — --mamba-ssm-dtype bfloat16 96.89% 100.00% 0.00% 0.00%
TP8 — --mamba-ssm-dtype bfloat16 96.51% 99.47% 0.53% 0.00%

TP4 used a 65,536-token context, 64 evaluator threads, and 32,768 maximum output tokens. TP8 used a 131,072-token context, 1,200 evaluator threads, and 4,096 maximum output tokens.

Speed Tests and Profiling

Environment:

  • Hardware: AMD Instinct MI355X (gfx950)
  • Docker image: rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261102
  • Model: amd/GLM-5.3-Flash-Quark-MXFP4, snapshot b5688f25491202978c19c4d036eef579f61bbe07
  • AITER: 3015e3639 (285579b62 with GLM-5.3 MXFP4 MoE changes)
  • Validated source SHA: a8e0045395 (GLM-5.3 integration branch with this change)

Decode state update latency

One KDA decode step per layer, CUDA-graph replay. FP32 is the current row-streaming path; BF16 is this change.

TP Batch FP32 state (µs) BF16 state (µs) Delta
4 1 2.69 2.37 -12.1%
4 8 4.28 3.43 -19.9%
4 16 6.40 4.47 -30.1%
4 32 9.55 6.72 -29.7%
4 64 21.00 10.36 -50.7%
4 80 26.27 14.52 -44.7%
8 1 2.60 2.34 -10.2%
8 16 4.93 3.43 -30.4%
8 64 10.81 6.75 -37.5%
8 128 21.10 10.40 -50.7%
8 256 41.37 22.15 -46.5%
8 512 88.04 41.73 -52.6%

Every enabled BF16 row-streaming cell beat the generic BF16 kernel, 0.7-29%, across all decode CUDA-graph batch sizes.

Prefill latency

One KDA layer, HIP chunked prefill. The last column is a BF16 pool without this change, which falls back to Triton chunk_kda.

TP Sequences x tokens FP32 state (µs) BF16 state (µs) Delta BF16 before this change (µs)
4 1 x 8,192 185 181 -1.9% 599
4 1 x 16,384 317 312 -1.8% 1,153
4 2 x 8,192 350 345 -1.2% 874
4 16 x 1,024 169 168 -0.6% 714
4 4 x 8,192 540 537 -0.6% 1,456
8 1 x 8,192 126 125 -0.3% 497
8 1 x 16,384 202 201 -0.5% 863
8 2 x 8,192 244 242 -0.4% 577
8 16 x 1,024 119 119 -0.3% 352
8 4 x 8,192 358 358 -0.0% 839

The BF16 pool runs at FP32 speed and removes the 2.3-4.3x Triton fallback. The end-to-end and agentic results below were measured with an earlier revision of this change that copied the BF16 state to FP32 around the same kernel (21-33 µs per call).

MTP verify latency

KDA target verify, 6 draft tokens per request, chain, CUDA-graph replay. The verify kernel already supports a BF16 pool; this is the effect of the state dtype.

TP Requests FP32 state (µs) BF16 state (µs) Delta
4 1 8.41 7.34 -12.7%
4 16 24.25 18.05 -25.6%
4 64 93.45 42.95 -54.0%
4 80 124.95 64.06 -48.7%
8 1 7.51 6.73 -10.4%
8 16 19.38 13.40 -30.9%
8 64 41.17 26.48 -35.7%
8 80 56.81 31.34 -44.8%

With MTP on TP4 GSM8K, the mean accept length was 2.875 with the FP32 state and 2.809 with the BF16 state (-2.3%).

End-to-end serving

GLM-5.3-Flash snapshot b5688f25491202978c19c4d036eef579f61bbe07, TP4, 65,536 context, random 8K input / 1K output. Both arms use source a8e0045395 and differ only by --mamba-ssm-dtype bfloat16.

A. FP32 state — dtype unset:

Concurrency TTFT (ms) ITL (ms) E2EL (ms) Output tok/s
4 419.11 6.060 6787.83 603.19
8 658.88 6.770 7988.16 1025.42
16 1137.03 7.915 10108.17 1618.72
32 2083.36 9.380 13506.65 2425.14
64 3998.27 11.545 19573.46 3348.49

B. BF16 state — --mamba-ssm-dtype bfloat16 (delta versus FP32 state):

Concurrency TTFT (ms) Delta ITL (ms) Delta E2EL (ms) Delta Output tok/s Delta
4 421.69 +0.62% 6.000 -0.99% 6729.94 -0.85% 608.42 +0.87%
8 661.38 +0.38% 6.690 -1.18% 7908.15 -1.00% 1034.99 +0.93%
16 1140.05 +0.27% 7.750 -2.08% 9950.53 -1.56% 1644.13 +1.57%
32 2095.80 +0.60% 9.060 -3.41% 13190.01 -2.34% 2481.42 +2.32%
64 4010.26 +0.30% 10.980 -4.89% 19002.78 -2.92% 3447.99 +2.97%

The BF16 state improves ITL by 0.99-4.89%, E2E latency by 0.85-2.92%, and output throughput by 0.87-2.97% across the complete TP4 matrix; TTFT is within 0.62%.

TP8 end-to-end serving

GLM-5.3-Flash TP8, 131,072 context, random 8K input / 1K output. Both arms use source a8e0045395 and differ only by --mamba-ssm-dtype bfloat16.

A. FP32 state — dtype unset:

Concurrency TTFT (ms) ITL (ms) E2EL (ms) Output tok/s
4 363.60 5.490 6110.41 669.91
8 593.69 5.990 7074.63 1157.16
16 1046.76 6.540 8554.72 1914.13
32 1950.89 7.570 11398.20 2872.91
64 3762.66 8.915 16411.67 3992.11

B. BF16 state — --mamba-ssm-dtype bfloat16 (delta versus FP32 state):

Concurrency TTFT (ms) Delta ITL (ms) Delta E2EL (ms) Delta Output tok/s Delta
4 367.73 +1.13% 5.460 -0.55% 6084.84 -0.42% 672.80 +0.43%
8 596.39 +0.45% 5.940 -0.83% 7033.68 -0.58% 1163.79 +0.57%
16 1049.12 +0.23% 6.450 -1.38% 8459.84 -1.11% 1935.36 +1.11%
32 1951.23 +0.02% 7.400 -2.25% 11222.30 -1.54% 2916.90 +1.53%
64 3761.69 -0.03% 8.650 -2.97% 16136.69 -1.68% 4060.14 +1.70%

The BF16 state improves TP8 ITL by 0.55-2.97%, E2E latency by 0.42-1.68%, and output throughput by 0.43-1.70% across the complete matrix; TTFT is within 1.13%.

Agentic serving

InferenceX AgentX agentic benchmark, TP4, MTP speculative decoding, concurrency 64, 3,600 s profiling. Both arms in each table differ only by the KDA state dtype.

With HiCache, write_through, TileLang DSA, measured acceptance length:

KDA state Per-GPU tok/s P90 interactivity ITL p50 (ms) ITL p90 (ms) TTFT p90 (s) Profiled requests
FP32 — dtype unset 43,202 14.7 23.61 67.85 8.75 5,133
BF16 — --mamba-ssm-dtype bfloat16 49,610 18.5 20.89 53.96 7.01 5,787
Delta +14.8% +25.8% -11.5% -20.5% -19.9% +12.7%

With HiCache, write_back (--hicache-io-backend kernel --hicache-mem-layout page_first), 216 GB/rank, Triton DSA, simulated acceptance length 3.56, with #48112; mean of two runs per arm:

KDA state Per-GPU tok/s P90 interactivity ITL p50 (ms) ITL p90 (ms) TTFT p90 (s) Profiled requests
FP32 — dtype unset 81,437 48.0 12.20 20.87 1.29 8,824
BF16 — --mamba-ssm-dtype bfloat16 87,873 59.7 11.43 16.73 1.05 9,399
Delta +7.9% +24.6% -6.3% -19.8% -18.6% +6.5%

Every arm passed the InferenceX health check with 0 crashes, 0 OOM warnings, and 0.0% errors. The BF16 state nearly doubles the state-pool slots in about the same memory (841 to 1,630 slots at write_back) and raises device KV capacity from 12,351,744 to 13,031,168 tokens. With write_through it also raises the server prefix-cache hit rate from 87.2% to 89.5%; with write_back the hit rate is already 95.7% for FP32 and 96.4% for BF16. Repeated write_back runs agree within 1.1% on throughput and 2.7 points on P90 interactivity.

Checklist

Scoped pre-commit (isort, Ruff, Ruff format, codespell, registered-test validation) passes.


CI States

Latest PR Test (Base): ❌ Run #43459044132
Latest PR Test (Extra): ❌ Run #43459043462
Latest PR Test (AMD ROCm 10): ❌ Run #43459043590

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Add a gfx950 row-streaming packed KDA decode kernel and route validated safe-gate KDA geometry through it. Unsupported hardware, layouts, dtypes, dimensions, and indices retain the existing decode path.

(cherry picked from commit 6b1ca08)
@Raiden-Makoto
Raiden-Makoto force-pushed the RM/glm53-kda-bf16-state-pr branch from 1a9cc0a to 36505a3 Compare November 11, 2026 18:51
@Raiden-Makoto
Raiden-Makoto marked this pull request as ready for review November 11, 2026 18:51

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@Raiden-Makoto
Raiden-Makoto force-pushed the RM/glm53-kda-bf16-state-pr branch from 36505a3 to a064751 Compare November 13, 2026 16:04
Raiden-Makoto added a commit to Raiden-Makoto/squidward that referenced this pull request Nov 13, 2026

@Jacob0226 Jacob0226 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bingxche

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants