Repository navigation
[Perf] [AMD] GLM5.3: Run KDA decode and HIP prefill on a BF16 state pool on gfx950 - #49036
Open
Raiden-Makoto wants to merge 2 commits into
Open
Raiden-Makoto wants to merge 2 commits into
Raiden-Makoto wants to merge 2 commits into
Conversation
Raiden-Makoto
added a commit
to Raiden-Makoto/squidward
that referenced
this pull request
Nov 7, 2026
Add a gfx950 row-streaming packed KDA decode kernel and route validated safe-gate KDA geometry through it. Unsupported hardware, layouts, dtypes, dimensions, and indices retain the existing decode path. (cherry picked from commit 6b1ca08)
Raiden-Makoto
force-pushed
the
RM/glm53-kda-bf16-state-pr
branch
from
November 11, 2026 18:51
1a9cc0a to
36505a3
Compare
Raiden-Makoto
marked this pull request as ready for review
November 11, 2026 18:51
Raiden-Makoto
requested review from
hanming-lu,
hebiao064,
yizhang2077 and
yuan-luo
as code owners
November 11, 2026 18:51
Jacob0226
reviewed
Nov 13, 2026
Raiden-Makoto
force-pushed
the
RM/glm53-kda-bf16-state-pr
branch
from
November 13, 2026 16:04
36505a3 to
a064751
Compare
Raiden-Makoto
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH and
celve
as code owners
November 13, 2026 16:04
Raiden-Makoto
added a commit
to Raiden-Makoto/squidward
that referenced
this pull request
Nov 13, 2026
Collaborator
|
/tag-and-rerun-ci |
This was referenced Nov 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
GLM-5.3-Flash keeps the recurrent state of its 34 KDA layers in FP32 (
mamba_ssm_dtyperesolves tofloat32). Decode and MTP verify are bound by reading and rewriting that state, and the state pool takes GPU memory that the KV cache and prefix cache could use.--mamba-ssm-dtype bfloat16already exists, but on gfx950 both fast paths reject a BF16 pool: the row-streaming packed decode (#44659) and the HIP chunked prefill fall back to the generic Triton kernels. This change runs both on a BF16 state pool. All recurrence math stays FP32; only the stored state is BF16.Depends on #44659, which adds the row-streaming decode kernel. This branch carries that commit; the change under review is a06475153e.
Modifications
chunk_kdaacross every recurrence path, tracking snapshots, and no Triton fallback fromTritonKDAKernel.extend.Enable with:
Accuracy Tests
Focused validation passed 15 gfx950 tests and 105 subtests on upstream main
56e38dfe0fwith #44659 and this change.Full GLM-5.3-Flash GSM8K on snapshot
b5688f25491202978c19c4d036eef579f61bbe07: 1,319 examples, thinking enabled, temperature 1.0, top-p 0.95, sampling seed unset.--mamba-ssm-dtype bfloat16--mamba-ssm-dtype bfloat16TP4 used a 65,536-token context, 64 evaluator threads, and 32,768 maximum output tokens. TP8 used a 131,072-token context, 1,200 evaluator threads, and 4,096 maximum output tokens.
Speed Tests and Profiling
Environment:
gfx950)rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261102amd/GLM-5.3-Flash-Quark-MXFP4, snapshotb5688f25491202978c19c4d036eef579f61bbe073015e3639(285579b62with GLM-5.3 MXFP4 MoE changes)a8e0045395(GLM-5.3 integration branch with this change)Decode state update latency
One KDA decode step per layer, CUDA-graph replay. FP32 is the current row-streaming path; BF16 is this change.
Every enabled BF16 row-streaming cell beat the generic BF16 kernel, 0.7-29%, across all decode CUDA-graph batch sizes.
Prefill latency
One KDA layer, HIP chunked prefill. The last column is a BF16 pool without this change, which falls back to Triton
chunk_kda.The BF16 pool runs at FP32 speed and removes the 2.3-4.3x Triton fallback. The end-to-end and agentic results below were measured with an earlier revision of this change that copied the BF16 state to FP32 around the same kernel (21-33 µs per call).
MTP verify latency
KDA target verify, 6 draft tokens per request, chain, CUDA-graph replay. The verify kernel already supports a BF16 pool; this is the effect of the state dtype.
With MTP on TP4 GSM8K, the mean accept length was 2.875 with the FP32 state and 2.809 with the BF16 state (-2.3%).
End-to-end serving
GLM-5.3-Flash snapshot
b5688f25491202978c19c4d036eef579f61bbe07, TP4, 65,536 context, random 8K input / 1K output. Both arms use sourcea8e0045395and differ only by--mamba-ssm-dtype bfloat16.A. FP32 state — dtype unset:
B. BF16 state —
--mamba-ssm-dtype bfloat16(delta versus FP32 state):The BF16 state improves ITL by 0.99-4.89%, E2E latency by 0.85-2.92%, and output throughput by 0.87-2.97% across the complete TP4 matrix; TTFT is within 0.62%.
TP8 end-to-end serving
GLM-5.3-Flash TP8, 131,072 context, random 8K input / 1K output. Both arms use source
a8e0045395and differ only by--mamba-ssm-dtype bfloat16.A. FP32 state — dtype unset:
B. BF16 state —
--mamba-ssm-dtype bfloat16(delta versus FP32 state):The BF16 state improves TP8 ITL by 0.55-2.97%, E2E latency by 0.42-1.68%, and output throughput by 0.43-1.70% across the complete matrix; TTFT is within 1.13%.
Agentic serving
InferenceX AgentX agentic benchmark, TP4, MTP speculative decoding, concurrency 64, 3,600 s profiling. Both arms in each table differ only by the KDA state dtype.
With HiCache,
write_through, TileLang DSA, measured acceptance length:--mamba-ssm-dtype bfloat16With HiCache,
write_back(--hicache-io-backend kernel --hicache-mem-layout page_first), 216 GB/rank, Triton DSA, simulated acceptance length 3.56, with #48112; mean of two runs per arm:--mamba-ssm-dtype bfloat16Every arm passed the InferenceX health check with 0 crashes, 0 OOM warnings, and 0.0% errors. The BF16 state nearly doubles the state-pool slots in about the same memory (841 to 1,630 slots at
write_back) and raises device KV capacity from 12,351,744 to 13,031,168 tokens. Withwrite_throughit also raises the server prefix-cache hit rate from 87.2% to 89.5%; withwrite_backthe hit rate is already 95.7% for FP32 and 96.4% for BF16. Repeatedwrite_backruns agree within 1.1% on throughput and 2.7 points on P90 interactivity.Checklist
--mamba-ssm-dtypealready exists.Scoped pre-commit (isort, Ruff, Ruff format, codespell, registered-test validation) passes.
CI States
Latest PR Test (Base): ❌ Run #43459044132
Latest PR Test (Extra): ❌ Run #43459043462
Latest PR Test (AMD ROCm 10): ❌ Run #43459043590
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: