Repository navigation
feat(glm52 mono): serve a BF16-attention checkpoint - #3266
Open
XiaobingSuper wants to merge 1 commit into
Open
XiaobingSuper wants to merge 1 commit into
XiaobingSuper wants to merge 1 commit into
Conversation
GLM-5.2 mono read attention only as PTPC FP8, so amd/GLM-5.2-MXFP4 served without online_quant_config refused before building and ran the production path. Mono now reads the attention storage production loaded. On a BF16 checkpoint q_a/kv_a, q_b and W_o are read in place, row-major, and the absorbed W_UK / W_UV keep production's per-tensor FP8 with group-128 FP8 activations (AttentionWeight.BF16_ABSORB_FP8). The format is taken from the loaded linears, agreed over TP and named in the kernel; such a deployment runs the external indexer. At S=8 the 32 MB BF16 q_a/kv_a weight is read in one sample tile instead of two. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
valar-cop
reviewed
Nov 12, 2026
valar-cop
left a comment
valar-cop
left a comment
There was a problem hiding this comment.
a few small things..
per-width numerics for the tp4 bf16 s=6/10/12 mtp builds? gsm8k on/off and per-layer deviation
ci green.
| self._refused.add(rows) | ||
| logger.warning("GLM-5.2 MonoKernel fallback before launch: %s", error) | ||
| return False | ||
| if self._attention is None: |
There was a problem hiding this comment.
the disagree path is never tested, only the agree one. add a test that stubs group_max/group_min to disagree and asserts rows is refused?
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ready for review. No blocking issues.
GLM-5.2 mono read attention only as PTPC FP8, so
amd/GLM-5.2-MXFP4served as shipped (BF16 attention, noonline_quant_config) refused every width and ran the production path. This serves it.Dev record
BF16mode would pack a copy of every projection, 7.4 GB a GPU at TP4.AttentionWeight.BF16_ABSORB_FP8reads the BF16 projections in place (unit_bf16_rows) with production's FP8 W_UK / W_UV. The format comes from the loaded linears (attention_format), agreed over TP, named in the kernel. Mixed formats andATOM_MLA_ABSORB_BF16=1are refused; the fused indexer stays PTPC-only.Named result
Mono on vs off at its fastest MTP depth, 1K/1K, golden acceptance, tok/s:
Gates
tests/models/glm52,tests/test_native_monokernel.py175 passed; black, ruff clean.Evidence
Output tok/s, mean of 2 runs, 1K/1K random,
--spec-decode-acceptance-length2.99 / 3.33 / 3.61 for MTP3 / 4 / 5, recipe flags (kv_cache_dtype fp8,index_cache_dtype fp4,block-size 64,max-num-seqs 2, graphs[1,2]), MI355X.Mono at MTP4 / MTP5 is slower than at MTP3 (TP4: 290.3 / 291.7 at C1, 310.8 / 370.0 at C2): each verify row adds about 9 experts' weights.
With an fp8 index cache and #3265 (DSA indexer mono on the indexers), mono off at its fastest depth (MTP5): TP4 323.0 / 475.4 at C1 / C2 against 268.5 / 469.5; TP8 410.1 / 607.7 against 281.1 / 527.1.
Mono layer time, TP4 (server trace, rank 0):
The BF16 cost over PTPC is the extra ~50 MB a layer at HBM rate. Rejected after a single-layer A/B: prefetching W_o into the MALL or into registers before the uv tasks, deferring the q_b prefetch, wave-0 pre-polls, BF16 W_UK / W_UV, a packed copy of the projections (2% faster, +7.4 GB a GPU).
Compile (gfx950, external indexer): every BF16 build at TP4 / TP8, S = 1..12, fits LDS; scratch only at TP4 S=6 (12 B), 10 (144 B), 12 (24 B).
Per layer against production on the same input (eager, 16 GSM8K questions, mean over 75 layers): attention output 2.52% (PTPC mono 2.61%), layer output 5.65% (5.70%).
🤖 Generated with Claude Code