Repository navigation
[DSA] Support group-scaled FP8 NoPE KV reads for CUDA TileLang - #39349
Draft
HanHan009527 wants to merge 4 commits into
Draft
HanHan009527 wants to merge 4 commits into
HanHan009527 wants to merge 4 commits into
Conversation
Extend the existing full and paged readers to 528-byte rows, bound sparse dequantization by the smaller of the physical pool and selected rows, and connect the BF16 TileLang consumer. Permit only matching TileLang phases with BF16 latent512/RoPE0 queries and DCP size 1 through the production backend validation entry point. Extract the reader and consumer slice from the GLM integration at abd9742 onto upstream main ad5af53. Retain HIP backend constraints and the existing 656-byte format. Add scale, index mapping, graph replay, consumer, and startup validation coverage. Review scope: invalid-index attention masking, variable KPool width, cached-prefix dispatch, and full GLM PD serving validation remain separate dependencies. No standalone performance gain is claimed.
Place the test under kernel/attention and propagate pytest exit status when invoked by the registered test runner.
21 of 71 tasks
This was referenced Nov 7, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Enable CUDA TileLang DSA to consume group528 FP8 NoPE KV rows for GLM-5.3-Flash. Related to #36830.
Modifications
Accuracy Tests
Validated commit
223a88446464on H20, 2026-09-20:Speed Tests and Profiling
GLM-5.3-Flash, 4×H20, TP4/EP4, same clean wheel/model; only KV dtype differs. Static-memory fraction 0.92, automatic KV capacity, context 262144, max-running 32, Mamba slots 64, chunk 8192, decode Graph 32; MTP/radix off. Each workload generates 1024 output tokens.
At 8.524 GiB KV allocation per GPU in both arms, capacity increases from 719,680 to 1,260,544 tokens (+75.15%). Observed simultaneous long requests increase 10→18 at 64K/C32 and 5→8 at 128K/C8.
Changes are FP8 relative to BF16: higher throughput is better; lower TTFT/TPOT is better. TTFT and TPOT below are means; TPOT includes prefill-induced stalls, not just decode-kernel time.
Result: more usable capacity, not a material throughput improvement. At capacity-constrained points, observed P95 completion improves 11–21% and mean TTFT improves 13–20%, while mean completion worsens 3–8% and mean TPOT worsens 57–70%.
Single-run, two-wave burst measurements (16/32/64/16 requests), not warmed steady-state or an SLO guarantee. No fatal OOM or request retractions; BF16 had recoverable allocator-OOM warnings. This compares against BF16, not existing raw512 FP8. PD/MTP/prefix, broader quality, invalid/all-empty attention masks and unfused KPool widths remain unvalidated.
Checklist
Developed with AI assistance.
CI States
Latest PR Test (Base): ❌ Run #35497208819
Latest PR Test (Extra): ❌ Run #35497208693
Latest PR Test (AMD ROCm 10): ❌ Run #35497208759