Skip to content

[DSA] Support group-scaled FP8 NoPE KV reads for CUDA TileLang - #39349

Draft
HanHan009527 wants to merge 4 commits into
sgl-project:mainfrom
bytedance-iaas:codex/pr/glm53-13-group528-nope-reader-review
Draft

HanHan009527 wants to merge 4 commits into
sgl-project:mainfrom
bytedance-iaas:codex/pr/glm53-13-group528-nope-reader-review

Conversation

@HanHan009527

@HanHan009527 HanHan009527 commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Enable CUDA TileLang DSA to consume group528 FP8 NoPE KV rows for GLM-5.3-Flash. Related to #36830.

Modifications

  • Add full/paged readers for the existing 528-byte layout (512 FP8 values + four FP32 scales), using selected-row dequantization when smaller than the full pool.
  • Connect the reader to the BF16 TileLang consumer and enable the supported CUDA configuration in startup validation. Backend defaults and existing RoPE/HIP paths are preserved; this is not native FP8 attention.

Accuracy Tests

Validated commit 223a88446464 on H20, 2026-09-20:

  • 34 tests + 15 subtests passed; pre-commit passed.
  • Both BF16 and FP8: GSM8K 63/64, same incorrect question, no errors or truncations; eight short exact answers, 128K generation and 130K known-answer retrieval passed.
  • 128/128 benchmark requests succeeded per arm, with matching request bodies and exact token counts.

Speed Tests and Profiling

GLM-5.3-Flash, 4×H20, TP4/EP4, same clean wheel/model; only KV dtype differs. Static-memory fraction 0.92, automatic KV capacity, context 262144, max-running 32, Mamba slots 64, chunk 8192, decode Graph 32; MTP/radix off. Each workload generates 1024 output tokens.

At 8.524 GiB KV allocation per GPU in both arms, capacity increases from 719,680 to 1,260,544 tokens (+75.15%). Observed simultaneous long requests increase 10→18 at 64K/C32 and 5→8 at 128K/C8.

Changes are FP8 relative to BF16: higher throughput is better; lower TTFT/TPOT is better. TTFT and TPOT below are means; TPOT includes prefill-induced stalls, not just decode-kernel time.

Input / concurrency Output tok/s, BF16→FP8 Change TTFT s, BF16→FP8 Change TPOT ms, BF16→FP8 Change
64K / 8 121.83→116.52 -4.36% 26.94→27.40 +1.70% 29.57→31.72 +7.27%
64K / 16 132.96→134.62 +1.25% 64.93→51.74 -20.31% 36.91→58.04 +57.25%
64K / 32 137.37→139.79 +1.76% 154.60→134.49 -13.00% 37.28→63.28 +69.74%
128K / 8 65.04→65.15 +0.18% 71.74→57.62 -19.68% 31.82→50.36 +58.27%

Result: more usable capacity, not a material throughput improvement. At capacity-constrained points, observed P95 completion improves 11–21% and mean TTFT improves 13–20%, while mean completion worsens 3–8% and mean TPOT worsens 57–70%.

Single-run, two-wave burst measurements (16/32/64/16 requests), not warmed steady-state or an SLO guarantee. No fatal OOM or request retractions; BF16 had recoverable allocator-OOM warnings. This compares against BF16, not existing raw512 FP8. PD/MTP/prefix, broader quality, invalid/all-empty attention masks and unfused KPool widths remain unvalidated.

Checklist

  • Registered tests and pre-commit pass.
  • Paired full-model correctness and capacity/performance results provided.

Developed with AI assistance.


CI States

Latest PR Test (Base): ❌ Run #35497208819
Latest PR Test (Extra): ❌ Run #35497208693
Latest PR Test (AMD ROCm 10): ❌ Run #35497208759

Extend the existing full and paged readers to 528-byte rows, bound sparse dequantization by the smaller of the physical pool and selected rows, and connect the BF16 TileLang consumer. Permit only matching TileLang phases with BF16 latent512/RoPE0 queries and DCP size 1 through the production backend validation entry point.

Extract the reader and consumer slice from the GLM integration at abd9742 onto upstream main ad5af53. Retain HIP backend constraints and the existing 656-byte format. Add scale, index mapping, graph replay, consumer, and startup validation coverage.

Review scope: invalid-index attention masking, variable KPool width, cached-prefix dispatch, and full GLM PD serving validation remain separate dependencies. No standalone performance gain is claimed.
Place the test under kernel/attention and propagate pytest exit status when invoked by the registered test runner.
@github-actions github-actions Bot added quant LLM Quantization jit-kernel labels Sep 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant