Skip to content

[Feature] MXFP4 KV Cache for Qwen3.5/3.8 (MHA) on SM120 - #38724

Open
Eightina wants to merge 6 commits into
sgl-project:mainfrom
Eightina:eightina/qwen3.8-kv-mxfp4
Open

Eightina wants to merge 6 commits into
sgl-project:mainfrom
Eightina:eightina/qwen3.8-kv-mxfp4

Conversation

@Eightina

@Eightina Eightina commented Sep 9, 2026 •

Copy link
Copy Markdown

Motivation

Add an OCP MXFP4 KV-cache recipe (--kv-cache-dtype mxfp4, block-32, E2M1 data + E8M0 scale) for dense MHA models on SM120 (RTX 5090 class), validated on Qwen3.5/3.8-27B. It stores KV in ~1.9× fewer bytes than FP8, for a measured +52% KV capacity (1.52×) at equal memory. End to end and self-contained in pure Triton (no sgl-kernel wheel change): codec + memory pool + a native Triton split-KV decode kernel + backend integration (FlashInfer PLAIN prefill, Triton native decode).

It also carries one general fix: checkpoint FP8 k_scale/v_scale are now applied through a single load-time gate keyed on --kv-cache-dtype, replacing scattered per-backend kv_cache_scales_valid guards.

Modifications

Standard MXFP4 layout — block-32, E2M1 packed (uint8) + one E8M0 byte (float8_e8m0fnu) per 32 values, blocked per head (never across heads); no global scale (unlike NVFP4). Per token per layer (K or V) on Qwen3.8 (4 KV heads, head_dim 256): 4×(128 packed + 8 scale) = 544 B, vs FP8 1024 B and BF16 2048 B.

Components (4 commits):

  • Scale gate (kv_cache.py): _checkpoint_kv_scales_apply decides whether the checkpoint's per-tensor FP8 scales apply, keyed on the raw --kv-cache-dtype string (fp8_e4m3/e5m2 → yes; auto → via kv_cache_quant_algo; bf16/mxfp4/nvfp4/mxfp8 → unit). One point covers compressed-tensors + ModelOpt + every backend.
  • Codec (kvfp4_tensor.py, mxfp4_quant.py): OCP quantize/dequant (saturating RNE, NaN/±Inf/zero-block policy) + a fused write kernel quant_store_kv_mxfp4 (amax → E8M0 → E2M1 pack → scatter, CUDA-graph safe); e8m0_to_f32 is built by bit-shift (avoids tl.exp2 FTZ flushing 2^-127).
  • Decode kernel (decode_attention_mxfp4_sm120.py): native Triton split-KV — per-head stage1 (MHA, fp32), grouped stage1 (GQA/MQA), stage2 split-combine; num_warps=8 / BLOCK_N=64; head_dim-agnostic (next_power_of_2, //32).
  • Serving: MXFP4KVCacheMethod + an access registry (PLAIN FlashInfer BF16 read / NATIVE_FP4 Triton inline), pool buffers + cell sizing, backend wiring, and a startup hook.

Startup guards (fail fast at arg parse): CUDA-only; MHA-only (no MLA); prefill=flashinfer with decode ∈ {flashinfer, triton}; the PLAIN decode path requires --disable-cuda-graph; the pool requires v_head_dim == head_dim. 21 files, +3097 / −36.

Validation

Checkpoint using unsloth/Qwen3.8-27B-NVFP4, FP8 (fp8_e4m3 + flashinfer) vs MXFP4 (mxfp4 + flashinfer prefill + triton decode), Qwen3.8-27B-NVFP4 on single RTX 5090, identical flags otherwise. Capacity/throughput are single-request at --mem-fraction-static 0.80 (a clean-capacity baseline, no mamba-state tuning); accuracy is the paired, truncation-excluded metric.

Metric FP8 MXFP4 Δ
KV capacity (max_total_num_tokens) 53503 81529 +52% (1.52×)
Decode throughput (single-req, 4k×128) 37.9 tok/s 35.4 tok/s −6.7%
GSM8K (1319, greedy, 4k) 98.19% 97.95% −0.24pt (McNemar p=0.450)
AIME25 (30×avg@8, T=0.6, 16k) 95.56% 96.30% +0.74pt (p=1.000)

Accuracy is statistically indistinguishable from FP8. MXFP4 inflates generation length (GSM8K mean +43%, AIME median +61% — KV noise flattens softmax concentration), raising the truncated rate (AIME 30%→40%).

Known Limitations / Follow-ups

  • SM120-only; MHA-only (no MLA); v_head_dim == head_dim required.
  • Prefill must be FlashInfer; the PLAIN decode path is validation-only (needs --disable-cuda-graph), while production decode uses the graph-safe Triton native kernel. A native Triton prefill is out of scope.
  • Decode −6.7% vs FP8 (single-request): the SM120 Triton kernel and PLAIN prefill are less optimized than the FlashInfer FP8 path.
  • The +52% capacity shrinks under concurrency: serving conc > 1 on this hybrid (GDN) model needs a mamba-state cache (--max-mamba-cache-size) that eats ~2.9 GB of the KV budget, pulling the gain down toward ~+20%.
  • Length inflation (+43–67%) discounts the effective capacity gain to roughly 1.06–1.3× for long-CoT workloads.

References

Checklist

  • Format code with pre-commit (isort clean; the black diffs on 5 files are pre-existing upstream style under a newer black — left untouched to avoid polluting this diff)
  • Add unit tests (codec / decode kernel / method + pool + scale gate; 237 CPU + 21 GPU pass)
  • Update documentation — N/A: no KV-cache-recipe doc precedent (nvfp4 / fp4_mx_block16 are undocumented), matching [Feature] MXFP4 KV Cache Decode for DSV4 on Hopper #32741
  • Accuracy and speed benchmark (Validation above; FP8 vs MXFP4, single-request capacity/throughput + paired truncation-excluded accuracy)
  • Follow the SGLang code style guidance

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

A compressed-tensors / ModelOpt checkpoint that declares an FP8 kv_cache_scheme
bakes per-tensor k_scale/v_scale onto each RadixAttention layer. They are FP8-KV
calibration factors: the write path divides K/V by them before the FP8 cast, and
the read path multiplies them back.

BaseKVCacheMethod loaded them unconditionally, so an explicit --kv-cache-dtype
override to a non-FP8 cache desynced read from write. With bf16 the K/V are
stored unscaled, yet every backend still applied the descale on read, shrinking
the attention output by ~k_scale (greedy decode drifts into garbage). Self-scaling
block formats (mxfp4/nvfp4/fp4_mx_block16) carry their own per-block scales and
must not be double-scaled either.

Gate the scales at the single load point on the runtime dtype, mirroring
load_kv_cache_scales() (which already loads only for fp8_e4m3) and
configure_kv_cache_dtype(): an explicit FP8 dtype applies them, and "auto" applies
them only when the checkpoint's kv_cache_quant_algo is FP8 -- exactly when
configure_kv_cache_dtype resolves "auto" to FP8. Otherwise (bf16, a self-scaling
block format, or "auto" under a non-FP8 algo such as an NVFP4 ModelOpt checkpoint
that still builds this method) fall back to unit scales, so every backend reads an
identity descale with no per-backend guard.

Fixes all backends at once (flashinfer/triton/fa3/flashmla/aiter read layer.k_scale
unconditionally). The fp8_e4m3/fp8_e5m2 and auto-with-FP8-scheme paths are
byte-for-byte unchanged.
Standalone MXFP4 (OCP MX v1.0 6.3) KV-cache codec: block-32 E2M1 values with one
E8M0 scale per head block, scale = 2^(floor(log2(amax)) - 2), saturating
round-to-nearest-even, and the OCP NaN / +-Inf / zero-block policy. Blocks never
span heads; a partial final block is zero-padded.

- MXFP4KVQuantizeUtil (kvfp4_tensor.py): eager pure-tensor quantize/dequantize,
  CUDA-graph capturable. Kept eager because a torch.compile'd slice + pad-to-even
  + nibble-pack graph was observed to skip the final output byte under some
  inductor versions, and the codec contract is bit-exactness.
- mxfp4_quant.py: a fused Triton kernel that quantizes and scatter-stores K/V
  (packed E2M1 data + E8M0 scales) into the paged pool in one pass -- slot 0
  reserved (CUDA-graph padding contract), no host sync. Bit-exact with the eager
  codec for bf16/fp16; gated to fall back to eager for fp32 / CPU /
  non-contiguous-last-dim / shape-mismatched inputs. Also hosts e8m0_to_f32, the
  E8M0-byte -> fp32 primitive shared with the decode-attention kernel (next
  commit), built from the fp32 bit pattern rather than tl.exp2 (whose approximate
  ex2 flushes the 2^-127 subnormal scale to zero).

Inert: no KVCacheMethod, CLI, pool or backend references these yet -- the serving
wiring lands in a later commit. Validated against an independent OCP reference
inlined in the test (shares no code with the runtime codec): codec conformance
(CPU) and fused-store bit-exactness + reserved-slot + support-gate (CUDA).
Native MXFP4 KV-cache decode attention for MHA/GQA/MQA that reads the packed
E2M1 + E8M0 pool buffers directly and dequantizes inline -- no BF16 PLAIN
materialization, so the KV bandwidth saving is realized on the memory-bound
decode path. Portable Triton, developed and validated on SM120 (Blackwell).

- decode_attention_mxfp4_sm120.py: a two-stage split-KV decode kernel mirroring
  the stock decode_attention.py structure. A per-head stage1 serves MHA in fp32;
  a grouped stage1 serves a whole GQA/MQA query group per program, unpacking the
  shared K/V once and using tl.dot over bf16 tiles (lossless for dequantized
  E2M1). E8M0 scales expand via the bit-exact e8m0_to_f32 (imported from the
  codec commit) and E2M1xE8M0 -> bf16 via pure-integer bit construction, avoiding
  the approximate ex2 FTZ that would zero the 2^-127 scale. stage2 merges splits.

Inert: no backend dispatches to it yet. The triton_backend wiring -- including
the num_warps=8 the grouped kernel needs at bs>=4 / long seq (4 warps spills the
BLOCK_N dequant tiles and runs ~15x slower) -- lands in the serving commit.

Test lives in test/registered/kernels/ops/attention/ (mirroring the kernel
source), with the pure-Torch decode golden inlined per the kernel-test convention
(sibling kernel tests are self-contained; the attention_methods test-kit is for
backend-fixture unittests only). The golden dequantizes the same packed bytes with
the production eager codec and runs explicit fp32 GQA attention, never calling the
Triton kernel. Validated on SM120 within frozen per-path thresholds (MHA
rel_l2 <= 3.5e-7; GQA/MQA rel_l2 <= 1.85e-3) over MHA/GQA/MQA, page sizes 1-128,
contiguous/shuffled/interleaved/non-monotonic slot layouts, a long multi-split
sequence and an all-zero request, plus CUDA-graph capture/replay byte-determinism
for the decode and fused-write kernels.
… decode + flashinfer PLAIN)

Add --kv-cache-dtype mxfp4 end to end. MXFP4KVCacheMethod stores strict OCP
block-32 E2M1 data + E8M0 scales through a fused Triton write kernel and reads
them back as BF16 (PLAIN); the KV pool allocates the packed/scale buffers and
sizes cells accordingly, with a v_head_dim == head_dim guard.

Serving wiring: the triton backend gains a native MXFP4 decode path that reads
the packed E2M1 + E8M0 scales inline through the MXFP4 decode kernel
(num_warps=8, kv-splits floor 32), while flashinfer resolves the KV dtype it
presents from the attention access (PLAIN -> BF16) for prefill and the
validation-only decode. A server-arg hook rejects unsupported combinations at
parse time: CUDA-only, MHA-only, prefill=flashinfer with decode in
{flashinfer, triton}, and PLAIN decode without --disable-cuda-graph.

No per-backend scale-validity guards are carried over: the checkpoint FP8-scale
loading gate already reduces layer.k_scale/v_scale to identity for mxfp4 (and
every non per-tensor-FP8 cache), so both backends read scales unconditionally.

Tests: MXFP4 method contract and the checkpoint-scale gate (CPU), pool
write/PLAIN-read/slot-move and checkpoint-scale-ignore (CUDA), hook validation
(CPU), and an SM120 end-to-end serving test driving the shared basic decode
correctness probes over flashinfer PLAIN prefill + native triton decode (CUDA).
@Eightina
Eightina force-pushed the eightina/qwen3.8-kv-mxfp4 branch from cb0100b to b2ada37 Compare September 9, 2026 19:51
@Eightina

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@Eightina

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2ada378e1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

This comment has been minimized.

detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…mily + S-EXT1 + riders + S-RACE + LLGUARD/ENVREG), S-EXT1 and S-RACE items CLOSED, entry-256's raw-env deviation and LL-vs-PCIe-IPC collision warning both marked CLOSED with the BOOT_DELTA (h') needle in landing terms, sgl-project#35617 left as the sole FAM-A7 survivor, sgl-project#44732 re-pointed to the operator's merge-sgl-project#5 answer, and the KV-ROUTE DECISION RECORD added (HOLD NVFP4-KV until armed PCIe-IPC is measured post-merge; preferred-if-ever sequence merge -> sgl-project#43356-delta -> sgl-project#46452 NOT sgl-project#43240 -> optional sgl-project#43471 -> fork BUILD-A QSA FP4 reads (template sgl-project#38724) -> BUILD-B L3 scale sidecar -> one GPU window; sgl-project#46141 template, sgl-project#44462 NEXT, MXFP4 declined, entry-247 re-sync watch-string modelopt_e4m3_k32)
The MXFP4 native decode path bought KV capacity at the cost of latency:
at 98k context it was 17% slower per output token than the flashinfer
FP8 baseline. Server-side profiling (profile_by_stage, DECODE trace) put
weight GEMMs at 75-84% of step time and identical in absolute terms on
both sides, leaving the grouped stage-1 attention kernel as the only
differing term: 0.324 vs 0.130 ms per layer-step, 2.5x.

Dequantize E2M1 x E8M0 through a 4096-entry LUT indexed by
(scale_byte << 4) | code instead of reconstructing each element
arithmetically. bf16 is exact for these values -- a 2-bit mantissa times
a power-of-two scale always fits an 8-bit significand. The table is
built eagerly when the KV pool is created, since a capture-time build
would be baked into the CUDA graph.

Nibble-interleave the K/V tiles. Lane j of the low half holds logical
element 2j and lane j of the high half holds 2j+1, so a single
packed-byte load and a single scale load feed both nibbles, replacing
element-wise addressing by offs_d // 2 that read every byte twice and
needed a select to pick the nibble. tl.dot is invariant under
permutation of the contraction dim, so Q is loaded in the same even/odd
order; for V the permutation lands on the output dim and is undone by
two strided stores. This halves the register-resident staging path and
drops a full-width uint8 intermediate: 255 regs + 16 spills -> 178 regs
and no spills, which in turn admits 4 warps and two CTAs per SM.

Derive the kv-split floor from a whole number of CTA waves rather than a
hardcoded 32. Stage 1 is register-bound, so exceeding the
SMs * CTAs-per-SM grid by one CTA buys a second wave that the whole
kernel waits on -- 84 splits (336 CTAs) run 0.120 ms where 88 (352) take
0.186. The device-side schedule in get_num_kv_splits_triton inflates the
core count by log2(seq_len/64), which suits the lighter bf16 CTAs but
overshoots here, so the bound has to come from this cap.
stage1_launch_config() publishes the measured occupancy so the backend
schedule cannot silently desync from a retuned kernel.

Grouped stage-1 on SM120 (170 SMs), bs=1, 24Q/4KV, head_dim 256, under
CUDA graph:

              100k ctx   150k ctx
    before     0.344 ms   0.515 ms
    after      0.120 ms   0.171 ms   (-65%, -67%)

End to end, streaming ITL p50 at 98k context: 29.14 -> 24.65 ms against
24.92 ms for FP8, so 1.1% ahead of the baseline while still holding
1.524x the KV capacity (160,109 vs 105,071 tokens). Doubling the splits
grows the attn_logits scratch from 49.5 to 100 MB, which fits the
existing headroom.

Accuracy is unchanged against the frozen references: grouped rel_l2 is
1.155e-03 under a 1.85e-03 cap, the residual being kernel accumulation
order only since both sides consume identical dequantized values.
@Eightina
Eightina force-pushed the eightina/qwen3.8-kv-mxfp4 branch from 9f55083 to 6657f08 Compare October 29, 2026 14:10
@github-actions

This comment has been minimized.

Stage 2 merges the per-split partials of the MXFP4 decode path. Its cost
tracked MAX_KV_SPLITS alone -- 85/127/170/255 splits cost
0.0159/0.0235/0.0297/0.0435 ms regardless of context length or CTA count,
a fixed ~187 ns (~520 cycles) per split. That is the signature of the
loop-carried e_max -> old_scale -> acc chain paying a full load latency every
iteration, and it explains why num_stages had no effect whatsoever: both
loads sat inside `if split_kv_end > split_kv_start`, and the pipeliner cannot
hoist loads out of a data-dependent branch.

Replacing the branch with predication lets the pipeliner run several splits
ahead. An out-of-range split now contributes old_scale == 1 and
exp_logic == 0, leaving acc, e_sum and e_max untouched, which is exactly what
the branch did. The tl.where on old_scale is load-bearing: if the first split
is empty, e_max - n_e_max is -inf - -inf, and a bare exp would push NaN into
the accumulator.

Predication on its own is a small loss (at num_stages=2, 0.0159 -> 0.0178);
the win comes entirely from the pipelining it unlocks, so the two go
together. Cold-L2 sweep at 100k, bs=1, 24 heads, 85 splits:

  num_stages   1       2       3       4       6       8
  stage 2      0.0147  0.0178  0.0097  0.0080  0.0083  0.0096

Stage 2, cold L2 (stage 1 measured identical on both sides):

  seq        before   after
   32,768    0.0149   0.0080   -46.3%
  100,000    0.0159   0.0080   -49.7%
  150,000    0.0161   0.0083   -48.4%

Whole decode attention against the flashinfer FP8 KV baseline, both run
through the same cold-L2 harness (bs=1, 24Q/4KV, head_dim 256, page_size 1):

  seq        FP8      MXFP4
    8,192    0.0313   0.0283    -9.4%
   32,768    0.0684   0.0609   -10.9%
   65,536    0.1209   0.0965   -20.2%
  100,000    0.1804   0.1351   -25.1%
  150,000    0.2665   0.1891   -29.0%

The two paths are structurally identical, so at 100k the merge halves compare
directly: flashinfer's PersistentVariableLengthMergeStatesKernel costs
0.0088 ms against our 0.0079, where before this change we were at 0.0159.

Numerically this is a no-op rather than a tolerance change. Over 14 shapes
(seq in {32, 64, 100, 1000, 32768, 100000, 150000} x max_splits in {8, 85})
the output is bitwise identical to the branched version under an int32 view,
including seq=32 with max_splits=85 where 84 of the 85 splits are empty --
the exact path this patch rewrites. Unit tests pass (4 tests + 6 subtests).

Also recorded as a negative result: splitting the V dim over extra CTAs does
nothing. Going from 24 to 384 CTAs stayed inside noise and regressed below 64
dims per CTA, because it trades request size for CTA count at constant bytes
in flight and so cannot raise memory concurrency.
@github-actions

Copy link
Copy Markdown
Contributor

Required checks failed on 5b65492; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant