Repository navigation
Conversation
A compressed-tensors / ModelOpt checkpoint that declares an FP8 kv_cache_scheme bakes per-tensor k_scale/v_scale onto each RadixAttention layer. They are FP8-KV calibration factors: the write path divides K/V by them before the FP8 cast, and the read path multiplies them back. BaseKVCacheMethod loaded them unconditionally, so an explicit --kv-cache-dtype override to a non-FP8 cache desynced read from write. With bf16 the K/V are stored unscaled, yet every backend still applied the descale on read, shrinking the attention output by ~k_scale (greedy decode drifts into garbage). Self-scaling block formats (mxfp4/nvfp4/fp4_mx_block16) carry their own per-block scales and must not be double-scaled either. Gate the scales at the single load point on the runtime dtype, mirroring load_kv_cache_scales() (which already loads only for fp8_e4m3) and configure_kv_cache_dtype(): an explicit FP8 dtype applies them, and "auto" applies them only when the checkpoint's kv_cache_quant_algo is FP8 -- exactly when configure_kv_cache_dtype resolves "auto" to FP8. Otherwise (bf16, a self-scaling block format, or "auto" under a non-FP8 algo such as an NVFP4 ModelOpt checkpoint that still builds this method) fall back to unit scales, so every backend reads an identity descale with no per-backend guard. Fixes all backends at once (flashinfer/triton/fa3/flashmla/aiter read layer.k_scale unconditionally). The fp8_e4m3/fp8_e5m2 and auto-with-FP8-scheme paths are byte-for-byte unchanged.
Standalone MXFP4 (OCP MX v1.0 6.3) KV-cache codec: block-32 E2M1 values with one E8M0 scale per head block, scale = 2^(floor(log2(amax)) - 2), saturating round-to-nearest-even, and the OCP NaN / +-Inf / zero-block policy. Blocks never span heads; a partial final block is zero-padded. - MXFP4KVQuantizeUtil (kvfp4_tensor.py): eager pure-tensor quantize/dequantize, CUDA-graph capturable. Kept eager because a torch.compile'd slice + pad-to-even + nibble-pack graph was observed to skip the final output byte under some inductor versions, and the codec contract is bit-exactness. - mxfp4_quant.py: a fused Triton kernel that quantizes and scatter-stores K/V (packed E2M1 data + E8M0 scales) into the paged pool in one pass -- slot 0 reserved (CUDA-graph padding contract), no host sync. Bit-exact with the eager codec for bf16/fp16; gated to fall back to eager for fp32 / CPU / non-contiguous-last-dim / shape-mismatched inputs. Also hosts e8m0_to_f32, the E8M0-byte -> fp32 primitive shared with the decode-attention kernel (next commit), built from the fp32 bit pattern rather than tl.exp2 (whose approximate ex2 flushes the 2^-127 subnormal scale to zero). Inert: no KVCacheMethod, CLI, pool or backend references these yet -- the serving wiring lands in a later commit. Validated against an independent OCP reference inlined in the test (shares no code with the runtime codec): codec conformance (CPU) and fused-store bit-exactness + reserved-slot + support-gate (CUDA).
Native MXFP4 KV-cache decode attention for MHA/GQA/MQA that reads the packed E2M1 + E8M0 pool buffers directly and dequantizes inline -- no BF16 PLAIN materialization, so the KV bandwidth saving is realized on the memory-bound decode path. Portable Triton, developed and validated on SM120 (Blackwell). - decode_attention_mxfp4_sm120.py: a two-stage split-KV decode kernel mirroring the stock decode_attention.py structure. A per-head stage1 serves MHA in fp32; a grouped stage1 serves a whole GQA/MQA query group per program, unpacking the shared K/V once and using tl.dot over bf16 tiles (lossless for dequantized E2M1). E8M0 scales expand via the bit-exact e8m0_to_f32 (imported from the codec commit) and E2M1xE8M0 -> bf16 via pure-integer bit construction, avoiding the approximate ex2 FTZ that would zero the 2^-127 scale. stage2 merges splits. Inert: no backend dispatches to it yet. The triton_backend wiring -- including the num_warps=8 the grouped kernel needs at bs>=4 / long seq (4 warps spills the BLOCK_N dequant tiles and runs ~15x slower) -- lands in the serving commit. Test lives in test/registered/kernels/ops/attention/ (mirroring the kernel source), with the pure-Torch decode golden inlined per the kernel-test convention (sibling kernel tests are self-contained; the attention_methods test-kit is for backend-fixture unittests only). The golden dequantizes the same packed bytes with the production eager codec and runs explicit fp32 GQA attention, never calling the Triton kernel. Validated on SM120 within frozen per-path thresholds (MHA rel_l2 <= 3.5e-7; GQA/MQA rel_l2 <= 1.85e-3) over MHA/GQA/MQA, page sizes 1-128, contiguous/shuffled/interleaved/non-monotonic slot layouts, a long multi-split sequence and an all-zero request, plus CUDA-graph capture/replay byte-determinism for the decode and fused-write kernels.
… decode + flashinfer PLAIN)
Add --kv-cache-dtype mxfp4 end to end. MXFP4KVCacheMethod stores strict OCP
block-32 E2M1 data + E8M0 scales through a fused Triton write kernel and reads
them back as BF16 (PLAIN); the KV pool allocates the packed/scale buffers and
sizes cells accordingly, with a v_head_dim == head_dim guard.
Serving wiring: the triton backend gains a native MXFP4 decode path that reads
the packed E2M1 + E8M0 scales inline through the MXFP4 decode kernel
(num_warps=8, kv-splits floor 32), while flashinfer resolves the KV dtype it
presents from the attention access (PLAIN -> BF16) for prefill and the
validation-only decode. A server-arg hook rejects unsupported combinations at
parse time: CUDA-only, MHA-only, prefill=flashinfer with decode in
{flashinfer, triton}, and PLAIN decode without --disable-cuda-graph.
No per-backend scale-validity guards are carried over: the checkpoint FP8-scale
loading gate already reduces layer.k_scale/v_scale to identity for mxfp4 (and
every non per-tensor-FP8 cache), so both backends read scales unconditionally.
Tests: MXFP4 method contract and the checkpoint-scale gate (CPU), pool
write/PLAIN-read/slot-move and checkpoint-scale-ignore (CUDA), hook validation
(CPU), and an SM120 end-to-end serving test driving the shared basic decode
correctness probes over flashinfer PLAIN prefill + native triton decode (CUDA).
cb0100b to
b2ada37
Compare
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2ada378e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
…mily + S-EXT1 + riders + S-RACE + LLGUARD/ENVREG), S-EXT1 and S-RACE items CLOSED, entry-256's raw-env deviation and LL-vs-PCIe-IPC collision warning both marked CLOSED with the BOOT_DELTA (h') needle in landing terms, sgl-project#35617 left as the sole FAM-A7 survivor, sgl-project#44732 re-pointed to the operator's merge-sgl-project#5 answer, and the KV-ROUTE DECISION RECORD added (HOLD NVFP4-KV until armed PCIe-IPC is measured post-merge; preferred-if-ever sequence merge -> sgl-project#43356-delta -> sgl-project#46452 NOT sgl-project#43240 -> optional sgl-project#43471 -> fork BUILD-A QSA FP4 reads (template sgl-project#38724) -> BUILD-B L3 scale sidecar -> one GPU window; sgl-project#46141 template, sgl-project#44462 NEXT, MXFP4 declined, entry-247 re-sync watch-string modelopt_e4m3_k32)
The MXFP4 native decode path bought KV capacity at the cost of latency:
at 98k context it was 17% slower per output token than the flashinfer
FP8 baseline. Server-side profiling (profile_by_stage, DECODE trace) put
weight GEMMs at 75-84% of step time and identical in absolute terms on
both sides, leaving the grouped stage-1 attention kernel as the only
differing term: 0.324 vs 0.130 ms per layer-step, 2.5x.
Dequantize E2M1 x E8M0 through a 4096-entry LUT indexed by
(scale_byte << 4) | code instead of reconstructing each element
arithmetically. bf16 is exact for these values -- a 2-bit mantissa times
a power-of-two scale always fits an 8-bit significand. The table is
built eagerly when the KV pool is created, since a capture-time build
would be baked into the CUDA graph.
Nibble-interleave the K/V tiles. Lane j of the low half holds logical
element 2j and lane j of the high half holds 2j+1, so a single
packed-byte load and a single scale load feed both nibbles, replacing
element-wise addressing by offs_d // 2 that read every byte twice and
needed a select to pick the nibble. tl.dot is invariant under
permutation of the contraction dim, so Q is loaded in the same even/odd
order; for V the permutation lands on the output dim and is undone by
two strided stores. This halves the register-resident staging path and
drops a full-width uint8 intermediate: 255 regs + 16 spills -> 178 regs
and no spills, which in turn admits 4 warps and two CTAs per SM.
Derive the kv-split floor from a whole number of CTA waves rather than a
hardcoded 32. Stage 1 is register-bound, so exceeding the
SMs * CTAs-per-SM grid by one CTA buys a second wave that the whole
kernel waits on -- 84 splits (336 CTAs) run 0.120 ms where 88 (352) take
0.186. The device-side schedule in get_num_kv_splits_triton inflates the
core count by log2(seq_len/64), which suits the lighter bf16 CTAs but
overshoots here, so the bound has to come from this cap.
stage1_launch_config() publishes the measured occupancy so the backend
schedule cannot silently desync from a retuned kernel.
Grouped stage-1 on SM120 (170 SMs), bs=1, 24Q/4KV, head_dim 256, under
CUDA graph:
100k ctx 150k ctx
before 0.344 ms 0.515 ms
after 0.120 ms 0.171 ms (-65%, -67%)
End to end, streaming ITL p50 at 98k context: 29.14 -> 24.65 ms against
24.92 ms for FP8, so 1.1% ahead of the baseline while still holding
1.524x the KV capacity (160,109 vs 105,071 tokens). Doubling the splits
grows the attn_logits scratch from 49.5 to 100 MB, which fits the
existing headroom.
Accuracy is unchanged against the frozen references: grouped rel_l2 is
1.155e-03 under a 1.85e-03 cap, the residual being kernel accumulation
order only since both sides consume identical dequantized values.
9f55083 to
6657f08
Compare
This comment has been minimized.
This comment has been minimized.
Stage 2 merges the per-split partials of the MXFP4 decode path. Its cost
tracked MAX_KV_SPLITS alone -- 85/127/170/255 splits cost
0.0159/0.0235/0.0297/0.0435 ms regardless of context length or CTA count,
a fixed ~187 ns (~520 cycles) per split. That is the signature of the
loop-carried e_max -> old_scale -> acc chain paying a full load latency every
iteration, and it explains why num_stages had no effect whatsoever: both
loads sat inside `if split_kv_end > split_kv_start`, and the pipeliner cannot
hoist loads out of a data-dependent branch.
Replacing the branch with predication lets the pipeliner run several splits
ahead. An out-of-range split now contributes old_scale == 1 and
exp_logic == 0, leaving acc, e_sum and e_max untouched, which is exactly what
the branch did. The tl.where on old_scale is load-bearing: if the first split
is empty, e_max - n_e_max is -inf - -inf, and a bare exp would push NaN into
the accumulator.
Predication on its own is a small loss (at num_stages=2, 0.0159 -> 0.0178);
the win comes entirely from the pipelining it unlocks, so the two go
together. Cold-L2 sweep at 100k, bs=1, 24 heads, 85 splits:
num_stages 1 2 3 4 6 8
stage 2 0.0147 0.0178 0.0097 0.0080 0.0083 0.0096
Stage 2, cold L2 (stage 1 measured identical on both sides):
seq before after
32,768 0.0149 0.0080 -46.3%
100,000 0.0159 0.0080 -49.7%
150,000 0.0161 0.0083 -48.4%
Whole decode attention against the flashinfer FP8 KV baseline, both run
through the same cold-L2 harness (bs=1, 24Q/4KV, head_dim 256, page_size 1):
seq FP8 MXFP4
8,192 0.0313 0.0283 -9.4%
32,768 0.0684 0.0609 -10.9%
65,536 0.1209 0.0965 -20.2%
100,000 0.1804 0.1351 -25.1%
150,000 0.2665 0.1891 -29.0%
The two paths are structurally identical, so at 100k the merge halves compare
directly: flashinfer's PersistentVariableLengthMergeStatesKernel costs
0.0088 ms against our 0.0079, where before this change we were at 0.0159.
Numerically this is a no-op rather than a tolerance change. Over 14 shapes
(seq in {32, 64, 100, 1000, 32768, 100000, 150000} x max_splits in {8, 85})
the output is bitwise identical to the branched version under an int32 view,
including seq=32 with max_splits=85 where 84 of the 85 splits are empty --
the exact path this patch rewrites. Unit tests pass (4 tests + 6 subtests).
Also recorded as a negative result: splitting the V dim over extra CTAs does
nothing. Going from 24 to 384 CTAs stayed inside noise and regressed below 64
dims per CTA, because it trades request size for CTA count at constant bytes
in flight and so cannot raise memory concurrency.
|
Required checks failed on 5b65492; this PR cannot be merged until they pass. PR title (run)
|
Motivation
Add an OCP MXFP4 KV-cache recipe (
--kv-cache-dtype mxfp4, block-32, E2M1 data + E8M0 scale) for dense MHA models on SM120 (RTX 5090 class), validated on Qwen3.5/3.8-27B. It stores KV in ~1.9× fewer bytes than FP8, for a measured +52% KV capacity (1.52×) at equal memory. End to end and self-contained in pure Triton (no sgl-kernel wheel change): codec + memory pool + a native Triton split-KV decode kernel + backend integration (FlashInfer PLAIN prefill, Triton native decode).It also carries one general fix: checkpoint FP8
k_scale/v_scaleare now applied through a single load-time gate keyed on--kv-cache-dtype, replacing scattered per-backendkv_cache_scales_validguards.Modifications
Standard MXFP4 layout — block-32, E2M1 packed (uint8) + one E8M0 byte (
float8_e8m0fnu) per 32 values, blocked per head (never across heads); no global scale (unlike NVFP4). Per token per layer (K or V) on Qwen3.8 (4 KV heads, head_dim 256): 4×(128 packed + 8 scale) = 544 B, vs FP8 1024 B and BF16 2048 B.Components (4 commits):
kv_cache.py):_checkpoint_kv_scales_applydecides whether the checkpoint's per-tensor FP8 scales apply, keyed on the raw--kv-cache-dtypestring (fp8_e4m3/e5m2 → yes;auto→ viakv_cache_quant_algo; bf16/mxfp4/nvfp4/mxfp8 → unit). One point covers compressed-tensors + ModelOpt + every backend.kvfp4_tensor.py,mxfp4_quant.py): OCP quantize/dequant (saturating RNE, NaN/±Inf/zero-block policy) + a fused write kernelquant_store_kv_mxfp4(amax → E8M0 → E2M1 pack → scatter, CUDA-graph safe);e8m0_to_f32is built by bit-shift (avoidstl.exp2FTZ flushing 2^-127).decode_attention_mxfp4_sm120.py): native Triton split-KV — per-head stage1 (MHA, fp32), grouped stage1 (GQA/MQA), stage2 split-combine;num_warps=8/BLOCK_N=64; head_dim-agnostic (next_power_of_2,//32).MXFP4KVCacheMethod+ an access registry (PLAINFlashInfer BF16 read /NATIVE_FP4Triton inline), pool buffers + cell sizing, backend wiring, and a startup hook.Startup guards (fail fast at arg parse): CUDA-only; MHA-only (no MLA); prefill=flashinfer with decode ∈ {flashinfer, triton}; the PLAIN decode path requires
--disable-cuda-graph; the pool requiresv_head_dim == head_dim. 21 files, +3097 / −36.Validation
Checkpoint using unsloth/Qwen3.8-27B-NVFP4, FP8 (
fp8_e4m3+ flashinfer) vs MXFP4 (mxfp4 + flashinfer prefill + triton decode), Qwen3.8-27B-NVFP4 on single RTX 5090, identical flags otherwise. Capacity/throughput are single-request at--mem-fraction-static 0.80(a clean-capacity baseline, no mamba-state tuning); accuracy is the paired, truncation-excluded metric.max_total_num_tokens)Accuracy is statistically indistinguishable from FP8. MXFP4 inflates generation length (GSM8K mean +43%, AIME median +61% — KV noise flattens softmax concentration), raising the truncated rate (AIME 30%→40%).
Known Limitations / Follow-ups
v_head_dim == head_dimrequired.--disable-cuda-graph), while production decode uses the graph-safe Triton native kernel. A native Triton prefill is out of scope.--max-mamba-cache-size) that eats ~2.9 GB of the KV budget, pulling the gain down toward ~+20%.References
Checklist
CI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.