Skip to content

[Feature] MXFP4 KV Cache Decode for DSV4 on Hopper - #32741

Closed
TobyMint wants to merge 31 commits into
sgl-project:mainfrom
TobyMint:feat/dsv4-mxfp4-kvcache-on-hopper
Closed

TobyMint wants to merge 31 commits into
sgl-project:mainfrom
TobyMint:feat/dsv4-mxfp4-kvcache-on-hopper

Conversation

@TobyMint

@TobyMint TobyMint commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

MXFP4 KV cache for DeepSeek V4 on Hopper (SM90/H20), end to end: codec + memory pool + fused decode attention kernel + backend integration. The decode kernel is a JIT-compiled port of the FlashMLA three-stage split-KV design (scheduler metadata kernel + persistent WGMMA main kernel + combine kernel), covering all 44 layers (SWA + C4/C128 + attn_sink) in one call.

The PR is self-contained: the kernel and its vendored dependencies live under python/sglang/kernels/jit/ and compile at first use via load_jit — no sgl-kernel wheel changes required.

MXFP4 Layout (368 B/token)

[224 B packed E2M1 | 14 B E8M0 + 2 B pad | 128 B BF16 RoPE]
  • E2M1 (4-bit FP: 1S+2E+1M) nope data, block-size 32 (14 groups)
  • E8M0 8-bit exponent-only scale per group (dequantized in-register via bits << 23)
  • BF16 RoPE (unquantized)
  • Row-major contiguous, no page tail padding; no global scale needed (unlike NVFP4)
  • 16-byte-aligned row (368 B = 16×23) enables 128-bit loads; the NVFP4 reference's 380 B is only 4-byte aligned

Memory capacity gain vs FP8: ~+37% full-token capacity (bytes/token 7705 → 5609; the NVFP4 reference gains +31.75%).

Decode Kernel (kernels/jit/csrc/mxfp4_dsv4_decode_sm90/)

FlashMLA-style fused split-KV decode for MXFP4:

  1. Scheduler metadata kernel — 1 CTA × 1 thread computes per-SM split assignments with device-side length clamping (CUDA-graph safe: runs inside the captured graph, re-executes with replayed top-k lengths)
  2. Persistent main kernel — 384 threads (3 warp groups: producer dequant→SMEM, QK+PV local, PV remote), WGMMA 64×64×16 / 64×256×16; E8M0 block-32 dequant via bit-shift, E2M1 via LUT+PRMT (4 instructions per 16 elements)
  3. Combine kernel — merges split-K partials with PDL (programmatic dependent launch)

Entry point: entry.cuh (TVM-FFI single translation unit, explicit instantiations for h_q 64/128). Python wrapper: kernels/ops/attention/mxfp4_dsv4_decode_sm90.py (FlashMLASchedMeta, per-geometry scratch reuse, C++-side stream resolution via TVMFFIEnvGetStream).

Vendored dependencies (provenance)

Component Source
params.h, defines.h, utils.h, flashmla_utils.h, kerutils/, combine FlashMLA upstream @ 05e26647
config.h, dequant.h, layout.h, components/helpers.h, splitkv skeleton SGLang reference PR #31269 (NVFP4)
splitkv_mla.cuh (MXFP4 variant), entry.cuh This PR

CUTLASS/CUTE headers come from the JIT build's extra_dependencies=["cutlass"] (pinned cutlass @ 147f5673, matching the reference PR). kerutils includes were converted to relative paths during vendoring.

Changes

  • Codec (srt/layers/attention/dsv4/mxfp4_k_cache.py): quantize/dequant for the 368 B row (Triton), paged variants
  • Compressor (compressor_v2.py): _forward_mxfp4 covering all 44 layers (C0/C4/C128), decode-plan boundary semantics
  • Memory pool: parameterized enablement via --kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4 (env alias with deprecation warning); pool sizing/accounting unified
  • Startup guards: HiSparse + MXFP4 combination rejected at init (incompatible sparse layout); DeepseekV4 + fp4_e2m1 on non-SM90 fails fast with a clear error
  • Backend (deepseek_v4_backend.py): _forward_mxfp4_decode_flashmla — one fused call per layer; scheduler metadata shared per geometry across layers, re-created during CUDA-graph capture (top-k lengths are replayed device inputs)
  • Tests: test/registered/kernels/ops/attention/test_mxfp4_dsv4_decode.py (17 kernel cases: C0/C4/C128 × flash/profiling geometries, graph replay + length clamping, invalid contracts, scheduler-refresh regressions) + test_mxfp4_k_cache.py (5 codec cases) + test_dsv4_hisparse_pool.py (3 pool-guard cases) + test_dsv4_compress_write_pad.py (3 restored cases); benchmark in test/registered/kernels/benchmark/attention/bench_mxfp4_dsv4_decode.py
  • E2E: test/registered/kernels/e2e/test_mxfp4_dsv4_e2e.py — 8-GPU H200 server + gsm8k, exercising pool configurator → compressor → CUDA-graph capture (CI stage extra-b)
  • Removed the first-generation standalone JIT kernel (mxfp4_decode.cuh + _forward_mxfp4_decode) superseded by the FlashMLA port

Validation

  • Kernel SASS was byte-identical to the AOT (cmake) build of the same sources at port time; the subsequent refactors (V32 path removal, aligned 128-bit row loads) are instruction-level changes covered by the same unit suite
  • Unit: 28/28 kernel + codec + pool-guard tests pass; 8-GPU e2e integration test passes locally
  • End-to-end on 8×H20 (TP8, DeepSeek-V4-Flash), FP8 (--kv-cache-dtype fp8_e4m3) vs MXFP4 (--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4), identical server flags otherwise:
Metric FP8 baseline MXFP4 (this PR) Δ
Decode throughput (server-side, 4k×256, conc 32) 1013 tok/s 980 tok/s −3.3%
GSM8K (1319, greedy, max_tokens 1024; truncated samples excluded) 96.89% 96.89% 0.0pt
AIME25 (480 = 30×16, greedy, max_tokens 32768; 73 truncated samples excluded from both) 99.57% pass@1 99.78% pass@1 +0.2pt

Known gaps / follow-ups

  • Small-batch (b≤8) kernel-level latency vs FP8: +8–12% (kernel-inherent scheduling granularity; end-to-end TPOT already within 5%)
  • Prefill-path throughput: −6.8% vs FP8 (out of scope for this PR; triton-based MXFP4 prefill)
  • Deep-context + high-concurrency decode (64 concurrent, 8k ctx, no rolling prefill): decode TPOT +17% vs FP8 (33.1 vs 28.3 ms); end-to-end 8k→8k throughput on par (1677 vs 1652 tok/s). One full AIME25 eval run showed −44% throughput vs FP8, but four controlled reruns (5-question replica, rolling short-prompt, 8k×8k fixed) could not reproduce it and measured MXFP4 at parity or faster — attributed to environment, not the kernel.
  • sm_90a-only (Hopper); no SM100 support (non-SM90 + fp4_e2m1 is rejected at startup)
  • HiSparse (sparse KV cache) not supported with MXFP4 — rejected at startup

References

  • PR [Deepseek v4 & GLM-5.2] SM90 nvfp4 kvcache #31269 — NVFP4 KV cache for DSV4 in sgl-kernel (reference implementation for the FlashMLA three-stage decode)
  • FlashMLA upstream @ 05e26647 — scheduler/combine kernels and kerutils
  • This PR takes the jit_kernel approach (JIT CUDA via TVM-FFI) instead of AOT CUTLASS in sgl-kernel, keeping the PR self-contained

Status: Ready for review — kernel + integration complete, end-to-end validated on 8×H20.


CI States

Latest PR Test (Base): ❌ Run #32231903565
Latest PR Test (Extra): ❌ Run #32231903211

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@TobyMint TobyMint changed the title [RFC][WIP] MXFP4 KV Cache Decode Kernel for DSV4 on Hopper (jit_kernel) [RFC][WIP] MXFP4 KV Cache Decode Kernel for DSV4 on Hopper Jul 29, 2026
@b8zhong b8zhong mentioned this pull request Aug 5, 2026
15 of 41 tasks
@TobyMint
TobyMint force-pushed the feat/dsv4-mxfp4-kvcache-on-hopper branch from 4e8bf04 to a0f7671 Compare August 10, 2026 08:42
@github-actions github-actions Bot added quant LLM Quantization jit-kernel labels Aug 12, 2026
@TobyMint TobyMint changed the title [RFC][WIP] MXFP4 KV Cache Decode Kernel for DSV4 on Hopper [Feature] MXFP4 KV Cache Decode for DSV4 on Hopper (JIT FlashMLA-style) Aug 12, 2026
@TobyMint TobyMint changed the title [Feature] MXFP4 KV Cache Decode for DSV4 on Hopper (JIT FlashMLA-style) [Feature] MXFP4 KV Cache Decode for DSV4 on Hopper Aug 12, 2026
@TobyMint
TobyMint marked this pull request as ready for review August 13, 2026 01:56
Every 368-byte row (and page-block stride) is a multiple of 16 bytes, so
the producer's NoPE and RoPE payloads can be loaded with single aligned
vector loads instead of the NVFP4-era scalar u32 assembly (that layout's
380-byte rows were only 4-byte aligned). Replaces load_u32_evict_last /
load_packed_e2m1x16_unaligned / load_bf16x8_unaligned with one u64 and
one 128-bit load, keeping the L1::evict_last / L2::128B cache hints.
SASS shows only LDG.E.128 in the producer.
… session

The scheduler's split assignment depends on the per-call top-k lengths,
but the generation kernel only ran when the metadata buffer was first
allocated. Eager (non-graph) decode reuses the stale assignment once a
growing sequence crosses a 64-token block boundary, silently dropping
newly added KV blocks; a later CUDA-graph capture with the same geometry
reused the first graph's scheduler instance and never recorded the
generation kernel.

- wrapper: re-run the scheduler on every eager call (buffers reused);
  during capture it still runs only on a fresh buffer so the kernel is
  recorded once and re-executed on replay with replayed lengths
- backend: detect capture-session start (capturing edge) and clear the
  capture scheduler dict, so each graph records its own generation
- tests: eager stale-scheduler regression (same meta, lengths crossing a
  64-block boundary) and fresh-meta-per-capture regression; the eager
  case fails on the pre-fix code
The paged dequant kernel masked the stores with valid_src, so invalid
token indices left the output row untouched: torch.empty garbage (or a
pre-allocated buffer's old contents) leaked into the returned tensor.
The CPU reference path explicitly zeroed the output, and the API/test
documented zero rows — the GPU kernel disagreed. The OOB test only
passed by luck (freshly cudaMalloc'd pages happen to be zero).

- kernel: write zeroed values for invalid rows via tl.where
- test: pre-fill the output with non-zero values, cover both negative
  and above-capacity indices (fails on the pre-fix kernel)
The DSV4 pool factory passes use_mxfp4 to every pool class, but
HiSparseC4DevicePool never gained the parameter: enabling HiSparse
crashed with 'unexpected keyword argument' even with the default FP8 KV
cache, and its positional super().__init__ shifted start_layer into the
parent's use_mxfp4 slot (a non-None start_layer would have silently
enabled the MXFP4 layout).

- signature aligned with the parent (use_mxfp4 after enable_memory_saver)
- super().__init__ now passes all keyword arguments
- MXFP4 + HiSparse is rejected at construction with a clear error
- regression tests: FP8 pool init, MXFP4 rejection, positional
  super().__init__ arg alignment (fail on the pre-fix code)
Commit 0d07d17e0 dropped _dflash_draft_cell_size, the DFLASH draft-KV
memory budget, the DSV4 speculative compress-state ring capacity check,
and their regression tests, claiming they were restored to upstream.
They are not: main still uses all of them. Merging the removal would
under-price DFLASH draft KV memory and silently let large draft-token
configs overflow the compress-state ring.

Restored verbatim from upstream/main (including the DCP rank factor in
_dflash_draft_cell_size), keeping only the MXFP4-specific
_get_bytes_per_full_token branch. The ring-capacity CPU tests are back
in test/registered/unit/mem_cache/test_dsv4_compress_write_pad.py.
…non-SM90, add e2e integration test

- Remove the four SM100 kerutils headers (1407 lines) vendored from
  FlashMLA: the SM90 DSV4 kernel never instantiates them and device.cuh
  included them unconditionally
- Strip trailing whitespace (24 sites) so git diff --check is clean
- Fail fast at server-arg resolution: fp4_e2m1 on a non-SM90 GPU raises
  instead of failing at the first JIT/attention call
- Add test/registered/kernels/e2e/test_mxfp4_dsv4_e2e.py: launches the
  real 8-GPU server (pool configurator, attention backend, compressor,
  CUDA-graph capture) and runs GSM8K accuracy, covering the integration
  surface kernel-level unit tests cannot reach
…defensive num_splits tail fill

The tile scheduler caps each SM part at a fixed block payload, so batches
larger than the SM-part count give one part multiple requests. The producer's
epilogue-drain barrier waited on the wrong phase parity (bar_phase_o ^ 1 —
one phase past the one the consumers had completed), so the persistent kernel
spun forever. Server capture warmup (bs up to 256) hangs 100%; b <= 78 worked
only because no part ever served two requests.

Also fill the num_splits prefix sum for requests the partition loop never
reached (uninitialized memory for the combine kernel) and add a regression
test pinning b = num_sm_parts + 1 and 2 * num_sm_parts.
The wrapper cached o_accum/lse_accum in a module-global dict keyed by
(batch, s_q, h_q, device, head_dim_v) with no eviction. A CUDA-graph
capture sweep creates one entry per captured batch size, permanently
pinning ~0.5 GiB (h_q=64) / ~0.85 GiB (h_q=128) of FP32 scratch per
runner on an H20 capture list — measured with a 19-batch sweep — and the
globals are shared, so two runners with a matching geometry in one
process would race on the same buffers.

flash_mla_with_kvcache_dsv4_mxfp4 now accepts split_accum_buffers: a
caller-owned (lse rows, o rows) workspace prefix-sliced per call (every
batch only touches accumulator rows [0, batch + sm parts), so all
captured graphs share one stable base address). The DSV4 attention
backend owns one arena per query-head count, sized for
max_running_requests x draft tokens plus the SM-part count, growing
(with retirement, graphs keep addressing old arenas) only when an eager
batch exceeds the configured capacity. The module-global cache remains
only as the fallback for standalone callers (tests, benchmarks).

Regression test sweeps a capture-like batch list through the arena path:
the global stays empty, results match the default path, and undersized
or wrong-shaped arenas fail loudly instead of truncating.
forward_unified returned right after _forward_mxfp4, so the
online_c128_mtp.write_prefix_states call that closes every other store
path never ran for MXFP4 caches.  With SGLANG_OPT_USE_ONLINE_COMPRESS=1
plus SGLANG_EXPERIMENTAL_ONLINE_C128_MTP=1 and speculative decoding, the
target-verify step silently skipped its C128 prefix-state write and the
next commit continued from stale state.

Extract the write into _write_online_c128_prefix_states and call it from
both store paths.  Regression test spies the controller on a minimal
mixin host: the MXFP4 branch must record the write with the verify mode
(falls back to the batch mode without _original_forward_mode); verified
red on the pre-fix control flow.
Three inconsistencies, all reproduced before the fix:

- The quantizer rounded log2(amax/6) to nearest, so every group whose
  amax lands in (6*2^k, 6*2^k*sqrt(2)] picked a scale one octave too
  small and saturated its max element to the largest E2M1 code — amax 8
  decoded back as 6 (up to 29% off the group max).  ceil (the MX
  convention, matching the NVFP4 paths) keeps the amax representable;
  both the Triton kernel and the CPU reference switch together and clamp
  below the reserved byte 255.

- The fused decode kernel treated E8M0 byte 0 as 0.0f while the Triton
  and CPU dequantizers applied 2^-127; the format has no zero encoding
  (bytes 0..254 = 2^-127..2^127, 255 reserved NaN).  The kernel now
  returns the spec value, mapping byte 0 to the 2^-127 denormal and 255
  to NaN, and the Triton dequantizer builds the scale from the FP32
  exponent pattern instead of exp2, which flushes that denormal to zero.

Tests pin the ceil byte and exact max-element recovery across rounding
boundaries (including the previously saturated band) and the byte-0
decode; both go red on the pre-fix code.
The field predates the two-way namespace coverage lint and had no NS()
marker, so the rebased main's lint (test_server_args_namespaces) fails
on this branch.
@TobyMint
TobyMint force-pushed the feat/dsv4-mxfp4-kvcache-on-hopper branch from c1a5004 to 1b9a7a3 Compare August 18, 2026 14:20
The compressor_v2 merge left two blank lines before _forward_unified_hip;
CI's black hook rejects it (local pre-commit ran before the rebase, so it
slipped through).
The SM90 kernel hardcodes head dim 512, 64/128 query heads, int32 index
and length tensors, contiguous row-major layouts, same-device placement,
and 128-bit vector loads of the 368-byte cache rows. The wrapper only
checked dtype/ndim and a 4-byte cache alignment, so e.g. a [B, 1, 64, 256]
bf16 q or an int64 indices tensor passed validation and the kernel read
memory with hardcoded 512-element int32 strides (out-of-bounds /
reinterpreted reads), and a 4-mod-16 cache base pointer faulted the
kernel's uint4 loads.

Validate the whole contract in Python: head dims, head count, indices
geometry ([B, S_Q] matching q, width % 64 for primary and extra), int32
index/length dtypes, float32 attn_sink, topk_length shape [B], device
consistency for every tensor, contiguity, and 16-byte base-pointer
alignment for both caches. Regression tests cover each predicate
(verified red on the pre-fix wrapper).
…e per device

The JIT entry's contract violations used fprintf + exit(1), which kills
the whole serving process — and because exit() skips the interpreter's
buffer flush, the scheduler dies without even the message in its stdout
log (verified: a pytest run driving the old path exits 1 with a zero-byte
log). Replace fail() with a [[noreturn]] helper that throws
tvm::ffi ValueError, which propagates to Python as an exception.

The SM90 capability verdict was a process-global static bool, so in a
mixed-GPU process the first device to call in decided for every later
device. Key the cached verdict by device id instead.

Also align the Python module-gate message with its behavior (SM90 only,
not 'or later').
The SM90 check in the DeepSeek V4 kv-dtype pass only ran under
if is_cuda(), so ROCm/XPU/MUSA builds accepted --kv-cache-dtype
fp4_e2m1 and died later at the JIT compile with an opaque error (the
device string cannot distinguish them: get_device() reports 'cuda' on
ROCm too). Gate on is_cuda() and reject with an explicit message.

fp4_e2m1 as a --kv-cache-dtype spelling is the DSV4 MXFP4 recipe, but
non-DSV4 models fell through arg resolution into the KV-cache quant
registry, which rejected it as 'deprecated' deep in pool construction
with a message contradicting this PR's semantics. The generic
_handle_kv4_compatibility slot (which runs after the DSV4 slot) now
rejects it for other archs at startup, pointing at nvfp4 /
fp4_mx_block16 instead.

Both guards verified red on the pre-fix code.
The quantizer derived the RNE denominator from tl.math.exp2(byte - 127).
E8M0 byte 0 is 2^-127, a subnormal, and ex2.approx flushes it to zero, so
every `magnitude >= denominator * midpoint` rung held at magnitude 0 and an
all-zero block was encoded as code 3 (1.5) instead of code 0.  All-zero rows
reach the pool in production via idle dummy stores.  Build the denominator
from the FP32 exponent bits instead, matching the dequant kernel.

Verified: output is bit-identical on healthy data (1.26M packed bytes, 0
mismatches), quantize kernel perf unchanged, GSM8K-200 e2e smoke passes.
Regression test asserts zero blocks pack to zero nibbles and round-trip to
exactly zero.
@TobyMint

Copy link
Copy Markdown
Contributor Author

Per the review request, this PR is now split into a 4-part series based on current main, with files moved to the post-#36704 JIT kernel layout (jit/csrc/deepseek_v4/, wrappers under kernels/ops/attention/dsv4/, e2e under test/registered/quant/):

  1. [1/4][JIT] Vendor FlashMLA upstream files for the DSV4 MXFP4 decode kernel #37135 — vendor FlashMLA upstream files (no behavior change)
  2. [2/4][Kernels] Add MXFP4 KV cache codec (Triton quantize/dequant) #37136 — MXFP4 KV cache codec (Triton quantize/dequant)
  3. [3/4][Kernel] Fused MXFP4 DSV4 decode attention for SM90 (FlashMLA-style, JIT) #37137 — fused MXFP4 DSV4 decode kernel for SM90 (FlashMLA-style, JIT)
  4. [4/4][DeepSeek V4] MXFP4 KV cache for the DSV4 serving stack on Hopper #37138 — serving-stack integration (CLI/pools/backend/compressor + e2e)

Each part is independently reviewable; 1–3 are inert without 4. Content is identical to this PR apart from the re-base onto main (arg_groups/kv_cache_hook refactor), the layout moves, and conflict re-applications in server_args.py / deepseek_v4_backend.py. This PR stays open until the series is reviewed; happy to close it once the series lands or fold parts back if reviewers prefer.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Thanks @TobyMint. Closing this because you have 8 PRs open and none updated in 7 days, over our soft cap of 5 for idle PRs.

Reopen the two or three you most want reviewed - a short active set moves faster than a long idle one.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant