Repository navigation
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
4e8bf04 to
a0f7671
Compare
Every 368-byte row (and page-block stride) is a multiple of 16 bytes, so the producer's NoPE and RoPE payloads can be loaded with single aligned vector loads instead of the NVFP4-era scalar u32 assembly (that layout's 380-byte rows were only 4-byte aligned). Replaces load_u32_evict_last / load_packed_e2m1x16_unaligned / load_bf16x8_unaligned with one u64 and one 128-bit load, keeping the L1::evict_last / L2::128B cache hints. SASS shows only LDG.E.128 in the producer.
… session The scheduler's split assignment depends on the per-call top-k lengths, but the generation kernel only ran when the metadata buffer was first allocated. Eager (non-graph) decode reuses the stale assignment once a growing sequence crosses a 64-token block boundary, silently dropping newly added KV blocks; a later CUDA-graph capture with the same geometry reused the first graph's scheduler instance and never recorded the generation kernel. - wrapper: re-run the scheduler on every eager call (buffers reused); during capture it still runs only on a fresh buffer so the kernel is recorded once and re-executed on replay with replayed lengths - backend: detect capture-session start (capturing edge) and clear the capture scheduler dict, so each graph records its own generation - tests: eager stale-scheduler regression (same meta, lengths crossing a 64-block boundary) and fresh-meta-per-capture regression; the eager case fails on the pre-fix code
The paged dequant kernel masked the stores with valid_src, so invalid token indices left the output row untouched: torch.empty garbage (or a pre-allocated buffer's old contents) leaked into the returned tensor. The CPU reference path explicitly zeroed the output, and the API/test documented zero rows — the GPU kernel disagreed. The OOB test only passed by luck (freshly cudaMalloc'd pages happen to be zero). - kernel: write zeroed values for invalid rows via tl.where - test: pre-fill the output with non-zero values, cover both negative and above-capacity indices (fails on the pre-fix kernel)
The DSV4 pool factory passes use_mxfp4 to every pool class, but HiSparseC4DevicePool never gained the parameter: enabling HiSparse crashed with 'unexpected keyword argument' even with the default FP8 KV cache, and its positional super().__init__ shifted start_layer into the parent's use_mxfp4 slot (a non-None start_layer would have silently enabled the MXFP4 layout). - signature aligned with the parent (use_mxfp4 after enable_memory_saver) - super().__init__ now passes all keyword arguments - MXFP4 + HiSparse is rejected at construction with a clear error - regression tests: FP8 pool init, MXFP4 rejection, positional super().__init__ arg alignment (fail on the pre-fix code)
Commit 0d07d17e0 dropped _dflash_draft_cell_size, the DFLASH draft-KV memory budget, the DSV4 speculative compress-state ring capacity check, and their regression tests, claiming they were restored to upstream. They are not: main still uses all of them. Merging the removal would under-price DFLASH draft KV memory and silently let large draft-token configs overflow the compress-state ring. Restored verbatim from upstream/main (including the DCP rank factor in _dflash_draft_cell_size), keeping only the MXFP4-specific _get_bytes_per_full_token branch. The ring-capacity CPU tests are back in test/registered/unit/mem_cache/test_dsv4_compress_write_pad.py.
…non-SM90, add e2e integration test - Remove the four SM100 kerutils headers (1407 lines) vendored from FlashMLA: the SM90 DSV4 kernel never instantiates them and device.cuh included them unconditionally - Strip trailing whitespace (24 sites) so git diff --check is clean - Fail fast at server-arg resolution: fp4_e2m1 on a non-SM90 GPU raises instead of failing at the first JIT/attention call - Add test/registered/kernels/e2e/test_mxfp4_dsv4_e2e.py: launches the real 8-GPU server (pool configurator, attention backend, compressor, CUDA-graph capture) and runs GSM8K accuracy, covering the integration surface kernel-level unit tests cannot reach
… __main__ to e2e test
…defensive num_splits tail fill The tile scheduler caps each SM part at a fixed block payload, so batches larger than the SM-part count give one part multiple requests. The producer's epilogue-drain barrier waited on the wrong phase parity (bar_phase_o ^ 1 — one phase past the one the consumers had completed), so the persistent kernel spun forever. Server capture warmup (bs up to 256) hangs 100%; b <= 78 worked only because no part ever served two requests. Also fill the num_splits prefix sum for requests the partition loop never reached (uninitialized memory for the combine kernel) and add a regression test pinning b = num_sm_parts + 1 and 2 * num_sm_parts.
The wrapper cached o_accum/lse_accum in a module-global dict keyed by (batch, s_q, h_q, device, head_dim_v) with no eviction. A CUDA-graph capture sweep creates one entry per captured batch size, permanently pinning ~0.5 GiB (h_q=64) / ~0.85 GiB (h_q=128) of FP32 scratch per runner on an H20 capture list — measured with a 19-batch sweep — and the globals are shared, so two runners with a matching geometry in one process would race on the same buffers. flash_mla_with_kvcache_dsv4_mxfp4 now accepts split_accum_buffers: a caller-owned (lse rows, o rows) workspace prefix-sliced per call (every batch only touches accumulator rows [0, batch + sm parts), so all captured graphs share one stable base address). The DSV4 attention backend owns one arena per query-head count, sized for max_running_requests x draft tokens plus the SM-part count, growing (with retirement, graphs keep addressing old arenas) only when an eager batch exceeds the configured capacity. The module-global cache remains only as the fallback for standalone callers (tests, benchmarks). Regression test sweeps a capture-like batch list through the arena path: the global stays empty, results match the default path, and undersized or wrong-shaped arenas fail loudly instead of truncating.
forward_unified returned right after _forward_mxfp4, so the online_c128_mtp.write_prefix_states call that closes every other store path never ran for MXFP4 caches. With SGLANG_OPT_USE_ONLINE_COMPRESS=1 plus SGLANG_EXPERIMENTAL_ONLINE_C128_MTP=1 and speculative decoding, the target-verify step silently skipped its C128 prefix-state write and the next commit continued from stale state. Extract the write into _write_online_c128_prefix_states and call it from both store paths. Regression test spies the controller on a minimal mixin host: the MXFP4 branch must record the write with the verify mode (falls back to the batch mode without _original_forward_mode); verified red on the pre-fix control flow.
Three inconsistencies, all reproduced before the fix: - The quantizer rounded log2(amax/6) to nearest, so every group whose amax lands in (6*2^k, 6*2^k*sqrt(2)] picked a scale one octave too small and saturated its max element to the largest E2M1 code — amax 8 decoded back as 6 (up to 29% off the group max). ceil (the MX convention, matching the NVFP4 paths) keeps the amax representable; both the Triton kernel and the CPU reference switch together and clamp below the reserved byte 255. - The fused decode kernel treated E8M0 byte 0 as 0.0f while the Triton and CPU dequantizers applied 2^-127; the format has no zero encoding (bytes 0..254 = 2^-127..2^127, 255 reserved NaN). The kernel now returns the spec value, mapping byte 0 to the 2^-127 denormal and 255 to NaN, and the Triton dequantizer builds the scale from the FP32 exponent pattern instead of exp2, which flushes that denormal to zero. Tests pin the ceil byte and exact max-element recovery across rounding boundaries (including the previously saturated band) and the byte-0 decode; both go red on the pre-fix code.
The field predates the two-way namespace coverage lint and had no NS() marker, so the rebased main's lint (test_server_args_namespaces) fails on this branch.
c1a5004 to
1b9a7a3
Compare
The compressor_v2 merge left two blank lines before _forward_unified_hip; CI's black hook rejects it (local pre-commit ran before the rebase, so it slipped through).
The SM90 kernel hardcodes head dim 512, 64/128 query heads, int32 index and length tensors, contiguous row-major layouts, same-device placement, and 128-bit vector loads of the 368-byte cache rows. The wrapper only checked dtype/ndim and a 4-byte cache alignment, so e.g. a [B, 1, 64, 256] bf16 q or an int64 indices tensor passed validation and the kernel read memory with hardcoded 512-element int32 strides (out-of-bounds / reinterpreted reads), and a 4-mod-16 cache base pointer faulted the kernel's uint4 loads. Validate the whole contract in Python: head dims, head count, indices geometry ([B, S_Q] matching q, width % 64 for primary and extra), int32 index/length dtypes, float32 attn_sink, topk_length shape [B], device consistency for every tensor, contiguity, and 16-byte base-pointer alignment for both caches. Regression tests cover each predicate (verified red on the pre-fix wrapper).
…e per device The JIT entry's contract violations used fprintf + exit(1), which kills the whole serving process — and because exit() skips the interpreter's buffer flush, the scheduler dies without even the message in its stdout log (verified: a pytest run driving the old path exits 1 with a zero-byte log). Replace fail() with a [[noreturn]] helper that throws tvm::ffi ValueError, which propagates to Python as an exception. The SM90 capability verdict was a process-global static bool, so in a mixed-GPU process the first device to call in decided for every later device. Key the cached verdict by device id instead. Also align the Python module-gate message with its behavior (SM90 only, not 'or later').
The SM90 check in the DeepSeek V4 kv-dtype pass only ran under if is_cuda(), so ROCm/XPU/MUSA builds accepted --kv-cache-dtype fp4_e2m1 and died later at the JIT compile with an opaque error (the device string cannot distinguish them: get_device() reports 'cuda' on ROCm too). Gate on is_cuda() and reject with an explicit message. fp4_e2m1 as a --kv-cache-dtype spelling is the DSV4 MXFP4 recipe, but non-DSV4 models fell through arg resolution into the KV-cache quant registry, which rejected it as 'deprecated' deep in pool construction with a message contradicting this PR's semantics. The generic _handle_kv4_compatibility slot (which runs after the DSV4 slot) now rejects it for other archs at startup, pointing at nvfp4 / fp4_mx_block16 instead. Both guards verified red on the pre-fix code.
The quantizer derived the RNE denominator from tl.math.exp2(byte - 127). E8M0 byte 0 is 2^-127, a subnormal, and ex2.approx flushes it to zero, so every `magnitude >= denominator * midpoint` rung held at magnitude 0 and an all-zero block was encoded as code 3 (1.5) instead of code 0. All-zero rows reach the pool in production via idle dummy stores. Build the denominator from the FP32 exponent bits instead, matching the dequant kernel. Verified: output is bit-identical on healthy data (1.26M packed bytes, 0 mismatches), quantize kernel perf unchanged, GSM8K-200 e2e smoke passes. Regression test asserts zero blocks pack to zero nibbles and round-trip to exactly zero.
|
Per the review request, this PR is now split into a 4-part series based on current
Each part is independently reviewable; 1–3 are inert without 4. Content is identical to this PR apart from the re-base onto main (arg_groups/kv_cache_hook refactor), the layout moves, and conflict re-applications in |
|
Thanks @TobyMint. Closing this because you have 8 PRs open and none updated in 7 days, over our soft cap of 5 for idle PRs. Reopen the two or three you most want reviewed - a short active set moves faster than a long idle one. Some directories moved recently, so an older branch may need retargeting: |
Summary
MXFP4 KV cache for DeepSeek V4 on Hopper (SM90/H20), end to end: codec + memory pool + fused decode attention kernel + backend integration. The decode kernel is a JIT-compiled port of the FlashMLA three-stage split-KV design (scheduler metadata kernel + persistent WGMMA main kernel + combine kernel), covering all 44 layers (SWA + C4/C128 + attn_sink) in one call.
The PR is self-contained: the kernel and its vendored dependencies live under
python/sglang/kernels/jit/and compile at first use viaload_jit— no sgl-kernel wheel changes required.MXFP4 Layout (368 B/token)
bits << 23)Memory capacity gain vs FP8: ~+37% full-token capacity (bytes/token 7705 → 5609; the NVFP4 reference gains +31.75%).
Decode Kernel (
kernels/jit/csrc/mxfp4_dsv4_decode_sm90/)FlashMLA-style fused split-KV decode for MXFP4:
Entry point:
entry.cuh(TVM-FFI single translation unit, explicit instantiations for h_q 64/128). Python wrapper:kernels/ops/attention/mxfp4_dsv4_decode_sm90.py(FlashMLASchedMeta, per-geometry scratch reuse, C++-side stream resolution viaTVMFFIEnvGetStream).Vendored dependencies (provenance)
params.h,defines.h,utils.h,flashmla_utils.h,kerutils/,combineconfig.h,dequant.h,layout.h,components/helpers.h, splitkv skeletonsplitkv_mla.cuh(MXFP4 variant),entry.cuhCUTLASS/CUTE headers come from the JIT build's
extra_dependencies=["cutlass"](pinned cutlass @ 147f5673, matching the reference PR).kerutilsincludes were converted to relative paths during vendoring.Changes
srt/layers/attention/dsv4/mxfp4_k_cache.py): quantize/dequant for the 368 B row (Triton), paged variantscompressor_v2.py):_forward_mxfp4covering all 44 layers (C0/C4/C128), decode-plan boundary semantics--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4(env alias with deprecation warning); pool sizing/accounting unifieddeepseek_v4_backend.py):_forward_mxfp4_decode_flashmla— one fused call per layer; scheduler metadata shared per geometry across layers, re-created during CUDA-graph capture (top-k lengths are replayed device inputs)test/registered/kernels/ops/attention/test_mxfp4_dsv4_decode.py(17 kernel cases: C0/C4/C128 × flash/profiling geometries, graph replay + length clamping, invalid contracts, scheduler-refresh regressions) +test_mxfp4_k_cache.py(5 codec cases) +test_dsv4_hisparse_pool.py(3 pool-guard cases) +test_dsv4_compress_write_pad.py(3 restored cases); benchmark intest/registered/kernels/benchmark/attention/bench_mxfp4_dsv4_decode.pytest/registered/kernels/e2e/test_mxfp4_dsv4_e2e.py— 8-GPU H200 server + gsm8k, exercising pool configurator → compressor → CUDA-graph capture (CI stage extra-b)mxfp4_decode.cuh+_forward_mxfp4_decode) superseded by the FlashMLA portValidation
DeepSeek-V4-Flash), FP8 (--kv-cache-dtype fp8_e4m3) vs MXFP4 (--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4), identical server flags otherwise:Known gaps / follow-ups
References
Status: Ready for review — kernel + integration complete, end-to-end validated on 8×H20.
CI States
Latest PR Test (Base): ❌ Run #32231903565
Latest PR Test (Extra): ❌ Run #32231903211