Repository navigation
[Refactor] Native NVFP4 prefill via FlashInfer kv_cache_sf: retire the FP8 dequant workspace (O(n^2) + OOM fix; tvm-ffi fake-Segfault note) - #43471
Open
chengye75-star wants to merge 2 commits into
Conversation
chengye75-star
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
OrangeRedeng,
Qiaolin-Yu,
b8zhong,
ch-wan,
hebiao064,
ispobock,
merrymercy and
mmangkad
as code owners
October 10, 2026 02:23
Route NVFP4 prefill attention through FlashInfer's packed-FP4 paged prefill (kv_data_type=uint8 + run(kv_cache_sf=...)) instead of the FP8 dequant workspace. The kernel dequantizes block-scaled FP4 tiles on the fly, so the pool no longer gathers + dequantizes the entire cached prefix into a workspace on every chunked-prefill chunk. Why: - The FP8 dq workspace is allocated per *pool row* and counted into compute_cell_size (1024 B/token); retiring it frees ~146 MiB fixed per TP rank and grows same-budget pool capacity by ~10%. - The workspace route re-dequantizes O(prefix) per chunk: transient bf16 temporaries scale with prefix length (CUDA OOM on ~45K cold prefixes) and total prefill time is quadratic in chunk count. - Native path consumes exactly the pool's existing tensors on SM120 (token-linear scale layout), zero copies. Notes: - SGLANG_NVFP4_NATIVE_PREFILL=0 restores the workspace route. - run() does not take causal/sm_scale/logits_soft_cap kwargs; the deprecated forward() shim injects them as wrapper attributes. A TypeError there aborts through tvm-ffi unwinding and surfaces as a bogus 'Segfault' with no core dump - see _set_paged_forward_attrs. - Global FP32 KV scales go through run(k_scale=/v_scale=), mirroring the trtllm_mha native decode route. - split-KV is force-disabled for NVFP4 KV by FlashInfer's _nvfp4_kv_requires_disabled_split_kv guard (qo<<kv corruption).
chengye75-star
force-pushed
the
feat/nvfp4-native-prefill
branch
from
October 10, 2026 02:24
bd79af8 to
4eae8ca
Compare
4 of 5 tasks
This was referenced Oct 10, 2026
…ack to the workspace path CUDA-graph replay of the native kv_cache_sf path is only proven stable for plain prompt-prefill. Target-verify / draft-extend now fall back to the dequant workspace route (graph-safe via sgl-project#43356) unless the operator opts in via SGLANG_NVFP4_NATIVE_SPEC=1 (small capture lists only). Also plans speculative prefill wrappers with float8_e4m3fn when downgraded.
detain
pushed a commit
to detain/sglang
that referenced
this pull request
Oct 19, 2026
…tune-gate test fix (b0fce3b), and correct the PCIe-IPC "ACTIVE" claim Entry 228 records the FI bump in both halves: the clean cherry-pick of upstream 44be30d/sgl-project#44591 (2 Dockerfile ARGs, the sole pyproject pin at python/pyproject.toml:42, and the engine.py:1925 assert_pkg_version FLOOR) and the operator's prod-venv swap 0.7.1 -> 0.7.2 (one-line freeze diff, no jit-cache/cubin companions). It carries the 53-module/198- symbol API audit (1 benign missing symbol), the pin-vs-wheel reasoning (assert_pkg_version is a FLOOR -- the real hazard was `uv pip install -e 'python[dev]'` silently downgrading the wheel back to 0.7.1), the deliberate sglang-kernel 0.4.9 floor with the sgl-project#44590 pairing constraint, the rollback order, the first-boot cost (0.7.2/120f holds nothing but a 0-byte log => re-JIT of sampling + XQA, autotune sweep cold; 147 s baseline), sgl-project#44115 as PERMANENTLY not-applicable (plan positional list byte-identical, no uniform_q_len), sgl-project#43471 as never version-gated, the native-ReplaySSM convergence lever and the new `cake` mm_fp4 backend. CORRECTION: entries 162-164 (PCIe-IPC vocab gathers) were never ACTIVE. The wheel satisfies a dependency; the transport is behind SGLANG_ENABLE_PCIE_IPC_ALLREDUCE = EnvBool(False) with no CLI flag, unset in the prod run script and absent from the live process env. Prod TP4 vocab gathers and the bf16 decode all-reduce remain on NCCL; testing it is the top pending A/B (harness arms pcie_ipc_on / pcie_ipc_on_ll / pcie_ipc_off_recheck). Annotated in place at all eight stale claims (header STACK block, 10-15 updated block, FOURTH-merge synced line, entry 162 bracket, merge-4 entry line, the merge-4 battery bullet, the entries-200-204 BEHAVIOR WATCH bullet, the LIVE VENV bullet). Entry 229 records the wheel-independent autotune-gate fix: 7P/1F -> 9P (stale auto-Mock met entry 225's b12x arm; the fp8_utils per-tensor patch pins a real production probe, so it is legitimate, not a workaround; WIP DBG scaffolding stripped pre-commit; diff is +29/-2, not +31/-2). Header/CAVEATS: STACK 227 -> 229, tip -> 2b46270, new updated: block for 228-229, MERGE sgl-project#5 STILL OWED refreshed (origin/main 716c915, +325), the FI-0.7.2 ADDENDUM to the pinned-baseline bullet (autotune_gate 8P -> 9P chain, NEW test_jit_cache 42P, kernels_namespace REMAINS 76P -- the "78P" claim is false and appears nowhere in this file), a new top-priority BEHAVIOR WATCH bullet for the first 0.7.2 boot, and the supersede-in-place annotation on the "NOT TAKEN (a) FLASHINFER IS NOT BUMPED" bullet (direction-correct, mechanism-wrong). Push state recorded truthfully: detain carries the branch up to 3c44685; these commits are unpushed. stack.txt only; LIVE ONLY AFTER RESTART.
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…mily + S-EXT1 + riders + S-RACE + LLGUARD/ENVREG), S-EXT1 and S-RACE items CLOSED, entry-256's raw-env deviation and LL-vs-PCIe-IPC collision warning both marked CLOSED with the BOOT_DELTA (h') needle in landing terms, sgl-project#35617 left as the sole FAM-A7 survivor, sgl-project#44732 re-pointed to the operator's merge-sgl-project#5 answer, and the KV-ROUTE DECISION RECORD added (HOLD NVFP4-KV until armed PCIe-IPC is measured post-merge; preferred-if-ever sequence merge -> sgl-project#43356-delta -> sgl-project#46452 NOT sgl-project#43240 -> optional sgl-project#43471 -> fork BUILD-A QSA FP4 reads (template sgl-project#38724) -> BUILD-B L3 scale sidecar -> one GPU window; sgl-project#46141 template, sgl-project#44462 NEXT, MXFP4 declined, entry-247 re-sync watch-string modelopt_e4m3_k32)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
… C closure -- the post-merge port queue is now decided ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE + head-sha comparison for EVERY deferred item. Verdicts: RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B cherry is now redundant; behavior note rides the restart). CLOSED (C-items whose PRs arrived and whose integration this stack performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/ sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/ sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard), C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle), C0a (arrival shas already recorded at entry 310 -- Section C now carries NO open code-integration items: C6/C10 retain only their explicitly-named docs/boot-measurement remainders, re-stated). STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617 (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in range). HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/ sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the sequence may be re-visited ONLY with the post-merge PCIe-IPC measurement at an operator sitting. sgl-project#44732: the 'one purge if done WITH the merge' premise LAPSED when merge sgl-project#5 completed without it -- later execution costs the SAME single purge once; no penalty, no haste. Recorded for the operator, not decided by B6.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Routes NVFP4 prefill attention on the FlashInfer backend through FlashInfer's native packed-FP4 paged prefill (
kv_data_type=uint8+run(..., kv_cache_sf=(k_scales, v_scales))), replacing the FP8 dequant-workspace route. The kernel dequantizes block-scaled FP4 tiles on the fly — the FP8 workspace disappears entirely from the prefill path.This is the follow-up to #43239 (HiCache host-pool geometry, PR #43240). This PR attacks the other NVFP4 structural problem found in the same incident investigation.
Problem
The
DEQUANT_WORKSPACEprefill access makesget_flashinfer_dequant_workspace_kv_buffergather + dequantize the entire cached prefix into the FP8 workspace on every chunked-prefill chunk:dq_k/dq_v_bufferare allocated per pool row count (not chunk size) — ~146 MiB fixed per TP rank at a 160K pool — andcompute_cell_size()charges 1024 B/token into the pool budget on top of that. The gather temporaries inside the dequant step scale with prefix length (bf16 intermediates for a ~45K-token prefix, per layer call).dequantize_prev_kv) on 2× RTX 5060 Ti (SM120) with a 160K-token pool.Changes
fp4_kv_cache_quant_method.py— the registry entry for NVFP4 prefill becomes_native_fp4({"flashinfer", "trtllm_mha"})(wasdequant_workspacefor flashinfer).dequant_workspace_dtype()returnsNone→ dq buffers are never allocated →compute_cell_sizedrops from 10816 to 9792 B/token (+10.5% same-budget pool capacity). Rollback:SGLANG_NVFP4_NATIVE_PREFILL=0restores the workspace route (registry is built from the env at import).flashinfer_backend.py:prefill_uses_native_fp4flag;flashinfer_kv_cache_dtypebecomesuint8(the packed convention the run path expects). The decode tensor-core heuristic keeps receivingfloat8_e4m3fnso decode routing is unchanged.forward_extendfeedspool.get_raw_kv_buffer()directly: packed[rows, heads, head_dim//2]uint8 KV + token-linear[rows, heads, head_dim//16]e4m3 block scales — on SM120 these are the pool's own tensors (zero copies; SM100's GenMHA-layout native buffers are unaffected and stay on the trtllm route).wrapper.run(..., kv_cache_sf=...)with the pool's global FP32 scales ask_scale/v_scale(same pairdequantize_prev_kvuses; mirrors the trtllm_mha native decode route)._nvfp4_kv_requires_disabled_split_kvguard.The "fake Segfault" trap (why the call goes through
run()+ attribute injection)BatchPrefillWithPagedKVCacheWrapper.run()does not acceptcausal/sm_scale/logits_soft_cap; the deprecatedforward()shim injects them as wrapper attributes (_causal,_sm_scale, …) before callingrun(). Passing them as kwargs torun()raisesTypeErrorinside the tvm-ffi call frame — the C++ unwinder turns it intoabort(), and tvm_ffi's crash handler prints a Rust-symbol "Segfault encountered" backtrace, swallows the real error, and prevents any core dump. This cost days of misdirected forensics (SIGSEGV is a red herring: no sanitizer, no gdb attach, no core will ever show a segfault because there is none)._set_paged_forward_attrs()mirrors theforward()contract exactly; the docstring documents the trap.Validation (Qwen3.8-27B-NVFP4 hybrid, TP2, SM120, HiCache kernel backend + NEXTN MTP)
All on the same machine/battery before vs after:
nvfp4_kv_dequantizeprimitive agree to bf16 output rounding (maxabs 0.0125); scattered token-slot page tables (page_size=1 radix semantics) gather payload and scale tensors identically (synthetic battery).accept_mean= 3.00 ×15, 2.99 ×2, zero corruption.avail memimproved ~0.3 GB/rank; the prefill transient that killed the previous config is structurally gone.Notes / scope
kv_cache_sfprefill support (present in 0.6.18;_nvfp4_kv_requires_disabled_split_kvand the uint8 packed convention are both there).fp4_mx_block16recipe untouched (it never used the workspace).Related: #43239 / PR #43240 (HiCache host pool geometry — independent bug on the same deployment).
CI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: