Skip to content

[Refactor] Native NVFP4 prefill via FlashInfer kv_cache_sf: retire the FP8 dequant workspace (O(n^2) + OOM fix; tvm-ffi fake-Segfault note) - #43471

Open
chengye75-star wants to merge 2 commits into
sgl-project:mainfrom
chengye75-star:feat/nvfp4-native-prefill
Open

chengye75-star wants to merge 2 commits into
sgl-project:mainfrom
chengye75-star:feat/nvfp4-native-prefill

Conversation

@chengye75-star

@chengye75-star chengye75-star commented Oct 10, 2026 •

Copy link
Copy Markdown

Summary

Routes NVFP4 prefill attention on the FlashInfer backend through FlashInfer's native packed-FP4 paged prefill (kv_data_type=uint8 + run(..., kv_cache_sf=(k_scales, v_scales))), replacing the FP8 dequant-workspace route. The kernel dequantizes block-scaled FP4 tiles on the fly — the FP8 workspace disappears entirely from the prefill path.

This is the follow-up to #43239 (HiCache host-pool geometry, PR #43240). This PR attacks the other NVFP4 structural problem found in the same incident investigation.

Problem

The DEQUANT_WORKSPACE prefill access makes get_flashinfer_dequant_workspace_kv_buffer gather + dequantize the entire cached prefix into the FP8 workspace on every chunked-prefill chunk:

  • Memory: dq_k/dq_v_buffer are allocated per pool row count (not chunk size) — ~146 MiB fixed per TP rank at a 160K pool — and compute_cell_size() charges 1024 B/token into the pool budget on top of that. The gather temporaries inside the dequant step scale with prefix length (bf16 intermediates for a ~45K-token prefix, per layer call).
  • Time: re-dequantizing the whole prefix per chunk is quadratic in chunk count for long prompts.
  • Production impact: a ~45K-token cold prefix OOM-killed the scheduler (CUDA OOM inside dequantize_prev_kv) on 2× RTX 5060 Ti (SM120) with a 160K-token pool.

Changes

  1. fp4_kv_cache_quant_method.py — the registry entry for NVFP4 prefill becomes _native_fp4({"flashinfer", "trtllm_mha"}) (was dequant_workspace for flashinfer). dequant_workspace_dtype() returns None → dq buffers are never allocated → compute_cell_size drops from 10816 to 9792 B/token (+10.5% same-budget pool capacity). Rollback: SGLANG_NVFP4_NATIVE_PREFILL=0 restores the workspace route (registry is built from the env at import).

  2. flashinfer_backend.py:

    • prefill_uses_native_fp4 flag; flashinfer_kv_cache_dtype becomes uint8 (the packed convention the run path expects). The decode tensor-core heuristic keeps receiving float8_e4m3fn so decode routing is unchanged.
    • forward_extend feeds pool.get_raw_kv_buffer() directly: packed [rows, heads, head_dim//2] uint8 KV + token-linear [rows, heads, head_dim//16] e4m3 block scales — on SM120 these are the pool's own tensors (zero copies; SM100's GenMHA-layout native buffers are unaffected and stay on the trtllm route).
    • Paged and ragged-merge call sites use wrapper.run(..., kv_cache_sf=...) with the pool's global FP32 scales as k_scale/v_scale (same pair dequantize_prev_kv uses; mirrors the trtllm_mha native decode route).
    • Split-KV is force-disabled for NVFP4 KV by FlashInfer's own _nvfp4_kv_requires_disabled_split_kv guard.

The "fake Segfault" trap (why the call goes through run() + attribute injection)

BatchPrefillWithPagedKVCacheWrapper.run() does not accept causal / sm_scale / logits_soft_cap; the deprecated forward() shim injects them as wrapper attributes (_causal, _sm_scale, …) before calling run(). Passing them as kwargs to run() raises TypeError inside the tvm-ffi call frame — the C++ unwinder turns it into abort(), and tvm_ffi's crash handler prints a Rust-symbol "Segfault encountered" backtrace, swallows the real error, and prevents any core dump. This cost days of misdirected forensics (SIGSEGV is a red herring: no sanitizer, no gdb attach, no core will ever show a segfault because there is none). _set_paged_forward_attrs() mirrors the forward() contract exactly; the docstring documents the trap.

Validation (Qwen3.8-27B-NVFP4 hybrid, TP2, SM120, HiCache kernel backend + NEXTN MTP)

All on the same machine/battery before vs after:

  • Byte-level: native prefill vs the workspace route's own nvfp4_kv_dequantize primitive agree to bf16 output rounding (maxabs 0.0125); scattered token-slot page tables (page_size=1 radix semantics) gather payload and scale tensors identically (synthetic battery).
  • 4× dual-session HiCache eviction/reload: accept_mean = 3.00 ×15, 2.99 ×2, zero corruption.
  • 3× 45K-token cold prefixes (the exact OOM repro from 10/09): all survived, 13 s/round, GPU memory flat to the MiB across all rounds (workspace route: hundreds of MB transient per layer call).
  • 10× 137K-resident turns at token usage 0.89–0.90 (above the 0.92 waterline the engine died near before): all green, accept 2.28–2.68.
  • Boot headroom: pool-end avail mem improved ~0.3 GB/rank; the prefill transient that killed the previous config is structurally gone.

Notes / scope

  • Requires FlashInfer with the NVFP4 kv_cache_sf prefill support (present in 0.6.18; _nvfp4_kv_requires_disabled_split_kv and the uint8 packed convention are both there).
  • Cross-attention asserts out of the native route (same as the workspace route did).
  • fp4_mx_block16 recipe untouched (it never used the workspace).
  • Speculative verify/draft-extend routing to trtllm native decode is unchanged.

Related: #43239 / PR #43240 (HiCache host pool geometry — independent bug on the same deployment).


CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Route NVFP4 prefill attention through FlashInfer's packed-FP4 paged
prefill (kv_data_type=uint8 + run(kv_cache_sf=...)) instead of the FP8
dequant workspace. The kernel dequantizes block-scaled FP4 tiles on
the fly, so the pool no longer gathers + dequantizes the entire cached
prefix into a workspace on every chunked-prefill chunk.

Why:
- The FP8 dq workspace is allocated per *pool row* and counted into
  compute_cell_size (1024 B/token); retiring it frees ~146 MiB fixed
  per TP rank and grows same-budget pool capacity by ~10%.
- The workspace route re-dequantizes O(prefix) per chunk: transient
  bf16 temporaries scale with prefix length (CUDA OOM on ~45K cold
  prefixes) and total prefill time is quadratic in chunk count.
- Native path consumes exactly the pool's existing tensors on SM120
  (token-linear scale layout), zero copies.

Notes:
- SGLANG_NVFP4_NATIVE_PREFILL=0 restores the workspace route.
- run() does not take causal/sm_scale/logits_soft_cap kwargs; the
  deprecated forward() shim injects them as wrapper attributes. A
  TypeError there aborts through tvm-ffi unwinding and surfaces as a
  bogus 'Segfault' with no core dump - see _set_paged_forward_attrs.
- Global FP32 KV scales go through run(k_scale=/v_scale=), mirroring
  the trtllm_mha native decode route.
- split-KV is force-disabled for NVFP4 KV by FlashInfer's
  _nvfp4_kv_requires_disabled_split_kv guard (qo<<kv corruption).
…ack to the workspace path

CUDA-graph replay of the native kv_cache_sf path is only proven stable for
plain prompt-prefill. Target-verify / draft-extend now fall back to the
dequant workspace route (graph-safe via sgl-project#43356) unless the operator opts
in via SGLANG_NVFP4_NATIVE_SPEC=1 (small capture lists only).
Also plans speculative prefill wrappers with float8_e4m3fn when downgraded.
detain pushed a commit to detain/sglang that referenced this pull request Oct 19, 2026
…tune-gate test fix (b0fce3b), and correct the PCIe-IPC "ACTIVE" claim

Entry 228 records the FI bump in both halves: the clean
cherry-pick of upstream 44be30d/sgl-project#44591 (2 Dockerfile ARGs,
the sole pyproject pin at python/pyproject.toml:42, and the
engine.py:1925 assert_pkg_version FLOOR) and the operator's
prod-venv swap 0.7.1 -> 0.7.2 (one-line freeze diff, no
jit-cache/cubin companions). It carries the 53-module/198-
symbol API audit (1 benign missing symbol), the pin-vs-wheel
reasoning (assert_pkg_version is a FLOOR -- the real hazard
was `uv pip install -e 'python[dev]'` silently downgrading
the wheel back to 0.7.1), the deliberate sglang-kernel 0.4.9
floor with the sgl-project#44590 pairing constraint, the rollback order,
the first-boot cost (0.7.2/120f holds nothing but a 0-byte
log => re-JIT of sampling + XQA, autotune sweep cold; 147 s
baseline), sgl-project#44115 as PERMANENTLY not-applicable (plan
positional list byte-identical, no uniform_q_len), sgl-project#43471 as
never version-gated, the native-ReplaySSM convergence lever
and the new `cake` mm_fp4 backend.

CORRECTION: entries 162-164 (PCIe-IPC vocab gathers) were
never ACTIVE. The wheel satisfies a dependency; the transport
is behind SGLANG_ENABLE_PCIE_IPC_ALLREDUCE = EnvBool(False)
with no CLI flag, unset in the prod run script and absent from
the live process env. Prod TP4 vocab gathers and the bf16
decode all-reduce remain on NCCL; testing it is the top
pending A/B (harness arms pcie_ipc_on / pcie_ipc_on_ll /
pcie_ipc_off_recheck). Annotated in place at all eight stale
claims (header STACK block, 10-15 updated block, FOURTH-merge
synced line, entry 162 bracket, merge-4 entry line, the
merge-4 battery bullet, the entries-200-204 BEHAVIOR WATCH
bullet, the LIVE VENV bullet).

Entry 229 records the wheel-independent autotune-gate fix:
7P/1F -> 9P (stale auto-Mock met entry 225's b12x arm; the
fp8_utils per-tensor patch pins a real production probe, so
it is legitimate, not a workaround; WIP DBG scaffolding
stripped pre-commit; diff is +29/-2, not +31/-2).

Header/CAVEATS: STACK 227 -> 229, tip -> 2b46270, new
updated: block for 228-229, MERGE sgl-project#5 STILL OWED refreshed
(origin/main 716c915, +325), the FI-0.7.2 ADDENDUM to the
pinned-baseline bullet (autotune_gate 8P -> 9P chain, NEW
test_jit_cache 42P, kernels_namespace REMAINS 76P -- the
"78P" claim is false and appears nowhere in this file), a new
top-priority BEHAVIOR WATCH bullet for the first 0.7.2 boot,
and the supersede-in-place annotation on the "NOT TAKEN (a)
FLASHINFER IS NOT BUMPED" bullet (direction-correct,
mechanism-wrong). Push state recorded truthfully: detain
carries the branch up to 3c44685; these commits are
unpushed. stack.txt only; LIVE ONLY AFTER RESTART.
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…mily + S-EXT1 + riders + S-RACE + LLGUARD/ENVREG), S-EXT1 and S-RACE items CLOSED, entry-256's raw-env deviation and LL-vs-PCIe-IPC collision warning both marked CLOSED with the BOOT_DELTA (h') needle in landing terms, sgl-project#35617 left as the sole FAM-A7 survivor, sgl-project#44732 re-pointed to the operator's merge-sgl-project#5 answer, and the KV-ROUTE DECISION RECORD added (HOLD NVFP4-KV until armed PCIe-IPC is measured post-merge; preferred-if-ever sequence merge -> sgl-project#43356-delta -> sgl-project#46452 NOT sgl-project#43240 -> optional sgl-project#43471 -> fork BUILD-A QSA FP4 reads (template sgl-project#38724) -> BUILD-B L3 scale sidecar -> one GPU window; sgl-project#46141 template, sgl-project#44462 NEXT, MXFP4 declined, entry-247 re-sync watch-string modelopt_e4m3_k32)
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
… C closure -- the post-merge port queue is now decided

ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE +
head-sha comparison for EVERY deferred item. Verdicts:
  RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B
    cherry is now redundant; behavior note rides the restart).
  CLOSED (C-items whose PRs arrived and whose integration this stack
    performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight
    precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/
    sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/
    sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard),
    C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle),
    C0a (arrival shas already recorded at entry 310 -- Section C now
    carries NO open code-integration items: C6/C10 retain only their
    explicitly-named docs/boot-measurement remainders, re-stated).
  STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to
    #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale
    bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617
    (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in
    range).
  HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/
    sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the
    sequence may be re-visited ONLY with the post-merge PCIe-IPC
    measurement at an operator sitting. sgl-project#44732: the 'one purge if done
    WITH the merge' premise LAPSED when merge sgl-project#5 completed without it
    -- later execution costs the SAME single purge once; no penalty,
    no haste. Recorded for the operator, not decided by B6.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant