Skip to content

[Quantization] Serve NVFP4-packed QAD checkpoints (PLE NVFP4 + modelopt_mixed MXFP8) - #41624

Open
L14nY1Wang wants to merge 2 commits into
sgl-project:mainfrom
L14nY1Wang:ple-bf16-hybrid-upstream
Open

L14nY1Wang wants to merge 2 commits into
sgl-project:mainfrom
L14nY1Wang:ple-bf16-hybrid-upstream

Conversation

@L14nY1Wang

@L14nY1Wang L14nY1Wang commented Sep 29, 2026 •

Copy link
Copy Markdown

Motivation

ModelOpt QAD packages such as local-inference-lab/Qwen3.8-Flash-Next-NVFP4 cannot be served as shipped today. Four independent gaps force users into either a patched fork or a ~107 GB offline re-export:

  1. modelopt_mixed renames every *.weight_scale to *.weight_scale_inv. Only the layers that resolve to MXFP8 own a weight_scale_inv; FP8 / NVFP4 / W4A16 layers keep a real per-tensor or per-group scale, which the blanket rename drops.
  2. The vision tower is keyed model.visual.* in the package, while visual.* is what some VL models pass to get_quant_method. Neither spelling resolves without a bidirectional candidate.
  3. Every MXFP8 dense backend requires n >= 128 and k >= 128. GDN's in_proj_ba is N=24 per TP4 rank (96 in total, so even TP1 fails), so MXFP8 checkpoints raise on the first forward unless those layers are re-materialized offline — the current workaround dequantizes 467 dense tensors (395 MXFP8 + the 72 N=24 projections) into a separate bf16 checkpoint.
  4. The PLE n-gram table is stored NVFP4-packed — U8 e2m1 codes [rows, D/2], e4m3 16-lane block scales [rows, D/16] and one fp32 global scale per table — declared by text_config.ple_embedding_dtype = "nvfp4". The PLE loader (shard regex, scale handling) and the offload gather know only bf16/fp8 tables.

Modifications

  1. model_loader/loader.py — keep the weight_scale → weight_scale_inv rename for mxfp8 / modelopt_mixed, but apply it only to layers whose prefix resolves to MXFP8 via resolve_quant_algo. Checkpoint names stay per-shard (q_proj, k_proj, ...) because the stacked-params mapping fuses them afterwards, so this cannot look the name up in params_dict the way the kimi_k3 / glm5_next helpers do.
  2. modelopt_quant.py — _quantized_layer_prefix_candidates gains a visual.* ↔ model.visual.* mapping so the vision tower resolves under either spelling.
  3. fp8.py — Fp8LinearMethod._dequantize_mxfp8_small_shape_to_bf16 runs MXFP8 dense layers below the kernels' minimum as a plain bf16 GEMM: serialized checkpoints fold their ue8m0 scales into the weight once at load time, online MXFP8 simply skips quantization, and apply short-circuits to F.linear. A logger.warning_once announces the fallback.
  4. models/qwen4_exp.py — NVFP4-packed PLE tables: a uint8 weight and an e4m3 block-scale buffer sized to this rank's TP shard (exactly like the bf16/fp8 tables they replace), a Triton gather that decodes e2m1 × 16-lane block scale × fp32 global scale straight out of the pinned-host offload table, shard_N.weight_scale rows loaded through the existing shard machinery, and the global scale loaded through _load_qwen4_exp_ple_buffer. NVFP4 tables are served from the offload table only (--ple-offload-embedding).
  5. Tests — real-kernel NVFP4 gather cases in test/registered/kernels/ops/embeddings/test_qwen4_ple_offload.py (kernel output vs the existing dequantize_nvfp4 reference, shard boundaries, zeroed out-of-shard rows) and a positive/negative pair for the small-shape fallback in test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py.

Accuracy Tests

local-inference-lab/Qwen3.8-Flash-Next-NVFP4 served as shipped, TP4/EP4:

eval result
GSM8K, 1319 questions, greedy 8-shot CoT 1288/1319 = 97.6% strict (1290 = 97.8% flexible; 11 answers hit the 2048-token cap)
GSM8K 32-question smoke, both arms served end-to-end 31/32 each, the same single failing question
NVIDIA model card, FP8 vs NVFP4 over 9 benchmarks parity (e.g. GPQA-Diamond 92.0 / 91.5, AA-LCR 71.9 / 74.1)

Speed Tests and Profiling

Setup — 4× RTX 5090 (SM120), TP4/EP4, one identical command line for every arm:

sglang serve --tensor-parallel-size 4 --ep-size 4 \
  --moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cutlass \
  --ple-offload-embedding --page-size 64 --chunked-prefill-size 2048 \
  --context-length 16384 --mamba-radix-cache-strategy extra_buffer_lazy \
  --mamba-ssm-dtype bfloat16 --max-running-requests 8 --max-mamba-cache-size 24 \
  --mem-fraction-static 0.90

vllm bench serve, random dataset 1024±95% in / 256±95% out, seed 42, concurrency ladder 1/4/8/32, 2×concurrency prompts per rung. Note the engine caps concurrency at 6 here (mamba state cache: max_mamba_cache_size=24, 4 state slots per request), so the c8/c32 rungs measure prefill-bound queue drain rather than decode scaling.

arm dense recipe PLE table c1 c4 c8 c32 tok/s c32 ITL p50 c1 TTFT mean
this PR, QAD package as shipped native MXFP8 NVFP4 106.66 270.81 381.81 386.56 9.79 ms 335 ms
same + SGLANG_FLASHINFER_AUTOTUNE_EXTEND=1 native MXFP8 NVFP4 106.68 272.31 389.69 393.05 9.69 ms 232 ms
nvidia/Qwen3.8-Flash-Next-NVFP4 (reference) NVFP4 FP8 113.96 286.20 423.68 426.75 9.67 ms 182 ms

Decode is at parity across the measured arms (c32 ITL p50 within 1.2%). The only difference is prefill-bound throughput, and it is not caused by the loader or gather changes in this PR:

  • Measured on the production call path (flashinfer_mxfp8_blockscaled_linear(backend="cutlass"), 204 per-rank MXFP8 dense layers, M=1024): the native MXFP8 stack costs 29.5 ms of host time per forward vs 2.34 ms for the bf16 stack (~145 µs vs ~11.5 µs per layer), while its GPU work is 2.4× cheaper — CUDA-graph replay is 3.65 ms vs 8.73 ms. Eager wall time (29.7 ms) is therefore almost entirely host-bound.
  • Prefill runs eager in every arm (prefill.backend='disabled'), so that host cost lands directly in TTFT; decode hides it inside the decode CUDA graph.
  • The package runs 204 MXFP8 dense linears per TP4 rank per forward (its other 72 MXFP8-checkpoint linears are the N=24 in_proj_{a,b} projections, which take the bf16 fallback; it has no nextn layers and the vision tower is unused for text prefill). Summed over those 204 shapes, MXFP8 wall is ≈30.1 ms vs ≈8.8 ms for the same linears in bf16 per 1024-token forward — so ≈21 ms of the ≈50 ms TTFT gap (232 ms vs 182 ms for the reference export) is the MXFP8 dense host path; the balance is not attributed per-operator yet. The MoE recipe is not a factor: both packages run NVFP4 experts for the language model (the package's 29 W4A16_NVFP4 layers are all in the vision tower).

Two consequences worth stating explicitly:

  • SGLANG_FLASHINFER_AUTOTUNE_EXTEND=1 (an existing opt-in) pre-tunes the large-M buckets at warmup — the autotune cache grows from 24 to 96 mxfp8_gemm records — which removes the first-request tuning tail (c1 mean TTFT 335 → 232 ms) and buys +1.7% at c32. The remaining gap is the per-layer eager dispatch cost above.
  • This MXFP8 eager-dispatch overhead is pre-existing SGLang behaviour, reachable by any MXFP8-dense model on this backend. It is orthogonal to this PR; the PR only makes those layers usable for this package at all.

Memory, disk and load (measured, same run):

arm package on disk load weights/card KV tokens host pinned / card
this PR 99 GB 49.6 s 20.86 GB 650,688 18.0 GB
nvidia official export 124 GB 58.2 s 21.57 GB 554,432 31.5 GB

Serving the NVFP4-packed PLE table where it lives needs 25 GB less disk, 8.6 s less load time and 13.5 GB less host RAM per rank than the official export, and keeps +17% KV cache on the same cards. Host figures are VmHWM summed over the four TP workers, so they include a ~11 GB/rank process baseline.

Checklist

  • Format your code according to the format-code guidance (ruff 0.15.1, the version pinned by .pre-commit-config.yaml: ruff check --select=F401,F821,UP037 and ruff format --diff both clean on all touched files).
  • Add unit tests: test/registered/kernels/ops/embeddings/test_qwen4_ple_offload.py (new NVFP4 cases; 14 passed / 3 skipped locally), test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py (14 cases in the class), and test.registered.unit.models.test_qwen4_exp_ple_table (21 passed / 1 skipped); all three unaffected.
  • Update documentation — no user-facing flag or docs page changes; the only new constraint (--ple-offload-embedding is required for NVFP4-packed PLE tables) is enforced with an explicit NotImplementedError.
  • Provide accuracy and speed benchmark results — see the sections above.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #36526842974
Latest PR Test (Extra): ❌ Run #36526842725
Latest PR Test (AMD ROCm 10): ❌ Run #36526842940

@github-actions github-actions Bot added the quant LLM Quantization label Sep 29, 2026
@L14nY1Wang
L14nY1Wang force-pushed the ple-bf16-hybrid-upstream branch from 4cc04b8 to 3d221c9 Compare September 29, 2026 04:17
@L14nY1Wang
L14nY1Wang marked this pull request as ready for review September 29, 2026 04:18
…pt_mixed MXFP8)

Unblock local-inference-lab/Qwen3.8-Flash-Next-NVFP4 style QAD packages, which
ship packed PLE tables and MXFP8 ue8m0 block scales that today's loaders and
dense kernels reject, without offline checkpoint surgery:

1. modelopt_mixed loader: rename weight_scale -> weight_scale_inv only for
   layers that resolve to MXFP8 in quantized_layers; the FP8/NVFP4/W4A16
   per-tensor and group scales keep their names. Checkpoint names stay
   per-shard (q_proj, k_proj, ...); the stacked params mapping fuses them
   afterwards, so a params_dict lookup cannot be used here.
2. ModelOptMixedPrecisionConfig prefix candidates: map visual.* <->
   model.visual.* so the vision tower resolves its quant method under either
   spelling.
3. Fp8LinearMethod: MXFP8 dense layers below the n >= 128 and k >= 128
   minimum every dense backend requires (GDN in_proj_ba is N=24 per TP4
   rank) dequantize to bf16 once at load time and run a plain bf16 GEMM
   instead of raising on the first forward.
4. Qwen4-Exp PLE: support ple_embedding_dtype=nvfp4 packed tables (U8 e2m1
   codes, e4m3 16-lane block scales, one fp32 global scale). A Triton gather
   decodes rows straight out of the pinned-host offload table; sharded weight
   and scale rows load through the existing shard machinery. The packed
   tables are sized to this rank's TP shard, exactly like the bf16/fp8
   tables they replace, and are served from the offload table only
   (--ple-offload-embedding).

Verified on the lab package (TP4/EP4): GSM8K 1319 strict 97.2%, and the
packed tables now cost 6.7 GiB of host memory per rank instead of 28.8 GiB.
@L14nY1Wang
L14nY1Wang force-pushed the ple-bf16-hybrid-upstream branch from 3d221c9 to 4c5e1b9 Compare September 29, 2026 04:22
@AndreasBis

Copy link
Copy Markdown

FP4-Local (nvidia/Qwen3.8-Flash-Next-FP4-Local) needs both this PR and #46153: #46153 decodes its CSF-compressed NVFP4 scales and the vision tower at load, and this PR adds the NVFP4 PLE table and MXFP8 layers. End-to-end results are in #46153.

detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
… C closure -- the post-merge port queue is now decided

ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE +
head-sha comparison for EVERY deferred item. Verdicts:
  RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B
    cherry is now redundant; behavior note rides the restart).
  CLOSED (C-items whose PRs arrived and whose integration this stack
    performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight
    precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/
    sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/
    sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard),
    C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle),
    C0a (arrival shas already recorded at entry 310 -- Section C now
    carries NO open code-integration items: C6/C10 retain only their
    explicitly-named docs/boot-measurement remainders, re-stated).
  STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to
    #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale
    bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617
    (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in
    range).
  HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/
    sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the
    sequence may be re-visited ONLY with the post-merge PCIe-IPC
    measurement at an operator sitting. sgl-project#44732: the 'one purge if done
    WITH the merge' premise LAPSED when merge sgl-project#5 completed without it
    -- later execution costs the SAME single purge once; no penalty,
    no haste. Recorded for the operator, not decided by B6.
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants