Repository navigation
[Quantization] Serve NVFP4-packed QAD checkpoints (PLE NVFP4 + modelopt_mixed MXFP8) - #41624
Open
L14nY1Wang wants to merge 2 commits into
Open
L14nY1Wang wants to merge 2 commits into
L14nY1Wang wants to merge 2 commits into
Conversation
L14nY1Wang
force-pushed
the
ple-bf16-hybrid-upstream
branch
from
September 29, 2026 04:17
4cc04b8 to
3d221c9
Compare
L14nY1Wang
marked this pull request as ready for review
September 29, 2026 04:18
L14nY1Wang
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
HaiShaw,
OrangeRedeng,
b8zhong,
ch-wan and
mmangkad
as code owners
September 29, 2026 04:18
…pt_mixed MXFP8) Unblock local-inference-lab/Qwen3.8-Flash-Next-NVFP4 style QAD packages, which ship packed PLE tables and MXFP8 ue8m0 block scales that today's loaders and dense kernels reject, without offline checkpoint surgery: 1. modelopt_mixed loader: rename weight_scale -> weight_scale_inv only for layers that resolve to MXFP8 in quantized_layers; the FP8/NVFP4/W4A16 per-tensor and group scales keep their names. Checkpoint names stay per-shard (q_proj, k_proj, ...); the stacked params mapping fuses them afterwards, so a params_dict lookup cannot be used here. 2. ModelOptMixedPrecisionConfig prefix candidates: map visual.* <-> model.visual.* so the vision tower resolves its quant method under either spelling. 3. Fp8LinearMethod: MXFP8 dense layers below the n >= 128 and k >= 128 minimum every dense backend requires (GDN in_proj_ba is N=24 per TP4 rank) dequantize to bf16 once at load time and run a plain bf16 GEMM instead of raising on the first forward. 4. Qwen4-Exp PLE: support ple_embedding_dtype=nvfp4 packed tables (U8 e2m1 codes, e4m3 16-lane block scales, one fp32 global scale). A Triton gather decodes rows straight out of the pinned-host offload table; sharded weight and scale rows load through the existing shard machinery. The packed tables are sized to this rank's TP shard, exactly like the bf16/fp8 tables they replace, and are served from the offload table only (--ple-offload-embedding). Verified on the lab package (TP4/EP4): GSM8K 1319 strict 97.2%, and the packed tables now cost 6.7 GiB of host memory per rank instead of 28.8 GiB.
L14nY1Wang
force-pushed
the
ple-bf16-hybrid-upstream
branch
from
September 29, 2026 04:22
3d221c9 to
4c5e1b9
Compare
5 tasks done
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
… C closure -- the post-merge port queue is now decided ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE + head-sha comparison for EVERY deferred item. Verdicts: RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B cherry is now redundant; behavior note rides the restart). CLOSED (C-items whose PRs arrived and whose integration this stack performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/ sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/ sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard), C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle), C0a (arrival shas already recorded at entry 310 -- Section C now carries NO open code-integration items: C6/C10 retain only their explicitly-named docs/boot-measurement remainders, re-stated). STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617 (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in range). HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/ sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the sequence may be re-visited ONLY with the post-merge PCIe-IPC measurement at an operator sitting. sgl-project#44732: the 'one purge if done WITH the merge' premise LAPSED when merge sgl-project#5 completed without it -- later execution costs the SAME single purge once; no penalty, no haste. Recorded for the operator, not decided by B6.
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ModelOpt QAD packages such as
local-inference-lab/Qwen3.8-Flash-Next-NVFP4cannot be served as shipped today. Four independent gaps force users into either a patched fork or a ~107 GB offline re-export:modelopt_mixedrenames every*.weight_scaleto*.weight_scale_inv. Only the layers that resolve to MXFP8 own aweight_scale_inv; FP8 / NVFP4 / W4A16 layers keep a real per-tensor or per-group scale, which the blanket rename drops.model.visual.*in the package, whilevisual.*is what some VL models pass toget_quant_method. Neither spelling resolves without a bidirectional candidate.n >= 128andk >= 128. GDN'sin_proj_bais N=24 per TP4 rank (96 in total, so even TP1 fails), so MXFP8 checkpoints raise on the first forward unless those layers are re-materialized offline — the current workaround dequantizes 467 dense tensors (395 MXFP8 + the 72 N=24 projections) into a separate bf16 checkpoint.U8e2m1 codes[rows, D/2],e4m316-lane block scales[rows, D/16]and one fp32 global scale per table — declared bytext_config.ple_embedding_dtype = "nvfp4". The PLE loader (shard regex, scale handling) and the offload gather know only bf16/fp8 tables.Modifications
model_loader/loader.py— keep theweight_scale → weight_scale_invrename formxfp8/modelopt_mixed, but apply it only to layers whose prefix resolves to MXFP8 viaresolve_quant_algo. Checkpoint names stay per-shard (q_proj,k_proj, ...) because the stacked-params mapping fuses them afterwards, so this cannot look the name up inparams_dictthe way thekimi_k3/glm5_nexthelpers do.modelopt_quant.py—_quantized_layer_prefix_candidatesgains avisual.* ↔ model.visual.*mapping so the vision tower resolves under either spelling.fp8.py—Fp8LinearMethod._dequantize_mxfp8_small_shape_to_bf16runs MXFP8 dense layers below the kernels' minimum as a plain bf16 GEMM: serialized checkpoints fold their ue8m0 scales into the weight once at load time, online MXFP8 simply skips quantization, andapplyshort-circuits toF.linear. Alogger.warning_onceannounces the fallback.models/qwen4_exp.py— NVFP4-packed PLE tables: auint8weight and ane4m3block-scale buffer sized to this rank's TP shard (exactly like the bf16/fp8 tables they replace), a Triton gather that decodese2m1 × 16-lane block scale × fp32 global scalestraight out of the pinned-host offload table,shard_N.weight_scalerows loaded through the existing shard machinery, and the global scale loaded through_load_qwen4_exp_ple_buffer. NVFP4 tables are served from the offload table only (--ple-offload-embedding).test/registered/kernels/ops/embeddings/test_qwen4_ple_offload.py(kernel output vs the existingdequantize_nvfp4reference, shard boundaries, zeroed out-of-shard rows) and a positive/negative pair for the small-shape fallback intest/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py.Accuracy Tests
local-inference-lab/Qwen3.8-Flash-Next-NVFP4served as shipped, TP4/EP4:Speed Tests and Profiling
Setup — 4× RTX 5090 (SM120), TP4/EP4, one identical command line for every arm:
vllm bench serve, random dataset 1024±95% in / 256±95% out, seed 42, concurrency ladder 1/4/8/32, 2×concurrency prompts per rung. Note the engine caps concurrency at 6 here (mamba state cache:max_mamba_cache_size=24, 4 state slots per request), so the c8/c32 rungs measure prefill-bound queue drain rather than decode scaling.SGLANG_FLASHINFER_AUTOTUNE_EXTEND=1nvidia/Qwen3.8-Flash-Next-NVFP4(reference)Decode is at parity across the measured arms (c32 ITL p50 within 1.2%). The only difference is prefill-bound throughput, and it is not caused by the loader or gather changes in this PR:
flashinfer_mxfp8_blockscaled_linear(backend="cutlass"), 204 per-rank MXFP8 dense layers, M=1024): the native MXFP8 stack costs 29.5 ms of host time per forward vs 2.34 ms for the bf16 stack (~145 µs vs ~11.5 µs per layer), while its GPU work is 2.4× cheaper — CUDA-graph replay is 3.65 ms vs 8.73 ms. Eager wall time (29.7 ms) is therefore almost entirely host-bound.prefill.backend='disabled'), so that host cost lands directly in TTFT; decode hides it inside the decode CUDA graph.in_proj_{a,b}projections, which take the bf16 fallback; it has no nextn layers and the vision tower is unused for text prefill). Summed over those 204 shapes, MXFP8 wall is ≈30.1 ms vs ≈8.8 ms for the same linears in bf16 per 1024-token forward — so ≈21 ms of the ≈50 ms TTFT gap (232 ms vs 182 ms for the reference export) is the MXFP8 dense host path; the balance is not attributed per-operator yet. The MoE recipe is not a factor: both packages run NVFP4 experts for the language model (the package's 29W4A16_NVFP4layers are all in the vision tower).Two consequences worth stating explicitly:
SGLANG_FLASHINFER_AUTOTUNE_EXTEND=1(an existing opt-in) pre-tunes the large-M buckets at warmup — the autotune cache grows from 24 to 96mxfp8_gemmrecords — which removes the first-request tuning tail (c1 mean TTFT 335 → 232 ms) and buys +1.7% at c32. The remaining gap is the per-layer eager dispatch cost above.Memory, disk and load (measured, same run):
Serving the NVFP4-packed PLE table where it lives needs 25 GB less disk, 8.6 s less load time and 13.5 GB less host RAM per rank than the official export, and keeps +17% KV cache on the same cards. Host figures are
VmHWMsummed over the four TP workers, so they include a ~11 GB/rank process baseline.Checklist
.pre-commit-config.yaml:ruff check --select=F401,F821,UP037andruff format --diffboth clean on all touched files).test/registered/kernels/ops/embeddings/test_qwen4_ple_offload.py(new NVFP4 cases; 14 passed / 3 skipped locally),test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py(14 cases in the class), andtest.registered.unit.models.test_qwen4_exp_ple_table(21 passed / 1 skipped); all three unaffected.--ple-offload-embeddingis required for NVFP4-packed PLE tables) is enforced with an explicitNotImplementedError.CI States
Latest PR Test (Base): ❌ Run #36526842974
Latest PR Test (Extra): ❌ Run #36526842725
Latest PR Test (AMD ROCm 10): ❌ Run #36526842940