Skip to content

[sglang-miles] Add NVFP4 MoE weight checker support - #42283

Open
yueming-yuan wants to merge 37 commits into
sgl-project:sglang-milesfrom
yueming-yuan:yueming/nvfp4-checker
Open

yueming-yuan wants to merge 37 commits into
sgl-project:sglang-milesfrom
yueming-yuan:yueming/nvfp4-checker

Conversation

@yueming-yuan

@yueming-yuan yueming-yuan commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

NVFP4 MoE weights currently either compare raw storage (TRTLLM) or reject weight checking (CuteDSL). This makes equivalent scale encodings fail comparison and prevents checking the CuteDSL standard path after a weight update.

Decode TRTLLM and CuteDSL standard NVFP4 expert layouts into comparable values using the packed FP4 weights, block scales, and global scales. Check execution alphas and separately stored swizzled/MMA scales alongside their weights so stale derived parameters are detected. Existing quantization-error tolerance remains opt-in.

Validation:

  • pre-commit run --all-files: passed.
  • test/registered/e2e/rl/test_weight_checker_nvfp4.py: 5 tests passed on one B200 with PyTorch 2.13.0+cu130 and FlashInfer 0.6.18. Covers real backend layouts, equivalent scale encodings, corruption, CPU snapshots/chunking, and separate padded scales.
  • Existing test_weight_checker.py and test_weight_checker_comparator.py: 75 tests passed on the same B200.
  • CPU regressions: all 192 tests across four files passed against e159d0251060f9d476d2c4b0e17984cea8b95167; includes NVFP4 configuration, MoE dispatch/scales, and runtime context.
  • Targeted TE 4over6 probe on the B200: quantized the same seeded BF16 tensor (2 x 256 x 256) with regular NVFP4 and 4over6 using E4M3 maxima 448 and 256. QDQ strict equality failed as expected (max absolute differences 0.78125 and 1.0), while both had zero elements beyond combined ULP tolerance. Injected sign-bit corruption exceeded tolerance at 857 and 867 elements respectively. This is a small numerical probe, not model-level training/update validation.

This change does not provide per-update transport checksum verification or enable the Miles recipe checker flags. Quantization tolerance intentionally permits some byte differences; transport integrity requires a separate checksum over the transmitted representation.

The 4over6 probe does not establish a universal error bound. In particular, blockscale underflow to zero also makes this checker's per-side tolerance zero, so the current bound need not cover all valid cross-quantizer differences. Such boundary cases have not been validated on GPU.

The ordinary PR Base/Extra workflows are blocked by the upstream maintenance ancestry check (Required base commit: 8854857a991e on main). This PR intentionally targets sglang-miles; the CPU workflow above tests the exact PR head independently.

Scope: NVFP4 MoE TRTLLM and CuteDSL standard. NVFP4 Linear and other unsupported backends remain unsupported.


CI States

Latest PR Test (Base): ❌ Run #37076108675
Latest PR Test (Extra): ❌ Run #37076108303
Latest PR Test (AMD ROCm 10): ❌ Run #37076108687

Kangyan-Zhou and others added 30 commits September 29, 2026 17:30
…nd pin SentencePiece for InternVL (sgl-project#41777) (sgl-project#41788)

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
…ty check to a warning (sgl-project#41681) (sgl-project#41802)

Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ject#22623)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lication (sgl-project#27268, sgl-project#31759, sgl-project#30913, sgl-project#30912, sgl-project#33299, sgl-project#36384, sgl-project#38165)

Register adapter identities, stream their tensors within weight-update sessions, and publish checksum-verified fresh versions alongside generation. Preserve per-rank failure cleanup and request-prefix cancellation for adapter lifecycle management.

Co-authored-by: Mathew Han <mathewjhan@gmail.com>
Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…ism (sgl-project#21278, sgl-project#22663, sgl-project#33478)

Expose rank topology for P2P weight mapping and create replicas using scoped runtime-context overrides. Restore the previous configuration on exit, including failed context entry.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…RL weight checks (sgl-project#29339)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…30499, sgl-project#37466, sgl-project#37887)

Co-authored-by: fzyzcjy <ch271828n@outlook.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…gl-project#27723)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…tils (sgl-project#35127)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Shi Dong <shi.dong@radixark.ai>
…weight checks (sgl-project#37649)

Co-authored-by: Zhichenzzz <zczeng@uw.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120

Projects

None yet

Development

Successfully merging this pull request may close these issues.