Repository navigation
[sglang-miles] Add NVFP4 MoE weight checker support - #42283
Open
yueming-yuan wants to merge 37 commits into
Open
yueming-yuan wants to merge 37 commits into
yueming-yuan wants to merge 37 commits into
Conversation
…nd pin SentencePiece for InternVL (sgl-project#41777) (sgl-project#41788) Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
…ty check to a warning (sgl-project#41681) (sgl-project#41802) Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com>
…allel math (sgl-project#18639, sgl-project#30742) Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
…l-project#18642, sgl-project#36404) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ject#22623) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…gl-project#25141, sgl-project#29874, sgl-project#31251, sgl-project#32376) Co-authored-by: Yusheng Su <yushengsu.thu@gmail.com>
…lication (sgl-project#27268, sgl-project#31759, sgl-project#30913, sgl-project#30912, sgl-project#33299, sgl-project#36384, sgl-project#38165) Register adapter identities, stream their tensors within weight-update sessions, and publish checksum-verified fresh versions alongside generation. Preserve per-rank failure cleanup and request-prefix cancellation for adapter lifecycle management. Co-authored-by: Mathew Han <mathewjhan@gmail.com> Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com> Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…ism (sgl-project#21278, sgl-project#22663, sgl-project#33478) Expose rank topology for P2P weight mapping and create replicas using scoped runtime-context overrides. Restore the previous configuration on exit, including failed context entry. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: yueming-yuan <yym022502@gmail.com>
…capture + meta_info threading
…RL weight checks (sgl-project#29339) Co-authored-by: yueming-yuan <yym022502@gmail.com>
…30499, sgl-project#37466, sgl-project#37887) Co-authored-by: fzyzcjy <ch271828n@outlook.com> Co-authored-by: yueming-yuan <yym022502@gmail.com>
…ed weights into a host-local checkpoint (sgl-project#30366, sgl-project#28524)
…gl-project#27110, sgl-project#38718) Co-authored-by: Shi Dong <hello@dongshi.me>
…gl-project#27723) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: yueming-yuan <yym022502@gmail.com>
…educe-fusion auto-enable
…CL all-gather path (sgl-project#36110)
…tils (sgl-project#35127) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Shi Dong <shi.dong@radixark.ai>
…weight checks (sgl-project#37649) Co-authored-by: Zhichenzzz <zczeng@uw.edu>
…ace in the weight checker (sgl-project#40028)
guapisolo
requested review from
1am9trash,
AgainstEntropy,
Alisehen,
AniZpZ,
ByronHsu,
DarkSharpness,
Duyi-Wang,
Edwardf0t1,
FlamingoPg,
HydraQYH,
OrangeRedeng,
ShangmingCai,
YAMY1234,
celve,
hebiao064,
hubertlu-tw,
kevin-mii,
kkHuang-amd,
niehen6174,
ping1jing2,
rainj-me,
sogalin and
yichiche
as code owners
October 17, 2026 00:29
guapisolo
force-pushed
the
sglang-miles
branch
from
November 1, 2026 02:54
29b34bc to
b0ed761
Compare
This was referenced Nov 12, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NVFP4 MoE weights currently either compare raw storage (TRTLLM) or reject weight checking (CuteDSL). This makes equivalent scale encodings fail comparison and prevents checking the CuteDSL standard path after a weight update.
Decode TRTLLM and CuteDSL standard NVFP4 expert layouts into comparable values using the packed FP4 weights, block scales, and global scales. Check execution alphas and separately stored swizzled/MMA scales alongside their weights so stale derived parameters are detected. Existing quantization-error tolerance remains opt-in.
Validation:
pre-commit run --all-files: passed.test/registered/e2e/rl/test_weight_checker_nvfp4.py: 5 tests passed on one B200 with PyTorch 2.13.0+cu130 and FlashInfer 0.6.18. Covers real backend layouts, equivalent scale encodings, corruption, CPU snapshots/chunking, and separate padded scales.test_weight_checker.pyandtest_weight_checker_comparator.py: 75 tests passed on the same B200.e159d0251060f9d476d2c4b0e17984cea8b95167; includes NVFP4 configuration, MoE dispatch/scales, and runtime context.This change does not provide per-update transport checksum verification or enable the Miles recipe checker flags. Quantization tolerance intentionally permits some byte differences; transport integrity requires a separate checksum over the transmitted representation.
The 4over6 probe does not establish a universal error bound. In particular, blockscale underflow to zero also makes this checker's per-side tolerance zero, so the current bound need not cover all valid cross-quantizer differences. Such boundary cases have not been validated on GPU.
The ordinary PR Base/Extra workflows are blocked by the upstream maintenance ancestry check (
Required base commit: 8854857a991eonmain). This PR intentionally targetssglang-miles; the CPU workflow above tests the exact PR head independently.Scope: NVFP4 MoE TRTLLM and CuteDSL standard. NVFP4 Linear and other unsupported backends remain unsupported.
CI States
Latest PR Test (Base): ❌ Run #37076108675
Latest PR Test (Extra): ❌ Run #37076108303
Latest PR Test (AMD ROCm 10): ❌ Run #37076108687