Repository navigation
Conversation
…om the PD prefill server to release/v0.5.22 (sgl-project#44002) (sgl-project#44212) Co-authored-by: William Arnold <7565007+Aphoh@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
…arse+MTP B300 test to release/v0.5.22 (sgl-project#44192) (sgl-project#44214) Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com> Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
…allel math (sgl-project#18639, sgl-project#30742) Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
…gl-project#25141, sgl-project#29874, sgl-project#31251, sgl-project#32376) Co-authored-by: Yusheng Su <yushengsu.thu@gmail.com>
…lication (sgl-project#27268, sgl-project#31759, sgl-project#30913, sgl-project#30912, sgl-project#33299, sgl-project#36384, sgl-project#38165) Register adapter identities, stream their tensors within weight-update sessions, and publish checksum-verified fresh versions alongside generation. Preserve per-rank failure cleanup and request-prefix cancellation for adapter lifecycle management. Co-authored-by: Mathew Han <mathewjhan@gmail.com> Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com> Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…ism (sgl-project#21278, sgl-project#22663, sgl-project#33478) Expose rank topology for P2P weight mapping and create replicas using scoped runtime-context overrides. Restore the previous configuration on exit, including failed context entry. Co-authored-by: yueming-yuan <yym022502@gmail.com>
…capture + meta_info threading
…RL weight checks (sgl-project#29339) Co-authored-by: yueming-yuan <yym022502@gmail.com>
…30499, sgl-project#37466, sgl-project#37887) Co-authored-by: fzyzcjy <ch271828n@outlook.com> Co-authored-by: yueming-yuan <yym022502@gmail.com>
…ed weights into a host-local checkpoint (sgl-project#30366, sgl-project#28524)
…gl-project#27110, sgl-project#38718) Co-authored-by: Shi Dong <hello@dongshi.me>
…gl-project#27723) Co-authored-by: yueming-yuan <yym022502@gmail.com>
…educe-fusion auto-enable
…CL all-gather path (sgl-project#36110)
…tils (sgl-project#35127) Co-authored-by: Shi Dong <shi.dong@radixark.ai>
…weight checks (sgl-project#37649) Co-authored-by: Zhichenzzz <zczeng@uw.edu>
…ace in the weight checker (sgl-project#40028)
…e MXFP4 reload (sgl-project#37704, sgl-project#41074, sgl-project#41925) Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu> Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
…load; restore bf16 MoE layout (sgl-project#40821) Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…PC LoRA updates (sgl-project#40786) Keep H3 rollout and streamed-weight behavior on the release implementation. The release already contains the LoRA wrapper type guards. Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com> Co-authored-by: niehen6174 <niehen6174@gmail.com>
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972) Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
…y from metadata (sgl-project#39265) Co-authored-by: yueming-yuan <yym022502@gmail.com>
…] fix: report the port each scheduler actually bound to its clients (sgl-project#42470) Signed-off-by: rockdu <kangrdu@gmail.com>
…st (sgl-project#42436) Signed-off-by: rockdu <kangrdu@gmail.com>
…k for DeepSeek-V4.1 (sgl-project#42408)
… updates (sgl-project#39392) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
…V budget, UE8M0 partial blocks, mxfp8 fused indexer, NVFP4 input scales (sgl-project#43251)
…ir own buffer slots (sgl-project#44332)
`DeepseekV4DecoderLayer.bind_next` (run from `post_load_weights`) assigned the
next decoder layer to `self._next_layer`, which `nn.Module.__setattr__`
registers as a submodule. After the first load, `named_parameters()` dedups
layers 1.. under `model.layers.0._next_layer[._next_layer...]`, so every later
`load_weights` call (update_weights_from_tensor / _disk / _distributed) misses
`model.layers.{i}.*` and crashes with a KeyError. Only models with
`hc_pre_from_prev_sublayer` (V4.1) bind a next layer.
Hold the next layer as a plain attribute instead.
Since sgl-project#42589 the GPT-OSS and Inkling weight loaders read moe_tp_rank, moe_tp_size, moe_ep_rank and moe_ep_size off `self.experts`. With LoRA attached that module is FusedMoEWithLoRA, which only kept them as tp_rank / tp_size, so the first GPT-OSS LoRA weight load raised AttributeError: 'FusedMoEWithLoRA' object has no attribute 'moe_tp_rank'. The wrapper now forwards all four, like the other FusedMoE attributes the model reads on it. The unit test's base-layer stand-in also gains named_parameters, which BaseLayerWithLoRA.__init__ has read since the MoE LoRA support commit; its two existing cases failed without it.
… kernel The legacy CUDA kernel sizes its grid from the element count, so an empty input launches zero blocks. The launch is unchecked and leaves cudaErrorInvalidValue pending until the next kernel launch on the thread. The attention TP group now selects a custom communicator (sgl-project#35330), so with SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=0 an idle DP-attention rank all-reduces the zero-row embedding output through this path, and its next kernel fails with "CUDA error: invalid argument". An empty all-reduce exchanges nothing, so return the empty output directly.
…n instead of mixins (sglang-miles port) (sgl-project#44877) Signed-off-by: rockdu <kangrdu@gmail.com> Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com> Co-authored-by: JingwenGu0829 <75733630+JingwenGu0829@users.noreply.github.com>
Rockdu
force-pushed
the
sglang-miles
branch
2 times, most recently
from
October 31, 2026 06:43
865cb64 to
29b34bc
Compare
guapisolo
force-pushed
the
sglang-miles
branch
from
November 1, 2026 02:53
29b34bc to
b0ed761
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports Beam BF16 serving and BF16 LoRA serving onto
sglang-milesfor Miles RL. The source is #44747's head (f7473673dd, thebeambranch). The scope follows the upstream split, restricted to BF16: kernels from #45254, model from #45256, plus Beam LoRA serving, which the split leaves out.Three commits:
CustomAllReduceV2.supports_push.pyproject.tomladdsnvidia-cuda-tileiras==13.4.92, which the default cuTile router needs.BeamConfig,BeamForCausalLM, the hybrid-SWA registration, the Beam override, the Beam environment variables, thedefer_sumpath of the Triton MoE runner, the fixture and the BF16 tests.beam):models/beam.pyas onbeam, the qualifiedself_attn.gate_projtarget and the dense-MLP LoRA buffers for leading dense layers. The fixture adapters and the e2e test check each adapter against the merged-weight oracle.Not ported:
Streamed LoRA is already on
sglang-milesas [12/42].Adaptations to
sglang-miles:_fused_member_slottherefore skips the attention gate, which would otherwise land in the MLPgate_up_projslot 0. Onbeam,normalize_gate_up_projskipped it before stacking; that hunk is not needed here, because stacking only touches expert weights.LoRAAdapternormalizes targets once in__init__, so Fix the error message and dependency of openai backend #71's normalize-once change tolora.pyis not needed.Validation (4×B200; torch 2.14.1, sglang-kernel 0.4.9, flashinfer 0.7.1; the Beam fixture checkpoint):
test_beam.pypasses 2/2: CPU offload, plus prefill and cached decode across TP1, TP2, TP2+EP2, PP2, PP2 with graphs, TP4 and the DP-attention layouts.test_beam_kl.py6/6,test_beam_cache.py1/1 andtest_beam_mixed_chunk.py1/1 pass.test_beam_numerics.pypasses 6/7.test_single_token_compiled_moe_preserves_runner_scalefails the same way onbeam(f7473673dd) in the same environment (ReplicatedLinearhas nobf16x3_weight).test_beam_lora.pypasses 1/1, with 50 subtests: TP1, TP2, TP2+EP2 and PP2; csgmv and triton LoRA; graphs; dynamic loading; per-expert and shared-outer experts.--lora-use-virtual-experts(as Miles launches), the non-PP layouts also pass: 40 subtests.sliding_window - 1keys. The same happens with or without virtual experts, and it is not changed here.unit/models/test_beam*.pyandunit/lora/test_beam_lora_targets.pygive 61 passed. The two failing subtests also fail onbeam. Theunit/loraandtest_model_overrides.pysuites fail exactly the same tests as onsglang-miles(bff3f6ae7e).CI States
Latest PR Test (Base): ❌ Run #39351473795
Latest PR Test (Extra): ❌ Run #39351473585
Latest PR Test (AMD ROCm 10): ❌ Run #39351473797