Skip to content

[sglang-miles] [Feat] beam: Port Beam BF16 serving and LoRA - #45273

Draft
zhihengy wants to merge 53 commits into
sgl-project:sglang-milesfrom
zhihengy:sglang-miles-beam
Draft

zhihengy wants to merge 53 commits into
sgl-project:sglang-milesfrom
zhihengy:sglang-miles-beam

Conversation

@zhihengy

@zhihengy zhihengy commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Ports Beam BF16 serving and BF16 LoRA serving onto sglang-miles for Miles RL. The source is #44747's head (f7473673dd, the beam branch). The scope follows the upstream split, restricted to BF16: kernels from #45254, model from #45256, plus Beam LoRA serving, which the split leaves out.

Three commits:

  • Kernels (from [Feat] kernels: Beam model support #45254): the attention-gate and sandwich-norm fused ops, the FP32 router GEMMs (tiny GEMM, GEMV, BF16x3 cuBLAS and cuTile), the Beam all-reduce boundary, and CustomAllReduceV2.supports_push. pyproject.toml adds nvidia-cuda-tileiras==13.4.92, which the default cuTile router needs.
  • Model (from [Feat] beam: Add Beam model, runtime support and checkpoints (4/7) #45256): BeamConfig, BeamForCausalLM, the hybrid-SWA registration, the Beam override, the Beam environment variables, the defer_sum path of the Triton MoE runner, the fixture and the BF16 tests.
  • LoRA (Fix the error message and dependency of openai backend #71 on beam): models/beam.py as on beam, the qualified self_attn.gate_proj target and the dense-MLP LoRA buffers for leading dense layers. The fixture adapters and the e2e test check each adapter against the merged-weight oracle.

Not ported:

Streamed LoRA is already on sglang-miles as [12/42].

Adaptations to sglang-miles:

  • Dense adapter weights load per member into fused slots ([42/42]). _fused_member_slot therefore skips the attention gate, which would otherwise land in the MLP gate_up_proj slot 0. On beam, normalize_gate_up_proj skipped it before stacking; that hunk is not needed here, because stacking only touches expert weights.
  • Target normalization keeps the preserved qualified name before the model-aware hook ([27/42]).
  • LoRAAdapter normalizes targets once in __init__, so Fix the error message and dependency of openai backend #71's normalize-once change to lora.py is not needed.
  • Two Beam LoRA target tests now assert the per-member slots and the init-time normalization. One Beam unit test sets server args, which the true-on-policy RMSNorm ([1/42]) reads.

Validation (4×B200; torch 2.14.1, sglang-kernel 0.4.9, flashinfer 0.7.1; the Beam fixture checkpoint):

  • e2e test_beam.py passes 2/2: CPU offload, plus prefill and cached decode across TP1, TP2, TP2+EP2, PP2, PP2 with graphs, TP4 and the DP-attention layouts.
  • e2e test_beam_kl.py 6/6, test_beam_cache.py 1/1 and test_beam_mixed_chunk.py 1/1 pass.
  • e2e test_beam_numerics.py passes 6/7. test_single_token_compiled_moe_preserves_runner_scale fails the same way on beam (f7473673dd) in the same environment (ReplicatedLinear has no bf16x3_weight).
  • e2e test_beam_lora.py passes 1/1, with 50 subtests: TP1, TP2, TP2+EP2 and PP2; csgmv and triton LoRA; graphs; dynamic loading; per-expert and shared-outer experts.
    • With --lora-use-virtual-experts (as Miles launches), the non-PP layouts also pass: 40 subtests.
    • With the triton attention backend, prompts that cross the 512-token window exceed the per-token envelope on the first decode token (0.44 to 0.48 against 0.4). This is the pre-existing triton SWA decode window: decode attends sliding_window - 1 keys. The same happens with or without virtual experts, and it is not changed here.
  • Unit tests: unit/models/test_beam*.py and unit/lora/test_beam_lora_targets.py give 61 passed. The two failing subtests also fail on beam. The unit/lora and test_model_overrides.py suites fail exactly the same tests as on sglang-miles (bff3f6ae7e).
  • pre-commit passes on all changed files.

CI States

Latest PR Test (Base): ❌ Run #39351473795
Latest PR Test (Extra): ❌ Run #39351473585
Latest PR Test (AMD ROCm 10): ❌ Run #39351473797

Kangyan-Zhou and others added 30 commits October 13, 2026 23:15
…om the PD prefill server to release/v0.5.22 (sgl-project#44002) (sgl-project#44212)

Co-authored-by: William Arnold <7565007+Aphoh@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
…arse+MTP B300 test to release/v0.5.22 (sgl-project#44192) (sgl-project#44214)

Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
…lication (sgl-project#27268, sgl-project#31759, sgl-project#30913, sgl-project#30912, sgl-project#33299, sgl-project#36384, sgl-project#38165)

Register adapter identities, stream their tensors within weight-update sessions, and publish checksum-verified fresh versions alongside generation. Preserve per-rank failure cleanup and request-prefix cancellation for adapter lifecycle management.

Co-authored-by: Mathew Han <mathewjhan@gmail.com>
Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…ism (sgl-project#21278, sgl-project#22663, sgl-project#33478)

Expose rank topology for P2P weight mapping and create replicas using scoped runtime-context overrides. Restore the previous configuration on exit, including failed context entry.

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…RL weight checks (sgl-project#29339)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…30499, sgl-project#37466, sgl-project#37887)

Co-authored-by: fzyzcjy <ch271828n@outlook.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…weight checks (sgl-project#37649)

Co-authored-by: Zhichenzzz <zczeng@uw.edu>
…e MXFP4 reload (sgl-project#37704, sgl-project#41074, sgl-project#41925)

Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
…load; restore bf16 MoE layout (sgl-project#40821)

Co-authored-by: Yueming Yuan <yym022502@gmail.com>
Rockdu and others added 23 commits October 15, 2026 16:16
…PC LoRA updates (sgl-project#40786)

Keep H3 rollout and streamed-weight behavior on the release implementation. The release already contains the LoRA wrapper type guards.

Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
Co-authored-by: niehen6174 <niehen6174@gmail.com>
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
…y from metadata (sgl-project#39265)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…] fix: report the port each scheduler actually bound to its clients (sgl-project#42470)

Signed-off-by: rockdu <kangrdu@gmail.com>
… updates (sgl-project#39392)

Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
…V budget, UE8M0 partial blocks, mxfp8 fused indexer, NVFP4 input scales (sgl-project#43251)
`DeepseekV4DecoderLayer.bind_next` (run from `post_load_weights`) assigned the
next decoder layer to `self._next_layer`, which `nn.Module.__setattr__`
registers as a submodule. After the first load, `named_parameters()` dedups
layers 1.. under `model.layers.0._next_layer[._next_layer...]`, so every later
`load_weights` call (update_weights_from_tensor / _disk / _distributed) misses
`model.layers.{i}.*` and crashes with a KeyError. Only models with
`hc_pre_from_prev_sublayer` (V4.1) bind a next layer.

Hold the next layer as a plain attribute instead.
Since sgl-project#42589 the GPT-OSS and Inkling weight loaders read moe_tp_rank,
moe_tp_size, moe_ep_rank and moe_ep_size off `self.experts`. With LoRA
attached that module is FusedMoEWithLoRA, which only kept them as tp_rank /
tp_size, so the first GPT-OSS LoRA weight load raised
AttributeError: 'FusedMoEWithLoRA' object has no attribute 'moe_tp_rank'.
The wrapper now forwards all four, like the other FusedMoE attributes the
model reads on it.

The unit test's base-layer stand-in also gains named_parameters, which
BaseLayerWithLoRA.__init__ has read since the MoE LoRA support commit; its
two existing cases failed without it.
… kernel

The legacy CUDA kernel sizes its grid from the element count, so an empty
input launches zero blocks. The launch is unchecked and leaves
cudaErrorInvalidValue pending until the next kernel launch on the thread.

The attention TP group now selects a custom communicator (sgl-project#35330), so with
SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=0 an idle DP-attention rank all-reduces
the zero-row embedding output through this path, and its next kernel fails
with "CUDA error: invalid argument". An empty all-reduce exchanges nothing,
so return the empty output directly.
…n instead of mixins (sglang-miles port) (sgl-project#44877)

Signed-off-by: rockdu <kangrdu@gmail.com>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
Co-authored-by: JingwenGu0829 <75733630+JingwenGu0829@users.noreply.github.com>
@github-actions github-actions Bot added dependencies Pull requests that update a dependency file lora jit-kernel labels Oct 20, 2026
@Rockdu
Rockdu force-pushed the sglang-miles branch 2 times, most recently from 865cb64 to 29b34bc Compare October 31, 2026 06:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file jit-kernel lora

Projects

None yet

Development

Successfully merging this pull request may close these issues.