Skip to content

[Feat] beam: support LoRA on the Spindle v0.5.23 fork - #95

Merged
frrad merged 4 commits into
spindle-sglang-v0.5.23from
fred/beam-spindle
Nov 17, 2026
Merged

frrad merged 4 commits into
spindle-sglang-v0.5.23from
fred/beam-spindle

Conversation

@frrad

@frrad frrad commented Nov 10, 2026 •

Copy link
Copy Markdown

Why: Beam LoRA sampling in Spindle (modal-labs/spindle#252, modal-projects/miles#37). The pinned Beam model rejects LoRA adapters.

Source: backport of upstream sgl-project/sglang#48264, merged into main as 1ecf89bf but not in a release yet. Drop it once Spindle moves to a release that contains 1ecf89bf.

Differences from sgl-project#48264:

  • alloc_per_layer is added to mem_pool.py without the lora_layer_ids filter, because v0.5.23 has no PP LoRA.
  • The slot-clear None guard matches upstream main. It is needed here because v0.5.23 buffers never had None entries before. The other buffer readers (adapter load, get_tensor) only touch layers that own the module, so they never see None.
  • The per-adapter normalized-target-modules refactor in lora.py isn't ported, because it only affects load time.
  • The fork-only test_beam_lora.py replaces upstream's PP and startup-contract tests, which need features v0.5.23 lacks.

Shared code this changes for other models:

  • Compressed-tensors FP8 MoE builds its Triton quant info through one get_triton_quant_info, which the LoRA MoE runner needs. block_shape is only set for BLOCK weights and per_channel_quant only for CHANNEL weights, so the quant info is unchanged for non-LoRA serving.
  • MoE models without shared experts now get dense gate_up_proj/down_proj LoRA buffers on their dense layers. Before, those layers had no buffers.
  • Adapter weights under self_attn.gate_proj are no longer merged into the MLP gate_up_proj. As upstream, all-linear discovery includes qualified attention gates, including AFMoE and this fork's gated xLLM/K2 Horizon variants.

Testing: on eaeec888, the CPU unit tests in unit/lora, unit/layers/moe and the compressed-tensors scheme tests plus pre-commit pass, apart from 21 failures that also fail on the base c4182b22. Not rerun on GPU: Beam LoRA serving and greedy decode on a compressed-tensors FP8 MoE model.

Review follow-up: restored upstream attention-gate discovery by undoing eaeec888, removed the extra slot-clear comment, and removed the redundant hybrid startup test while retaining adapter-removal/base-slot-reuse coverage. All pre-commit checks pass on the follow-up. The two affected unit-test files could not run in the available CPU environment because imports fail on missing psutil. GPU validation remains outstanding.

🤖 Generated with Claude Code

Comment on lines +423 to +428
quant_info = self.get_triton_quant_info(layer)
return self.runner.run(dispatch_output, quant_info)
else:
quant_info = TritonMoeQuantInfo(
w13_weight=layer.w13_weight,
w2_weight=layer.w2_weight,
use_fp8_w8a8=True,
per_channel_quant=self.weight_quant.strategy
== QuantizationStrategy.CHANNEL,
w13_scale=layer.w13_weight_scale,
w2_scale=layer.w2_weight_scale,
a13_scale=layer.w13_input_scale,
a2_scale=layer.w2_input_scale,
)
return self.runner.run(dispatch_output, quant_info)
return self.runner.run(dispatch_output, self.get_triton_quant_info(layer))

def get_triton_quant_info(self, layer: torch.nn.Module) -> TritonMoeQuantInfo:

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copied from upstream commit sgl-project@1ecf89b (PR sgl-project#48264). It extracts the two Triton quant-info constructors into get_triton_quant_info() so ordinary inference and FusedMoEWithLoRA can share the same base quantization metadata. The modes remain distinct: block quantization supplies block_shape and per_channel_quant=False; channel/tensor quantization supplies block_shape=None and sets per_channel_quant from the strategy. AITER and FlashInfer keep their separate paths.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm rather than having a backport is it possible to bring in sgl-project#48264 from upstream? can use the synmon to figure out regressions for all current models

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not released yet upstream.

More generally, how do we think about upgrading sgl / miles base images?

My mental model has been:
1/ each config pins an image
2/ people who need new features bring in new images, updating our miles and sgl forks as necessary. they try not to break other models but don't test exhaustively
3/ each config bumps their deps as needed after testing
4/ todo: create a validation run for each config so that 3 can be more automated

@frrad frrad changed the title [Feat] beam: port LoRA support to the Spindle v0.5.23 fork [Feat] beam: support LoRA on the Spindle v0.5.23 fork Nov 11, 2026
@frrad
frrad marked this pull request as ready for review November 11, 2026 23:53
@frrad frrad added the run-ci Trigger CI on this PR label Nov 11, 2026
jybsuper and others added 3 commits November 16, 2026 20:33
Copy the applicable upstream implementations and regression bodies from sgl-project/sglang commit 1ecf89b. Only recontextualize hunks around the existing Spindle fork code. Pool API adaptations, fork-specific behavior, and additional regressions follow in a separate commit.
Adapt hybrid dense/MoE allocation to the pinned pool API, retain expert ownership and model target aliases, and skip absent layer buffers when clearing base or removed-adapter slots. Add Beam construction/allocation/startup/removal/reuse regressions and retain working CPU test collection.
all-linear target discovery added self_attn.gate_proj for any model with an
attention gate, so AFMoE and xLLM would start wrapping theirs. Beam already
declares the target in supported_lora_modules, so discovery leaves it to that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frrad
frrad force-pushed the fred/beam-spindle branch from 991fb37 to eaeec88 Compare November 16, 2026 20:52
@frrad
frrad requested a review from micahtyong November 16, 2026 20:52
frrad added a commit to modal-labs/spindle that referenced this pull request Nov 16, 2026
modal-projects/sglang#95 was rebased onto the current spindle-sglang-v0.5.23
tip and now adds the Beam attention gate to auto-detected LoRA targets only
for models that declare it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread python/sglang/srt/lora/utils.py Outdated
parts = name.split(".")
leaf_name = parts[-1]
parent_qualified_name = ".".join(parts[-2:])
# Models opt into PRESERVED_LORA_TARGET_NAMES via supported_lora_modules.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we drop this commit (eaeec88)? Upstream main also auto-targets self_attn.gate_proj for every model here, and the only model it changes is AFMoE, which we don't serve. We want this fork to stay as close to upstream as possible so the next SGLang upgrade is a clean drop.

Comment thread python/sglang/srt/lora/mem_pool.py Outdated
for buffers in (*self.A_buffer.values(), *self.B_buffer.values()):
for tensor in buffers:
tensor[buffer_id].zero_()
# Hybrid layers omit dense projections they do not own.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: upstream main already has this exact guard in _clear_buffer_slot_for_base, without this comment line. Dropping the comment makes this match upstream exactly.

lora_lm_head_module=None,
)

def test_hybrid_startup_prepares_base_slot(self):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is already covered by the base-slot-reuse part of test_hybrid_adapter_removal_and_base_slot_reuse. Can we cut it?

@frrad
frrad merged commit 568ab74 into spindle-sglang-v0.5.23 Nov 17, 2026
frrad added a commit to modal-labs/spindle that referenced this pull request Nov 17, 2026
Miles moves to d1915e68, the tip of modal-projects/miles#37 after review.
It drops the shared Bridge LoRA changes from the upstream port and keeps
Beam's FP32 router settings inside its own provider.

SGLang moves to 568ab742 on spindle-sglang-v0.5.23, the merge of
modal-projects/sglang#95, instead of the personal branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lora run-ci Trigger CI on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants