Repository navigation
[Feat] beam: support LoRA on the Spindle v0.5.23 fork - #95
Conversation
db757b5 to
e22646b
Compare
| quant_info = self.get_triton_quant_info(layer) | ||
| return self.runner.run(dispatch_output, quant_info) | ||
| else: | ||
| quant_info = TritonMoeQuantInfo( | ||
| w13_weight=layer.w13_weight, | ||
| w2_weight=layer.w2_weight, | ||
| use_fp8_w8a8=True, | ||
| per_channel_quant=self.weight_quant.strategy | ||
| == QuantizationStrategy.CHANNEL, | ||
| w13_scale=layer.w13_weight_scale, | ||
| w2_scale=layer.w2_weight_scale, | ||
| a13_scale=layer.w13_input_scale, | ||
| a2_scale=layer.w2_input_scale, | ||
| ) | ||
| return self.runner.run(dispatch_output, quant_info) | ||
| return self.runner.run(dispatch_output, self.get_triton_quant_info(layer)) | ||
|
|
||
| def get_triton_quant_info(self, layer: torch.nn.Module) -> TritonMoeQuantInfo: |
There was a problem hiding this comment.
Copied from upstream commit sgl-project@1ecf89b (PR sgl-project#48264). It extracts the two Triton quant-info constructors into get_triton_quant_info() so ordinary inference and FusedMoEWithLoRA can share the same base quantization metadata. The modes remain distinct: block quantization supplies block_shape and per_channel_quant=False; channel/tensor quantization supplies block_shape=None and sets per_channel_quant from the strategy. AITER and FlashInfer keep their separate paths.
There was a problem hiding this comment.
hmm rather than having a backport is it possible to bring in sgl-project#48264 from upstream? can use the synmon to figure out regressions for all current models
There was a problem hiding this comment.
It's not released yet upstream.
More generally, how do we think about upgrading sgl / miles base images?
My mental model has been:
1/ each config pins an image
2/ people who need new features bring in new images, updating our miles and sgl forks as necessary. they try not to break other models but don't test exhaustively
3/ each config bumps their deps as needed after testing
4/ todo: create a validation run for each config so that 3 can be more automated
Copy the applicable upstream implementations and regression bodies from sgl-project/sglang commit 1ecf89b. Only recontextualize hunks around the existing Spindle fork code. Pool API adaptations, fork-specific behavior, and additional regressions follow in a separate commit.
Adapt hybrid dense/MoE allocation to the pinned pool API, retain expert ownership and model target aliases, and skip absent layer buffers when clearing base or removed-adapter slots. Add Beam construction/allocation/startup/removal/reuse regressions and retain working CPU test collection.
all-linear target discovery added self_attn.gate_proj for any model with an attention gate, so AFMoE and xLLM would start wrapping theirs. Beam already declares the target in supported_lora_modules, so discovery leaves it to that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
991fb37 to
eaeec88
Compare
modal-projects/sglang#95 was rebased onto the current spindle-sglang-v0.5.23 tip and now adds the Beam attention gate to auto-detected LoRA targets only for models that declare it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| parts = name.split(".") | ||
| leaf_name = parts[-1] | ||
| parent_qualified_name = ".".join(parts[-2:]) | ||
| # Models opt into PRESERVED_LORA_TARGET_NAMES via supported_lora_modules. |
There was a problem hiding this comment.
Can we drop this commit (eaeec88)? Upstream main also auto-targets self_attn.gate_proj for every model here, and the only model it changes is AFMoE, which we don't serve. We want this fork to stay as close to upstream as possible so the next SGLang upgrade is a clean drop.
| for buffers in (*self.A_buffer.values(), *self.B_buffer.values()): | ||
| for tensor in buffers: | ||
| tensor[buffer_id].zero_() | ||
| # Hybrid layers omit dense projections they do not own. |
There was a problem hiding this comment.
Non-blocking: upstream main already has this exact guard in _clear_buffer_slot_for_base, without this comment line. Dropping the comment makes this match upstream exactly.
| lora_lm_head_module=None, | ||
| ) | ||
|
|
||
| def test_hybrid_startup_prepares_base_slot(self): |
There was a problem hiding this comment.
This test is already covered by the base-slot-reuse part of test_hybrid_adapter_removal_and_base_slot_reuse. Can we cut it?
Miles moves to d1915e68, the tip of modal-projects/miles#37 after review. It drops the shared Bridge LoRA changes from the upstream port and keeps Beam's FP32 router settings inside its own provider. SGLang moves to 568ab742 on spindle-sglang-v0.5.23, the merge of modal-projects/sglang#95, instead of the personal branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why: Beam LoRA sampling in Spindle (modal-labs/spindle#252, modal-projects/miles#37). The pinned Beam model rejects LoRA adapters.
Source: backport of upstream sgl-project/sglang#48264, merged into main as
1ecf89bfbut not in a release yet. Drop it once Spindle moves to a release that contains1ecf89bf.Differences from sgl-project#48264:
alloc_per_layeris added tomem_pool.pywithout thelora_layer_idsfilter, because v0.5.23 has no PP LoRA.Noneguard matches upstream main. It is needed here because v0.5.23 buffers never hadNoneentries before. The other buffer readers (adapter load,get_tensor) only touch layers that own the module, so they never seeNone.lora.pyisn't ported, because it only affects load time.test_beam_lora.pyreplaces upstream's PP and startup-contract tests, which need features v0.5.23 lacks.Shared code this changes for other models:
get_triton_quant_info, which the LoRA MoE runner needs.block_shapeis only set for BLOCK weights andper_channel_quantonly for CHANNEL weights, so the quant info is unchanged for non-LoRA serving.gate_up_proj/down_projLoRA buffers on their dense layers. Before, those layers had no buffers.self_attn.gate_projare no longer merged into the MLPgate_up_proj. As upstream, all-linear discovery includes qualified attention gates, including AFMoE and this fork's gated xLLM/K2 Horizon variants.Testing: on
eaeec888, the CPU unit tests inunit/lora,unit/layers/moeand the compressed-tensors scheme tests plus pre-commit pass, apart from 21 failures that also fail on the basec4182b22. Not rerun on GPU: Beam LoRA serving and greedy decode on a compressed-tensors FP8 MoE model.Review follow-up: restored upstream attention-gate discovery by undoing
eaeec888, removed the extra slot-clear comment, and removed the redundant hybrid startup test while retaining adapter-removal/base-slot-reuse coverage. All pre-commit checks pass on the follow-up. The two affected unit-test files could not run in the available CPU environment because imports fail on missingpsutil. GPU validation remains outstanding.🤖 Generated with Claude Code