Skip to content

[Feat] beam: support LoRA for BF16 and FP8 - #48264

Merged
Fridge003 merged 2 commits into
sgl-project:mainfrom
jybsuper:yanbin/beam-lora-main
Nov 6, 2026
Merged

Fridge003 merged 2 commits into
sgl-project:mainfrom
jybsuper:yanbin/beam-lora-main

Conversation

@jybsuper

@jybsuper jybsuper commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Enable Beam LoRA serving for BF16 and compressed-tensors FP8 checkpoints with the csgmv/triton LoRA backends and the auto/triton MoE runner.

Modifications

  • Preserve self_attn.gate_proj as an explicit attention target; bare gate_proj continues to target MLP/MoE projections. Qualified attention-gate discovery also applies to other models exposing that module name.
  • Keep Beam QKV and attention-gate projections wrappable under LoRA, and disable the deferred expert-sum path that bypasses LoRA wrappers. Retain ordinary attention collective fusion.
  • Allocate hybrid-model dense LoRA buffers only on layers containing the target, respecting PP boundaries and preserving existing shared-expert allocation.
  • Share compressed-tensors FP8 quant-info construction between ordinary inference and LoRA. Extend existing unit tests for normalization, allocation, Beam construction, and CHANNEL/TOKEN FP8.

Accuracy Tests

Manual validation on the initial port, ef6c8bcafb, before the final allocation/test follow-up:

Check Result
Targeted CPU units on H200 with CUDA hidden 21 passed; the four added regression cases failed on the pre-port parent
Tiny BF16 Beam on H200 and GB300: TP1/csgmv/static adapters and TP2/triton/graph-enabled dynamic adapters Passed against an independent merged-weight reference; maximum KL 0.003299, below 0.005
Full reflection/Beam-1.0-501B-A23B-FP8 on GB300, TP4, Triton, dynamic rank-4 adapters Passed loading/generation smoke; zero adapter matched base greedy tokens and all probed logprobs; nonzero o_proj-only and all-target adapters changed first-token logprobs

The full-checkpoint smoke used synthetic adapters and establishes loading/serving behavior, not model quality or merged-weight FP8 equivalence. Graph capture completed; replay was not explicitly logged. The temporary tiny-model fixtures/E2E scripts are not included in this PR.

For the final follow-up, fd9a341598, local source-isolated CPU checks passed 8 allocation cases and 2 loading/clearing probes; the pre-fix allocator failed the intended sparse-layer regression. These checks use production method bodies with real CPU tensors and isolated import/module boundaries. Full unit collection was blocked by missing local _lzma/Triton dependencies; this final revision has not been rerun on GPUs.

Speed Tests and Profiling

No throughput benchmark. Dense-buffer allocation is reduced without changing inference computation.

Checklist

  • Changed-file pre-commit checks and git diff --check pass.
  • Extend existing unit tests without adding a new E2E fixture suite.
  • Run pre-commit run --all-files; the remaining failure is existing Linux-only Rust APIs unavailable on macOS.
  • Rerun full CPU/GPU validation on the final revision when the nodes are available.

CI States

Latest PR Test (Base): ✅ Run #41756297099
Latest PR Test (Extra): ❌ Run #41756297324
Latest PR Test (AMD ROCm 10): ❌ Run #41756297075

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added the lora label Nov 3, 2026
@jybsuper
jybsuper force-pushed the yanbin/beam-lora-main branch from 387b871 to fd9a341 Compare November 3, 2026 23:13
@jybsuper jybsuper changed the title [Feat] beam: support legacy LoRA for BF16 and FP8 [Feat] beam: support LoRA for BF16 and FP8 Nov 3, 2026
@jybsuper
jybsuper force-pushed the yanbin/beam-lora-main branch from fd9a341 to ecc3b1b Compare November 4, 2026 01:28
@jybsuper
jybsuper marked this pull request as ready for review November 4, 2026 01:41

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@jybsuper jybsuper added the run-ci CI: run the baseline test suite on this PR label Nov 4, 2026
Limit non-shared dense MLP buffers to layers with matching linears while preserving PP-local and shared-expert allocation. Fix subtest assertion scope and cover the Beam channel/token FP8 recipe.
@jybsuper
jybsuper force-pushed the yanbin/beam-lora-main branch from ecc3b1b to 50be687 Compare November 4, 2026 01:46
@Fridge003
Fridge003 merged commit 1ecf89b into sgl-project:main Nov 6, 2026
276 of 302 checks passed
frrad pushed a commit to modal-projects/sglang that referenced this pull request Nov 10, 2026
Copy the applicable upstream implementations and regression bodies from sgl-project/sglang commit 1ecf89b. Only recontextualize hunks around the existing Spindle fork code. Pool API adaptations, fork-specific behavior, and additional regressions follow in a separate commit.
frrad pushed a commit to modal-projects/sglang that referenced this pull request Nov 16, 2026
Copy the applicable upstream implementations and regression bodies from sgl-project/sglang commit 1ecf89b. Only recontextualize hunks around the existing Spindle fork code. Pool API adaptations, fork-specific behavior, and additional regressions follow in a separate commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lora run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants