Skip to content

feat(beam): add Reflection Beam 1.0 RL support - #4149

Draft
zhihengy wants to merge 4 commits into
mainfrom
beam-support
Draft

zhihengy wants to merge 4 commits into
mainfrom
beam-support

Conversation

@zhihengy

Copy link
Copy Markdown
Collaborator

ci-sglang-pr: beam-miles

Day-0 RL support for Reflection Beam 1.0 (501B MoE, 23B active): GRPO with SGLang rollout and Megatron training in BF16 — full parameter on 96 H200 (64 colocated), and LoRA on 32 H200. Image: docker/Dockerfile.beam, i.e. radixark/miles:v0.1.2 plus SGLang beam-miles at 9ac1c65b. Recipe page: #4138.

  • Model (miles_plugins/models/beam.py): Beam layer spec on Megatron — sliding-window layers with RoPE and full-attention layers without it (SSMax query scaling), a sigmoid output gate with per-head bias, sandwich norms with depth-scaled residuals, and an FP32 router weight.
  • Weights (miles_plugins/megatron_bridge/beam.py): a Megatron-Bridge BeamForCausalLM bridge that loads the BF16 HF checkpoint directly (no torch_dist conversion), with the FP32 logit softcap and the fused [q, g, k, v] mapping, including LoRA adapter export.
  • Recipes: scripts/run_beam.py (execute on 96 H200, async; execute-colocate on 64 H200), scripts/run_beam_lora.py (32 H200), reward examples/beam/reward.py, model dimensions scripts/models/beam.py.

Shared changes that touch other models

  • LoRA through Megatron-Bridge starts from the full-model Bridge runtime config, asserts expert-TP 1 when experts carry LoRA (fix(peft)!: make expert LoRA exact under suppressed tensor-parallel communication NVIDIA-NeMo/Megatron-Bridge#5635), freezes expert_bias under any LoRA (previously multi-LoRA only), and passes data_parallel_random_init.
  • LoRA checkpoints save and restore the DistributedOptimizer parameter state (FP32 masters and moments); a checkpoint without it keeps a fresh optimizer.
  • --lora-target-modules reaches SGLang as module names; the attention self_attn.gate_proj stays qualified and is not counted as an expert projection.
  • register_hf_config_aliases registers SGLang's BeamConfig for model_type: beam.

Branch state: branched from beam-image (the Day-0 image source, on v0.1.2 9926b1ca). Before this leaves draft it needs main merged in: miles/backends/megatron_utils/lora/utils.py and its test conflict, and release-lock.json comes from the v0.1.2 release line.

🤖 Generated with Claude Code

zhihengy and others added 4 commits October 16, 2026 13:24
Image source for the Day-0 Beam release. SGLang is pinned to 9ac1c65b, the tip
of the public beam-miles branch on sgl-project/sglang.
Restructure docs/models/reflection/beam.md into the layout of
docs/models/index.md (as the Step-5-Preview page in #4104 does): model
introduction from the released HF config, both launchers (full parameter
and LoRA), the Beam SGLang pin, launch and resume for all three topologies,
and per-recipe parallelism, algorithm, optimizer and rollout settings.
Register the page after MiMo in docs.json and add a Reflection row to the
models table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both Beam launchers used prepare/execute/execute-colocate and hardcoded an
external Ray cluster. Match the convention #2431 gave every other launcher:

- execute -> train; add full-train (prepare + train in one command)
- execute-colocate -> a --colocate flag, as run_deepseek_v4.py does
- external Ray is now a mode, not an assumption: --join-ray-workers makes the
  launcher build the cluster from --ray-hostfile, and only without it do we set
  MILES_SCRIPT_EXTERNAL_RAY

The emitted command is unchanged: the train snapshot is byte-identical to the
old execute one. run_beam_lora.py had no committed snapshot at all; add it.
@micahtyong

Copy link
Copy Markdown
Contributor

Hi @zhihengy, thanks for putting this up. I'm wondering if you have a rough timeline on merging this into Miles upstream?

@frrad

frrad commented Nov 12, 2026

Copy link
Copy Markdown

Thanks for this PR. I'm following it to add support for Beam in modal-labs/spindle#252

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants