Skip to content

docs: add the Beam 1.0 RL doc - #4138

Merged
zhihengy merged 19 commits into
mainfrom
zhihengy/beam-docs
Oct 17, 2026
Merged

zhihengy merged 19 commits into
mainfrom
zhihengy/beam-docs

Conversation

@zhihengy

@zhihengy zhihengy commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Adds the RL recipe page for Reflection Beam 1.0 (501B MoE, 23B active).

  • docs/models/reflection/beam.md: hybrid sliding-window / full-attention (no RoPE, SSMax) model, both launchers (full parameter on 96 H200 or 64 colocated, LoRA on 32 H200), the per-recipe parallelism, algorithm, optimizer (NVMe-streamed for full parameter) and rollout settings, and resume.
  • Registered as a single page after Muse Spark in docs.json and the models table.

The scripts and docker/Dockerfile.beam live on the beam-image branch, which the page links; SGLang support is the beam-miles branch (9ac1c65b).

zhihengy and others added 14 commits October 16, 2026 13:40
Add docs/models/reflection/beam.md for Reflection's Beam 1.0 in the
six-section recipe layout, register it after Muse Spark in docs.json and
add a Reflection row to the models table. The scripts and the image come
from the beam-image branch, which the page links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The softcap, not the LM head, runs in FP32; the FP32 tensors are the
  router weights and expert-score biases.
- 9ac1c65b also carries Beam LoRA serving; later commits are on `beam`.
- LoRA optimizer state is ~4.6 GB per GPU; the adapter sync is ~8.7 GiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe SSMax and the depth-scaled residuals in words, as the other model
pages do, give SSMax its own highlight, and state the BF16 / FP8 / NVFP4
release and the 1M-token context from the model card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow the Inkling page: one row per model with active / total parameters,
layers and HF ID. Note there, instead of in 3.1, that the bridge loads only
the BF16 checkpoint (the FP8 and NVFP4 releases carry a
quantization_config, which the Beam bridge rejects).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow the Step-5-Preview page: 3.2 points at the published multi-arch
radixark/miles:beam image (radixark/miles:v0.1.2 + SGLang beam-miles
9ac1c65b + the beam-image branch, built by docker/Dockerfile.beam) instead
of a local docker build, and 4 opens with the Ray cluster the launchers
submit to, with one subsection per topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Beam launchers set MILES_SCRIPT_EXTERNAL_RAY=1 and never start Ray,
so add a 3.3 like DeepSeek-V4.1's: run the image on every node with the
checkpoint, datasets and --output-dir on shared storage, start the head and
the workers, and check the GPU count before launching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow the Inkling page: sections 4-5 describe the full-parameter recipe
(the LoRA launch command stays in 4.4), and a new section 6 LoRA RL gathers
the LoRA layout, targets, expert-TP 1, optimizer and LR schedule, and
adapter-only weight sync. Pairs Well With becomes section 7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the separate LoRA RL section: the LoRA row returns to the 5.1 table,
and a short 5.5 LoRA lists the flags that differ from the full-parameter
recipe. 5.4 Rollout now shows the SGLang and weight-update flags as a
block (values match the launcher snapshots). Pairs Well With is section 6
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zhihengy
zhihengy marked this pull request as ready for review October 16, 2026 14:50

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@zhihengy zhihengy changed the title docs: add the Beam 1.0 RL recipe page docs: add the Beam 1.0 RL doc Oct 16, 2026
Comment thread docs/models/reflection/beam.md Outdated
`prepare` downloads the checkpoint into `--model-dir` and DAPO-Math-17K and AIME25 into `--data-dir`; every node must see both at the same path. `run_beam_lora.py prepare` is identical, so one run serves both recipes:

```bash
python scripts/run_beam.py prepare --model-dir /root/models/Beam --data-dir /root/datasets

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we have a code path that can one-click to convert ckpt and start training. Could you use that path?

Comment thread docs/models/reflection/beam.md Outdated
description: Launch recipes for Reflection's Beam 1.0 (501B MoE, 23B active) — full-parameter BF16 GRPO on 96 or 64 H200 GPUs, and LoRA on 32.
---
The complete Beam RL implementation is open on the Miles
[`beam-image`](https://github.com/radixark/miles/tree/beam-image) branch. The scripts and image

@guapisolo guapisolo Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add miles branch name for beam model and open a draft PR? It will be easier to find SoT


`ray status` on the head must list 96, 64 or 32 GPUs before launching, for 4.2, 4.3 and 4.4 respectively.

### 4.2 Full parameter, 12 nodes

@guapisolo guapisolo Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this async trianing? If so, we can mentioned it

Comment thread docs/models/reflection/beam.md Outdated
--sglang-dtype bfloat16
--sglang-attention-backend triton
--sglang-moe-runner-backend triton
--sglang-context-length 5120

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add this constrians?


```bash
--sglang-dtype bfloat16
--sglang-attention-backend triton

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we only have triton? Better check sgl cookbook for the beam model to find best perf solution

…h from rollout flags

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zhihengy added a commit that referenced this pull request Oct 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@zhihengy zhihengy left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todo: verify alternative attention backend;
Other comments are fixed; this pr is merged for time reason

Follows the launcher change: execute becomes train, execute-colocate becomes
train --colocate, and --join-ray-workers replaces bringing the Ray cluster up
by hand.
Section 3.1 is now optional: the launch commands download what they need.
full-train fetches the checkpoint and datasets itself, so 3.1 no longer walks
through prepare; it keeps the facts (shared path, no Megatron conversion, the
answer-format eval file) and mentions prepare only for staging the download
on its own.
@zhihengy
zhihengy merged commit e1a791d into main Oct 17, 2026
28 checks passed
@zhihengy
zhihengy deleted the zhihengy/beam-docs branch October 17, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants