Skip to content

Add Beam BF16 LoRA support and a reference run - #252

Open
Fred Robinson (frrad) wants to merge 3 commits into
reference-dapo-multinodefrom
bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn
Open

Fred Robinson (frrad) wants to merge 3 commits into
reference-dapo-multinodefrom
bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn

Conversation

@frrad

@frrad Fred Robinson (frrad) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds BF16 LoRA support for reflection/Beam-1.0-501B-A23B (#235). It builds on two fork changes:

What changes in Spindle:

  • Two new named images, used only by the Beam recipe: miles-v0.2.0-3280b7a for training and sglang-v0.5.23-568ab74 for sampling. Following the frozen-image rule in Pin frozen, commit-named images in every bundled config #313, no existing image changes, so other configs are unaffected.
  • Multi-node miles-v0.2.0-* trainers. Each node commits its adapter checkpoint shards to the checkpoint volume, and the head reloads the volume before checking the capture. This only happens for captures on the volume, so single-node trainers behave as before.
  • Typed-config support so Spindle's replay, score-centering and target hooks work with v0.2.0's config objects. Also adds a debug-only dangerously_allow_checkpoint_revision_mismatch.
  • A two-node recipe and a reference client. The recipe is reference_runs/beam_config.py, with a 2×8 H200 trainer and an 8×B300 sampler. The client is reference_runs/beam.py, on the stock Tinker SDK.

Reference run

The reference client trained against this branch and was compared with the published Beam LoRA run and an earlier run on Miles v0.1.2. The goal is confidence that training on this stack is healthy, not a parity claim. The blog curve is a manual digitization, and its step axis isn't confirmed to count optimizer updates. Accuracy above the blog most likely comes from config differences, such as the loss, async lag and hardware, not a real improvement.

AIME25 accuracy, 30 problems × 8 samples:

Step Blog (approx.) Miles v0.1.2 This branch
0 43.0% 40.4% 43.8%
10 43.5% 45.0% 43.8%
20 49.1% 46.2% 45.0%
30 52.9% 53.8% 58.8%
40 57.6% 64.2% 65.4%
50 59.2% — 63.7%
60 60.1% — 70.4%

Accuracy settles at 64–70% from step 40 onward. Mean response length falls from 3,164 tokens to about 2,200 by step 55. The trainer call takes 332–499 s per update, against 478–574 s on v0.1.2.

The chart also shows an earlier run of this branch on Miles f45c63c (before review) and SGLang 991fb375. It tracks this run closely.

Beam LoRA on this branch vs Miles v0.1.2 vs blog

Testing

  • CPU: pre-commit and the CPU suite pass (1,513 passed, 10 skipped). This includes regression tests for volume captures racing checkpoint persistence, and for a failed commit not failing the trainer.
  • Two-node checkpoints:
    • Qwen3.5-4B on 2×8 H100: after a trainer restart, weights and optimizer restore exactly. A step from the restored optimizer matches the original within 0.043 max logprob; a fresh optimizer differs by 3.51.
    • Beam update-10 checkpoint (from the earlier run), restored on a warm and on a cold trainer: when saved again, every tensor is bit-identical to the original (625 weights, 624 exp_avg, 624 exp_avg_sq).
  • Reference run: Spindle a049c55 with Miles d1915e68 and SGLang 568ab742, through update 70 so far. d1915e68 has the same tree as the merged 3280b7a5.
  • Not run:
    • single-node miles-v0.2.0 after this change;
    • the Beam images under their final names, which are built from the same commits;
    • an FP8 MoE model on sglang-v0.5.23-568ab74.

@frrad Fred Robinson (frrad) changed the title Add an experimental Beam BF16 LoRA recipe and pinned runtimes Add Beam BF16 LoRA support Nov 11, 2026
@frrad
Fred Robinson (frrad) marked this pull request as ready for review November 11, 2026 23:53
@frrad Fred Robinson (frrad) changed the title Add Beam BF16 LoRA support Add Beam BF16 LoRA support and a reference run Nov 12, 2026
@frrad
Fred Robinson (frrad) force-pushed the bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn branch from 49a3a98 to 594edf2 Compare November 16, 2026 21:13
Micah Yong (micahtyong) added a commit that referenced this pull request Nov 17, 2026
Merge #252's current head. Its Miles v0.2.0 image already includes the
Kimi K3 change, so the K3 recipe now uses trainer_image "miles-v0.2.0"
and the K3-only trainer image and the fork_image() helper are gone.
kimi_k3_image.py keeps only the sampler image.

Kimi K3 trains with megatron_to_hf_mode "raw", which Spindle only passes
to Miles, so miles-v0.2.0 no longer rejects it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/deployment-configs.md Resolved

def _gather_capture(self, written: Path, path: Path) -> None:
"""Make every node's writes to a capture on the checkpoint volume visible here."""
if not on_checkpoint_volume(path):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this necessary to restore multi-node on 0.2.0 or why this change?


image = image_at(MILES_COMMIT)
# Beam on the Megatron Bridge (modal-projects/miles#37).
beam_image = image_at("3280b7a5e3c4399c1ac18bed568859c9068852c3")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just call this image_ or something? ideally this is just a linear list of commits on our shared 0.2.0 fork branch. mb add a comment about this. is image_020_0001_sha mb better? ff to rename the existing one to be consistent

revision="803bccbcb947d3041f5b57e11c986f8523a0cb56",
core_packages=CORE_SAMPLER_PACKAGES,
).add_local_python_source("spindle", ignore=ignore_config_source)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same naming thing as above

def on_checkpoint_volume(path: Path) -> bool:
"""Whether ``path`` lives on the checkpoint volume, where nodes see only committed writes."""
return (
"SPINDLE_CHECKPOINT_VOLUME" in os.environ

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does this relate to the existing _checkpoint_volume_path function? can we do something cleaner?

def run(args, batch_id, model, *batch_args, **kwargs):
rollout_data = batch_args[-1]
def run(args, batch_id, model, *slot_and_data, **kwargs):
# v0.2.0 passes adapter_slots before rollout_data; preserve both native arguments.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait this is very scary - can you do something more explicit here? like actually check the version and branch on that potentially?

self._bridge.save_slot(slot, path, include_optimizer=include_optimizer)
)
# Outside _run: a bad capture fails this request, not the shared trainer.
self._gather_capture(written, Path(path))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we adding this?

Fred Robinson (frrad) added a commit that referenced this pull request Nov 17, 2026
Reuse the stock Tinker workload with a pinned tokenizer and the existing
TP2/CP8 H200 topology. Keep the response cap at 2048 tokens; this is not a
256K-context validation. Record no execution until a GPU run exists.

Enable Miles v0.2.0 multi-node captures by committing each node's files
before coordinator reload and sharing the checkpoint I/O lock for
volume-backed captures, including their metadata writes. Local captures
keep their independent persistence path.

Adapt the generic backend changes from #252 commits 53f80f7, 594edf2 and
18054ed, with its owner's agreement. Leave Beam recipes and images out.
Fred Robinson (frrad) added a commit that referenced this pull request Nov 17, 2026
Reuse the stock Tinker workload with a pinned tokenizer and the existing
TP2/CP8 H200 topology. Keep the response cap at 2048 tokens; this is not a
256K-context validation. Record no execution until a GPU run exists.

Enable Miles v0.2.0 multi-node captures by committing each node's files
before coordinator reload and sharing the checkpoint I/O lock for
volume-backed captures, including their metadata writes. Local captures
keep their independent persistence path.

Adapt the generic backend changes from #252 commits 53f80f7, 594edf2 and
18054ed, with its owner's agreement. Leave Beam recipes and images out.
@frrad
Fred Robinson (frrad) force-pushed the bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn branch from 18054ed to 96ba5c4 Compare November 17, 2026 23:53
@frrad
Fred Robinson (frrad) changed the base branch from main to reference-dapo-multinode November 17, 2026 23:53
@frrad
Fred Robinson (frrad) force-pushed the bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn branch 6 times, most recently from c09c2f4 to e36bfcc Compare November 18, 2026 03:52
@frrad
Fred Robinson (frrad) force-pushed the bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn branch from d58899e to f7ed75a Compare November 18, 2026 04:09
Miles v0.1.1 and v0.2.0 put rollout_data at different positions in
run_forward_backward, because v0.2.0 adds adapter_slots before it. The
replay wrapper now binds each call to the wrapped function's own
signature and reads rollout_data by name, so positional and keyword
calls work on both versions. Installing the hooks fails at trainer
startup when that function has no rollout_data parameter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
beam-501b-a23b-lora-5k trains reflection/Beam-1.0-501B-A23B with BF16
LoRA on two eight-H200 trainer nodes, with separate attention and MLP
gate targets and routing replay. It trains on miles-v0.2.0-8c5de26 and
samples on sglang-v0.5.23-e003cf9, which serves Beam LoRA.

reference_runs/configs/beam_blog_repro.toml selects that recipe and
scripts/beam_blog_repro.py, a stock-Tinker client that trains on
DAPO-Math, evaluates on AIME25 and can resume from checkpoints. The
index holds an earlier 42-update execution, converted to the catalog
output format, and the digitized curve from the published Beam LoRA
run, which records source instead of Spindle provenance.

catalog.py now defines the output format every reference script
writes (Record, Evaluation, Update and read_output), and tests check
every indexed output and each script's output against it. A Spindle
index entry needs its trainer image, Miles commit, Tinker SDK and
hardware; an external entry records only its source.

The Beam script and its tests are fully typed, without ANN ignores.
AGENTS.md and the ignore list say that list only shrinks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test checked that /proc/PID/stat existed and then read it. When the
killed descendant disappeared between the two calls, the read raised
ProcessLookupError and the test failed, although the supervisor had
done its job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frrad
Fred Robinson (frrad) force-pushed the bb/https-modal-com-slack-com-archives-d0ce0tq2f7f-p-thr_dbnae2xvsn branch from f7ed75a to b5140f9 Compare November 18, 2026 04:20

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant