Skip to content

Add the two-node Qwen27B reference and Miles v0.2.0 capture support - #348

Open
Fred Robinson (frrad) wants to merge 3 commits into
reference-run-toolingfrom
reference-dapo-multinode
Open

Fred Robinson (frrad) wants to merge 3 commits into
reference-run-toolingfrom
reference-dapo-multinode

Conversation

@frrad

@frrad Fred Robinson (frrad) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Enable Miles v0.2.0 trainers to save checkpoints and publish adapters across nodes. Its capture APIs bypass the cross-node checkpoint writer used by v0.1.1; Spindle now commits every node's capture writes and reloads the coordinator's Volume before materializing the adapter. Volume-backed capture, persistence and publication share the checkpoint I/O lock so reloads cannot race open files. Local captures retain their existing concurrency.

Move qwen38-27b-lora-256k to miles-v0.2.0-8c5de26, keeping its fused LoRA/Bridge layout. Compiler cache isolation is not a bundled recipe default. Existing checkpoints require their original trainer image; they cannot restore with the new recipe default. The changelog states this explicitly.

Stack: this PR targets reference-run-tooling and depends on #372. The parent owns the generic launcher, tokenizer/config types and indexing. This layer owns the runtime/recipe changes, the Qwen3.8-27B definition, and its completed measurements. No generic tooling implementation is duplicated here. The generic capture changes were extracted from #252 with its author's agreement.

Validation

On 19c32a6:

  • Python 3.12, uv sync --locked --python 3.12.
  • Full uv run pre-commit run --all-files: passed, including ty.
  • Full CPU suite: 1,594 passed, 10 skipped in 120.54s. Tests cover capture gathering/failure and serialization with checkpoint persistence. The image-usage test now recognizes that the Qwen recipe uses the formerly unused v0.2.0 image.

GPU validation in progress

Both measurements use stock Tinker SDK 0.26.0, two eight-H200 trainer nodes (TP2/CP8), two four-H200 sampler replicas, one client, and the same 20-update DAPO-Math workload with a 2,048-response-token cap. This does not validate 256K sequences. Server-reported version is 0.1.7 for both frozen deployments; source identity and trainer revisions were checked in executing containers.

Run Frozen server commit Image Status
Baseline 8970cb1f69675e917db9baee9ad4b0abfbecdf61 miles-v0.1.1-52a7a99 Complete: all 20 updates accepted, final evaluation/checkpoint saved, client exit 0. Held-out accuracy 65/256 → 137/256; median interval 559.4s.
v0.2.0 retry 2b5cf4f13a6834bf680fbdaa3aa48329d961405b miles-v0.2.0-8c5de26 In progress: 10 accepted updates with finite metrics and adapter publication. Evaluations 0/5/10 and checkpoints 0/10 saved. Latest held-out result 102/256 at step 10, from 68/256 initially.

Completed baseline metrics and observed provenance are committed in this PR. No partial JSONL files are committed in either PR.

An earlier v0.2.0 attempt failed at its first backward pass when a worker read a generated TorchInductor module containing null bytes. The committed copy was valid when downloaded afterward, so its cause remains unconfirmed. The active retry uses run-only local trainer compiler-cache overrides. Their necessity has not been established; the bundled recipe retains its shared-cache defaults. Geographic placement also differs between runs. These one-seed measurements do not establish speed or learning parity.

Still required before merge: complete the v0.2.0 run through its final evaluation/checkpoint, then validate restoration from an updated checkpoint, including optimizer state and sampler publication on the shipped shared-cache configuration. Restore has not been tested. The rebased PR head has CPU coverage; the GPU runs use the frozen commits above.

Work-in-progress plot

Snapshot: completed baseline versus the retry through update 10. At the matched step-10 evaluation, accuracy is 104/256 versus 102/256. Timing spikes include scheduled evaluation/checkpoint overhead. Full comparison pending completion of the retry.

Qwen3.8-27B DAPO work-in-progress comparison

Base automatically changed from reference-dapo-qwen38 to main November 17, 2026 23:02
@frrad
Fred Robinson (frrad) changed the base branch from main to fix/restore-frozen-miles-image-names November 17, 2026 23:19
Base automatically changed from fix/restore-frozen-miles-image-names to main November 17, 2026 23:44
@frrad Fred Robinson (frrad) changed the title Add a two-node Qwen3.8-27B DAPO-Math reference Add a two-node Qwen27B DAPO reference and safe run recording Nov 18, 2026
@frrad Fred Robinson (frrad) changed the title Add a two-node Qwen27B DAPO reference and safe run recording Add a two-node Qwen3.8-27B DAPO reference and automatic run indexing Nov 18, 2026
@frrad
Fred Robinson (frrad) marked this pull request as ready for review November 18, 2026 01:57
Comment thread reference_runs/run.py Resolved
Comment thread reference_runs/run.py Resolved
@micahtyong

Copy link
Copy Markdown
Collaborator

Could we split this into two PRs based on main? The runtime change ships to users and the reference-run tooling doesn't, so they'd be easier to review and validate separately.

  1. Multi-node v0.2.0 captures. This covers the commit-on-every-node and reload-before-materialize change, sharing the checkpoint I/O lock, removing the single-node v0.2.0 check, and moving qwen38-27b-lora-256k to miles-v0.2.0-8c5de26 with the local Triton/Inductor caches. Before merging I'd want a two-node run that gets through several training steps, sampler weight syncs and a checkpoint save, then restores from that checkpoint. Right now the v0.2.0 run only reaches its initial evaluation and checkpoint. The image move also stops existing checkpoints for this recipe from restoring, so I'd like the changelog entry to say that clearly.
  2. Reference-run indexing and launcher changes. This covers catalog.py, the typed tokenizer and overrides, the new DAPO definition, and the partial JSONL snapshots. I left notes inline.

@frrad Fred Robinson (frrad) changed the title Add a two-node Qwen3.8-27B DAPO reference and automatic run indexing Enable Miles v0.2.0 multi-node captures and upgrade Qwen27B Nov 18, 2026
@frrad
Fred Robinson (frrad) changed the base branch from main to reference-run-tooling November 18, 2026 03:41
@frrad Fred Robinson (frrad) changed the title Enable Miles v0.2.0 multi-node captures and upgrade Qwen27B Add the two-node Qwen27B reference and Miles v0.2.0 capture support Nov 18, 2026
@frrad

Fred Robinson (frrad) commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Good call. I extracted the reference run machinery into #372 and stacked this PR onto that one.

The 0.2.0 validation run is still in-flight but otherwise both this PR and that one are ready for another look.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants