Repository navigation
Add the two-node Qwen27B reference and Miles v0.2.0 capture support - #348
Open
Fred Robinson (frrad) wants to merge 3 commits into
Open
Fred Robinson (frrad) wants to merge 3 commits into
Fred Robinson (frrad) wants to merge 3 commits into
Conversation
Fred Robinson (frrad)
force-pushed
the
reference-dapo-multinode
branch
from
November 17, 2026 23:06
924d653 to
c2d5b7d
Compare
Fred Robinson (frrad)
changed the base branch from
main
to
fix/restore-frozen-miles-image-names
November 17, 2026 23:19
Fred Robinson (frrad)
force-pushed
the
reference-dapo-multinode
branch
from
November 17, 2026 23:22
c2d5b7d to
47141f0
Compare
Base automatically changed from
fix/restore-frozen-miles-image-names
to
main
November 17, 2026 23:44
Fred Robinson (frrad)
marked this pull request as ready for review
November 18, 2026 01:57
Fred Robinson (frrad)
requested review from
Kailash Ranganathan (kailash109) and
Micah Yong (micahtyong)
as code owners
November 18, 2026 01:57
Collaborator
|
Could we split this into two PRs based on main? The runtime change ships to users and the reference-run tooling doesn't, so they'd be easier to review and validate separately.
|
Fred Robinson (frrad)
changed the base branch from
main
to
reference-run-tooling
November 18, 2026 03:41
Fred Robinson (frrad)
force-pushed
the
reference-dapo-multinode
branch
from
November 18, 2026 03:48
b06e49f to
6a3479d
Compare
Fred Robinson (frrad)
force-pushed
the
reference-dapo-multinode
branch
from
November 18, 2026 04:06
6a3479d to
3996abc
Compare
Contributor
Author
|
Good call. I extracted the reference run machinery into #372 and stacked this PR onto that one. The 0.2.0 validation run is still in-flight but otherwise both this PR and that one are ready for another look. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable Miles v0.2.0 trainers to save checkpoints and publish adapters across nodes. Its capture APIs bypass the cross-node checkpoint writer used by v0.1.1; Spindle now commits every node's capture writes and reloads the coordinator's Volume before materializing the adapter. Volume-backed capture, persistence and publication share the checkpoint I/O lock so reloads cannot race open files. Local captures retain their existing concurrency.
Move
qwen38-27b-lora-256ktomiles-v0.2.0-8c5de26, keeping its fused LoRA/Bridge layout. Compiler cache isolation is not a bundled recipe default. Existing checkpoints require their original trainer image; they cannot restore with the new recipe default. The changelog states this explicitly.Stack: this PR targets
reference-run-toolingand depends on #372. The parent owns the generic launcher, tokenizer/config types and indexing. This layer owns the runtime/recipe changes, the Qwen3.8-27B definition, and its completed measurements. No generic tooling implementation is duplicated here. The generic capture changes were extracted from #252 with its author's agreement.Validation
On 19c32a6:
uv sync --locked --python 3.12.uv run pre-commit run --all-files: passed, including ty.GPU validation in progress
Both measurements use stock Tinker SDK 0.26.0, two eight-H200 trainer nodes (TP2/CP8), two four-H200 sampler replicas, one client, and the same 20-update DAPO-Math workload with a 2,048-response-token cap. This does not validate 256K sequences. Server-reported version is 0.1.7 for both frozen deployments; source identity and trainer revisions were checked in executing containers.
8970cb1f69675e917db9baee9ad4b0abfbecdf61miles-v0.1.1-52a7a992b5cf4f13a6834bf680fbdaa3aa48329d961405bmiles-v0.2.0-8c5de26Completed baseline metrics and observed provenance are committed in this PR. No partial JSONL files are committed in either PR.
An earlier v0.2.0 attempt failed at its first backward pass when a worker read a generated TorchInductor module containing null bytes. The committed copy was valid when downloaded afterward, so its cause remains unconfirmed. The active retry uses run-only local trainer compiler-cache overrides. Their necessity has not been established; the bundled recipe retains its shared-cache defaults. Geographic placement also differs between runs. These one-seed measurements do not establish speed or learning parity.
Still required before merge: complete the v0.2.0 run through its final evaluation/checkpoint, then validate restoration from an updated checkpoint, including optimizer state and sampler publication on the shipped shared-cache configuration. Restore has not been tested. The rebased PR head has CPU coverage; the GPU runs use the frozen commits above.
Work-in-progress plot
Snapshot: completed baseline versus the retry through update 10. At the matched step-10 evaluation, accuracy is 104/256 versus 102/256. Timing spikes include scheduled evaluation/checkpoint overhead. Full comparison pending completion of the retry.