Repository navigation
Add opt-in Miles v0.2.0 runtime and native checkpoints - #289
Merged
Fred Robinson (frrad) merged 24 commits intoNov 16, 2026
Merged
Fred Robinson (frrad) merged 24 commits into
Fred Robinson (frrad) merged 24 commits into
Conversation
Fred Robinson (frrad)
force-pushed
the
bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji
branch
2 times, most recently
from
November 12, 2026 22:29
8714039 to
575dc5c
Compare
Fred Robinson (frrad)
force-pushed
the
bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji
branch
from
November 13, 2026 19:35
575dc5c to
9063c35
Compare
miles#38 now replays the original Spindle trainer fixes onto v0.2.0 as a94f7ae, replacing e6c491a, which is no longer on any branch. Adapter exports copy to CPU only inside materialize_adapter(), which the native checkpoint writer uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native captures had moved to the checkpoint volume on every topology because materialize_adapter only handled volume paths. Each sampler export then paid a barrier, two volume commits and a volume listing, about 11-20 s per update on one node. Captures now go to the configured capture directory as legacy captures do: local disk on one node, the checkpoint volume on several. Actors commit and materialize_adapter reads the volume only for captures on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spindle never set --lora-type, so each Miles image's default chose the adapter layout. v0.1.x defaults to lora, one lora_A shared across a fused projection. Miles v0.2.0 defaults to canonical_lora for serve entries, which gives each projection its own lora_A. The v0.2.0 runtime therefore trained a different adapter than v0.1.x for the same recipe. Recipes now name the layout, defaulting to lora on every image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass --lora-type lora on every image instead of exposing a recipe field: Miles v0.1.x multi-LoRA supports only lora, and v0.2.0 would otherwise default serve entries to canonical_lora. Hold the checkpoint I/O lock around captures and sampler publication only for native captures on the checkpoint volume, which a restore's volume reload could disturb. Single-node captures and every legacy image no longer wait behind a checkpoint persist. Drop checks that repeat Miles: slot shape checks before a native load (Miles' check_load_source compares keys and shapes) and the duplicated argument setup in v020. Read checkpoint_format from the runtime directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
actor_v020 imported actor.py for three helpers, which also ran every legacy import-time patch inside the v0.2.0 image: host synchronization (then disabled by an environment flag), BSHD context-parallel LoRA batching and cross-node write_checkpoint_dir publishing, none of which the native path uses. actor_shared.py now holds the profiling mixin, gloo timeout, startup mark and the replay patches both images need, with no import-time effects. Each actor installs exactly its own patches. The v0.2.0 image drops the now-unused SPINDLE_HOST_SYNC_PATCHES flag. The legacy actor tests import modules that need real torch before stubbing it and load a fresh actor_shared per test, so the file passes on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… failures Reject miles-v0.2.0 recipes that use raw mode, where Miles v0.2.0 requires canonical_lora but Spindle passes --lora-type lora, or more than one trainer node, which has never run. With captures now always on local disk, remove the code that served only multi-node captures: the volume branch and staging copy in materialize_adapter, the capture lock and the per-capture volume commits. materialize_adapter validates Miles' directory in place and renames it. Run materialize_adapter after the runtime call instead of inside it, as the legacy path does, so a capture that fails validation fails only its request instead of marking the shared trainer unavailable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore main's checkpoint metadata validation: rejecting a missing or non-integer optimizer_step affects every image and belongs in its own change. Stop forwarding SPINDLE_MILES_API and SPINDLE_CHECKPOINT_ROOT to Ray workers: the image already sets the first, and production never sets the second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop recording and checking LoRA dropout in native checkpoint metadata: it is not part of the parametrization Spindle sends Miles and changes no stored tensor. Keep the target-module check, because Miles rejecting a mismatch disables the whole trainer. Fold the missing-storage case into the damaged checkpoint test, drop the concurrent-capture test, which shared no state, and order actor_v020's imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the earlier results, recorded in canonical_lora layout on Miles e6c491a, with the 20-update reference run on Miles a94f7ae in the lora layout both images now train, on the fixed-hardware reference recipe. Document the restore round trip, the single-node bridge-mode scope, the LoRA layout and the reason older checkpoints are rejected on v0.2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fred Robinson (frrad)
force-pushed
the
bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji
branch
from
November 13, 2026 21:55
9063c35 to
016debf
Compare
Replace the run recorded on 619dc8b with one on 016debf, deployed with the frontend, trainer and samplers in us-east; the recorded deployment now carries that region. Keep the results README to what identifies the run; the data is metrics.jsonl. Document the GPU restore, weights-only, slot isolation and reproducibility checks, and the upload stalls seen with the trainer unpinned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorded deployment and metrics identify the run; the docs' Validation section names the Spindle and Miles commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace docs/miles-v020.md with the user-facing facts in the Images section of the deployment docs: which images exist, the v0.2.0 limits, the shared adapter layout and the checkpoint revision rule. Image pins live in code; migration notes and validation are in the pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the build step that imported the v0.2.0 actor on a GPU. It required baking Spindle's source into the image, so every Spindle change rebuilt the image and queued an H100 build step. The trainer imports the same modules at startup, and the source is now mounted at runtime as on the other images. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
This is basically ready for review, just needs to have its miles dep re-pointed after I merge modal-projects/miles#38 |
Fred Robinson (frrad)
marked this pull request as ready for review
November 13, 2026 23:57
Fred Robinson (frrad)
requested review from
Kailash Ranganathan (kailash109) and
Micah Yong (micahtyong)
as code owners
November 13, 2026 23:57
Micah Yong (micahtyong)
approved these changes
Nov 15, 2026
Micah Yong (micahtyong)
left a comment
Collaborator
There was a problem hiding this comment.
lgtm pending comments + repointing the miles pin
modal-projects/miles#38 was squash-merged into spindle-miles-v0.2.0 as 15afe21. That branch also carries the Kimi-K3 W4A16 LoRA change (miles#39), which the earlier pin a94f7ae did not include. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a native checkpoint or sampler export fails validation, or its target already exists, delete the hidden directory Miles wrote. Miles removes it only when its own write fails, so each rejected capture used to leave a full adapter on the trainer's disk. The location check still runs first, so only Miles's directory next to the target is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fred Robinson (frrad)
deleted the
bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji
branch
November 16, 2026 05:53
Fred Robinson (frrad)
added a commit
that referenced
this pull request
Nov 16, 2026
Main's Miles v0.2.0 runtime (#289) writes checkpoint_format from the runtime and adds lora_config.target_modules for native adapter checkpoints. Both go through CheckpointMetadata now; target_modules is an optional field and is only written for native checkpoints, so torch_dist files keep their shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
miles-v0.2.0trainer image, selected withtrainer_image = "miles-v0.2.0". Existing images and recipes are unchanged.Miles v0.2.0 replaces the slot checkpoint and export calls Spindle used with a native adapter-directory API: Miles writes each checkpoint or sampler export to an uncommitted directory and leaves committing it to the caller. The new path (
miles_runtime/v020.py,actor_v020.py,native_checkpoint.py) translates Spindle's calls. Before renaming a directory into place, Spindle checks that a checkpoint has every shard its metadata references at full length, and that a sampler export has its config and weights. A capture that fails these checks fails only its request.--lora-type lorato every image. Miles v0.1.x already defaults to it; v0.2.0 would otherwise default tocanonical_lora, a separatelora_Aper projection, and train a different adapter for the same recipe. Both images now export the same PEFT tensors to SGLang.bridgemode only; more nodes orrawmode are rejected at deploy time.checkpoint_format: miles_adapter_state. As before, restores require the Miles revision that saved them, so older checkpoints restore only on their original image.actor_shared.py, which patches nothing on import. The v0.2.0 actor no longer imports the legacy actor, so host-sync patches, BSHD context-parallel batching and cross-nodewrite_checkpoint_dirpublishing stay on the existing images. The legacy actor installs the same patches in the same order.15afe21onspindle-miles-v0.2.0, the merged Replay Spindle trainer fixes onto Miles v0.2.0 modal-projects/miles#38, replays Spindle's existing fixes onto v0.2.0: non-gated MoE expert targets,cp_comm_typefor both Bridge builders, and CPU staging when materializing adapters. DeepSeek-V4.1 raw multi-LoRA, FP8/FP4 loading, THD sample boundaries and Engram tuning stay on themiles-v0.1.2image; the GLM image is unchanged. The pin also includes the Kimi-K3 W4A16 LoRA support from Kimi-K3 W4A16 LoRA for Spindle modal-projects/miles#39, which landed on that branch first.Testing
GPU runs used the stock Tinker SDK 0.26.0, Qwen/Qwen3.5-4B with rank-32 LoRA, Miles
a94f7ae(the head of miles#38 before it merged), and four H100 trainer GPUs.016debf), reference recipe on the new image, frontend, trainer and samplers inus-east; all accepted. Deployment and metrics.One run per image, with about ±3 points of noise per evaluation, so learning parity is not established. The v0.1.1 reference ran on older code with the trainer region unpinned, so the faster client-side timings partly reflect placement: an earlier unpinned v0.2.0 run saw frontend-to-trainer uploads stall for 20–50 s on 7 of 20 updates, independent of the Miles version.
15afe21(a36f67d), a rerun of the same reference deployment completed 7 of 20 updates before it was stopped, all accepted. Training reward, response length and seconds between updates tracked thea94f7aerun; held-out accuracy was 16.0% before training and 21.5% at update 5. The restore, slot isolation and reproducibility checks were not repeated on15afe21.Not run: multi-node, raw and BSHD recipes, routing replay, and models other than Qwen3.5-4B.
🤖 Generated with Claude Code