Skip to content

Add opt-in Miles v0.2.0 runtime and native checkpoints - #289

Merged
Fred Robinson (frrad) merged 24 commits into
mainfrom
bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji
Nov 16, 2026
Merged

Fred Robinson (frrad) merged 24 commits into
mainfrom
bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji

Conversation

@frrad

@frrad Fred Robinson (frrad) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in miles-v0.2.0 trainer image, selected with trainer_image = "miles-v0.2.0". Existing images and recipes are unchanged.

Miles v0.2.0 replaces the slot checkpoint and export calls Spindle used with a native adapter-directory API: Miles writes each checkpoint or sampler export to an uncommitted directory and leaves committing it to the caller. The new path (miles_runtime/v020.py, actor_v020.py, native_checkpoint.py) translates Spindle's calls. Before renaming a directory into place, Spindle checks that a checkpoint has every shard its metadata references at full length, and that a sampler export has its config and weights. A capture that fails these checks fails only its request.

  • Same adapter as today. Spindle now passes --lora-type lora to every image. Miles v0.1.x already defaults to it; v0.2.0 would otherwise default to canonical_lora, a separate lora_A per projection, and train a different adapter for the same recipe. Both images now export the same PEFT tensors to SGLang.
  • Scope. Single-node trainers in bridge mode only; more nodes or raw mode are rejected at deploy time.
  • Checkpoints. Native checkpoints record checkpoint_format: miles_adapter_state. As before, restores require the Miles revision that saved them, so older checkpoints restore only on their original image.
  • Actors. Shared helpers and the replay patches both images need move to actor_shared.py, which patches nothing on import. The v0.2.0 actor no longer imports the legacy actor, so host-sync patches, BSHD context-parallel batching and cross-node write_checkpoint_dir publishing stay on the existing images. The legacy actor installs the same patches in the same order.
  • Miles pin. Fork commit 15afe21 on spindle-miles-v0.2.0, the merged Replay Spindle trainer fixes onto Miles v0.2.0 modal-projects/miles#38, replays Spindle's existing fixes onto v0.2.0: non-gated MoE expert targets, cp_comm_type for both Bridge builders, and CPU staging when materializing adapters. DeepSeek-V4.1 raw multi-LoRA, FP8/FP4 loading, THD sample boundaries and Engram tuning stay on the miles-v0.1.2 image; the GLM image is unchanged. The pin also includes the Kimi-K3 W4A16 LoRA support from Kimi-K3 W4A16 LoRA for Spindle modal-projects/miles#39, which landed on that branch first.

Testing

image

GPU runs used the stock Tinker SDK 0.26.0, Qwen/Qwen3.5-4B with rank-32 LoRA, Miles a94f7ae (the head of miles#38 before it merged), and four H100 trainer GPUs.

  • Reference run: 20 DAPO-Math updates on the final runtime code (016debf), reference recipe on the new image, frontend, trainer and samplers in us-east; all accepted. Deployment and metrics.
  • Restores, compared tensor by tensor: restoring with optimizer into a cold trainer and saving again is bitwise equal to the source (weights, Adam moments, step). A weights-only restore keeps fp32 weights bitwise and resets moments and step; on v0.1.1 it rounds the masters to bf16. Training continues from a restored step.
  • Slot isolation: two models with interleaved requests on one trainer ended as close to their serial controls as two serial runs of the same work end to each other.
  • Reproducibility: repeating one update from the same restored checkpoint is not bitwise equal on either image (up to about 40% of the update, largest in the first layers). Qwen3.5 recipes don't request deterministic kernels.
  • Legacy image: a smoke run on v0.1.1 with the final code passed, with its patches installed. CPU: 1,324 passed, 8 skipped.
Miles v0.1.1 (reference) Miles v0.2.0
Held-out accuracy, before / update 20 20.7% / 41.8% 19.9% / 40.2%
Median seconds between updates 73.5 64.1
Median trainer backward (s) 8.9 9.9
Median SDK backward (s) 16.6 12.0
Median publication (s) 7.9 2.7

One run per image, with about ±3 points of noise per evaluation, so learning parity is not established. The v0.1.1 reference ran on older code with the trainer region unpinned, so the faster client-side timings partly reflect placement: an earlier unpinned v0.2.0 run saw frontend-to-trainer uploads stall for 20–50 s on 7 of 20 updates, independent of the Miles version.

  • Final pin: after repointing to 15afe21 (a36f67d), a rerun of the same reference deployment completed 7 of 20 updates before it was stopped, all accepted. Training reward, response length and seconds between updates tracked the a94f7ae run; held-out accuracy was 16.0% before training and 21.5% at update 5. The restore, slot isolation and reproducibility checks were not repeated on 15afe21.

Not run: multi-node, raw and BSHD recipes, routing replay, and models other than Qwen3.5-4B.

🤖 Generated with Claude Code

@frrad
Fred Robinson (frrad) force-pushed the bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji branch 2 times, most recently from 8714039 to 575dc5c Compare November 12, 2026 22:29
Base automatically changed from reference-runs to main November 13, 2026 16:48
@frrad
Fred Robinson (frrad) force-pushed the bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji branch from 575dc5c to 9063c35 Compare November 13, 2026 19:35
Fred Robinson (frrad) and others added 18 commits November 13, 2026 21:37
miles#38 now replays the original Spindle trainer fixes onto v0.2.0 as
a94f7ae, replacing e6c491a, which is no longer on any branch. Adapter
exports copy to CPU only inside materialize_adapter(), which the native
checkpoint writer uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native captures had moved to the checkpoint volume on every topology because
materialize_adapter only handled volume paths. Each sampler export then paid a
barrier, two volume commits and a volume listing, about 11-20 s per update on
one node. Captures now go to the configured capture directory as legacy
captures do: local disk on one node, the checkpoint volume on several. Actors
commit and materialize_adapter reads the volume only for captures on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spindle never set --lora-type, so each Miles image's default chose the
adapter layout. v0.1.x defaults to lora, one lora_A shared across a fused
projection. Miles v0.2.0 defaults to canonical_lora for serve entries,
which gives each projection its own lora_A. The v0.2.0 runtime therefore
trained a different adapter than v0.1.x for the same recipe. Recipes now
name the layout, defaulting to lora on every image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass --lora-type lora on every image instead of exposing a recipe field:
Miles v0.1.x multi-LoRA supports only lora, and v0.2.0 would otherwise
default serve entries to canonical_lora.

Hold the checkpoint I/O lock around captures and sampler publication only
for native captures on the checkpoint volume, which a restore's volume
reload could disturb. Single-node captures and every legacy image no
longer wait behind a checkpoint persist.

Drop checks that repeat Miles: slot shape checks before a native load
(Miles' check_load_source compares keys and shapes) and the duplicated
argument setup in v020. Read checkpoint_format from the runtime directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
actor_v020 imported actor.py for three helpers, which also ran every legacy
import-time patch inside the v0.2.0 image: host synchronization (then
disabled by an environment flag), BSHD context-parallel LoRA batching and
cross-node write_checkpoint_dir publishing, none of which the native path
uses. actor_shared.py now holds the profiling mixin, gloo timeout, startup
mark and the replay patches both images need, with no import-time effects.
Each actor installs exactly its own patches. The v0.2.0 image drops the
now-unused SPINDLE_HOST_SYNC_PATCHES flag.

The legacy actor tests import modules that need real torch before stubbing
it and load a fresh actor_shared per test, so the file passes on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… failures

Reject miles-v0.2.0 recipes that use raw mode, where Miles v0.2.0 requires
canonical_lora but Spindle passes --lora-type lora, or more than one trainer
node, which has never run. With captures now always on local disk, remove the
code that served only multi-node captures: the volume branch and staging copy
in materialize_adapter, the capture lock and the per-capture volume commits.
materialize_adapter validates Miles' directory in place and renames it.

Run materialize_adapter after the runtime call instead of inside it, as the
legacy path does, so a capture that fails validation fails only its request
instead of marking the shared trainer unavailable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore main's checkpoint metadata validation: rejecting a missing or
non-integer optimizer_step affects every image and belongs in its own change.
Stop forwarding SPINDLE_MILES_API and SPINDLE_CHECKPOINT_ROOT to Ray workers:
the image already sets the first, and production never sets the second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stop recording and checking LoRA dropout in native checkpoint metadata: it is
not part of the parametrization Spindle sends Miles and changes no stored
tensor. Keep the target-module check, because Miles rejecting a mismatch
disables the whole trainer. Fold the missing-storage case into the damaged
checkpoint test, drop the concurrent-capture test, which shared no state, and
order actor_v020's imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the earlier results, recorded in canonical_lora layout on Miles
e6c491a, with the 20-update reference run on Miles a94f7ae in the lora layout
both images now train, on the fixed-hardware reference recipe. Document the
restore round trip, the single-node bridge-mode scope, the LoRA layout and
the reason older checkpoints are rejected on v0.2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frrad
Fred Robinson (frrad) force-pushed the bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji branch from 9063c35 to 016debf Compare November 13, 2026 21:55
Replace the run recorded on 619dc8b with one on 016debf, deployed with the
frontend, trainer and samplers in us-east; the recorded deployment now carries
that region. Keep the results README to what identifies the run; the data is
metrics.jsonl. Document the GPU restore, weights-only, slot isolation and
reproducibility checks, and the upload stalls seen with the trainer unpinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The recorded deployment and metrics identify the run; the docs' Validation
section names the Spindle and Miles commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace docs/miles-v020.md with the user-facing facts in the Images section
of the deployment docs: which images exist, the v0.2.0 limits, the shared
adapter layout and the checkpoint revision rule. Image pins live in code;
migration notes and validation are in the pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the build step that imported the v0.2.0 actor on a GPU. It required
baking Spindle's source into the image, so every Spindle change rebuilt the
image and queued an H100 build step. The trainer imports the same modules at
startup, and the source is now mounted at runtime as on the other images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frrad

Copy link
Copy Markdown
Contributor Author

This is basically ready for review, just needs to have its miles dep re-pointed after I merge modal-projects/miles#38

@frrad
Fred Robinson (frrad) marked this pull request as ready for review November 13, 2026 23:57
Comment thread tests/backends/test_miles_runtime.py Resolved

@micahtyong Micah Yong (micahtyong) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm pending comments + repointing the miles pin

modal-projects/miles#38 was squash-merged into spindle-miles-v0.2.0 as
15afe21. That branch also carries the Kimi-K3 W4A16 LoRA change
(miles#39), which the earlier pin a94f7ae did not include.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a native checkpoint or sampler export fails validation, or its
target already exists, delete the hidden directory Miles wrote. Miles
removes it only when its own write fails, so each rejected capture used
to leave a full adapter on the trainer's disk. The location check still
runs first, so only Miles's directory next to the target is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frrad
Fred Robinson (frrad) merged commit 0714567 into main Nov 16, 2026
3 checks passed
@frrad
Fred Robinson (frrad) deleted the bb/scope-work-to-the-migration-only-don-t-deploy-thr_npxpp9h8ji branch November 16, 2026 05:53
Fred Robinson (frrad) added a commit that referenced this pull request Nov 16, 2026
Main's Miles v0.2.0 runtime (#289) writes checkpoint_format from the
runtime and adds lora_config.target_modules for native adapter
checkpoints. Both go through CheckpointMetadata now; target_modules is
an optional field and is only written for native checkpoints, so
torch_dist files keep their shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants