Repository navigation
[spindle version] Release 0.1.8 - #360
Open
Micah Yong (micahtyong) wants to merge 2 commits into
Open
Micah Yong (micahtyong) wants to merge 2 commits into
Micah Yong (micahtyong) wants to merge 2 commits into
Conversation
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 23:20
Micah Yong (micahtyong)
force-pushed
the
micah/release-0.1.8
branch
2 times, most recently
from
November 18, 2026 00:50
f748c48 to
97f904d
Compare
Micah Yong (micahtyong)
force-pushed
the
micah/release-0.1.8
branch
from
November 18, 2026 01:21
97f904d to
83cc4da
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
force-pushed
the
micah/release-0.1.8
branch
from
November 18, 2026 02:02
83cc4da to
bd9752d
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Releases 0.1.8. The headline is that the Qwen3.5 LoRA recipes now train the linear-attention layers, which are three of every four layers. The dense Qwen and Nemotron-3 Nano LoRA recipes move to Miles native adapters, so checkpoints they saved before 0.1.8 load only on their original trainer image. A new recipe trains Nemotron 3.5 Super VL 120B-A12B on text. The release also returns temperature-1 logprobs for sampled tokens, scores multi-target batches with mixed candidate counts in one pass, and rewrites the profiling guide around tuning a deployment config.
spindle.__version__is now0.1.8.CHANGELOG.md: theUnreleasedsection and the undated0.1.8section become one0.1.8 - 2026-11-17section with a one-line headline and one heading each for Added, Changed, Removed and Fixed. It gains entries for PRs merged after thev0.1.7tag that had none: [trainer memory] Score full fine-tuning log-probs in token chunks #335 (megatron_cfg.logprob_chunk_tokens), Rewrite the profiling guide around tuning a deployment config #349 (profiling guide), [sampler throughput] Sample Nemotron-3.5-Lightning with its native MTP layer #323 (Nemotron-3.5-Lightning MTP sampling), [trainer throughput] Train Qwen3.5-4B FFT 64K data-parallel on four B300s or B200s #337–[trainer throughput] Train Qwen3.5/3.6-35B-A3B FFT 64K with expert parallelism only, on B300 or B200 #340 (64K full fine-tuning layouts), [trainer throughput] Compile FLA linear-attention kernels once instead of per batch length #333/[trainer throughput] Pass max_seqlen as an int so FlashAttention 4 compiles its backward once #334/[trainer throughput] Balance rows across data-parallel ranks and pad with short micro-batches #336 (full fine-tuning kernel compiles and row balancing), Revert the SPINDLE_EXCLUDE_RESERVED_CAPACITY environment variable #306 (SPINDLE_EXCLUDE_RESERVED_CAPACITYremoved), [trainer correctness] Restart Mamba state at each row packed into a micro-batch #244 (Mamba state in packed rows) and Let clients resubmit samples that failed on a registry read #291 (lost samples on registry reads). Entries from [audit fix] Support prompt_logprobs_last_n and reject unsupported sampling options with a 400 #344, Restore frozen Miles image names and name images by fork commit #343, Use Miles native canonical LoRA for dense Qwen and Nemotron Nano #312, Keep Miles LoRA trainers serving while a profiled step's trace is written #359, Group spindle usage cost by several labels #354 and Remove unused Miles images and require config coverage #361, which merged after this PR opened, are folded in too. [recipe] Sample Nemotron-3.5-Lightning 64K on B200 #364 is folded into the Nemotron-3.5-Lightning MTP entry. The seeded-images entry again says which images restore 0.1.7 training states, which Keep the Miles images that restore 0.1.7 checkpoints #370 keeps; merge Keep the Miles images that restore 0.1.7 checkpoints #370 first. Earlier entries no longer name images that Remove unused Miles images and require config coverage #361 removed: the image-name entry now points tomiles-v0.1.2-99e1ebe,miles-v0.2.0-8c5de26and the defaultmiles-v0.1.1-52a7a99, and the Qwen3.5 linear-attention entry no longer namessglang-v0.5.23-803bccb. Themiles-v0.2.0entry from Add opt-in Miles v0.2.0 runtime and native checkpoints #289 moves from 0.1.7 to 0.1.8, since it merged after the tag. The two-GPUmiles-v0.2.0start-up fix (Start Ray with enough CPUs for two-GPU Miles v0.2.0 trainers #345) is dropped, because that image is new in this release.@v0.1.8.When this merges, the release workflow tags
v0.1.8and publishes a GitHub Release from the0.1.8changelog section.E2E
Ran on
83cc4da9, this PR rebased onto #365. The head is nowbd9752d7, which adds only #361. #361 removes images that none of the configs below use. Clients used the stock Tinker SDK 0.39.1 throughexamples/shared-server/with_proxy_auth.py.get_server_capabilitiesreported0.1.8and83cc4da9on both deployments.Qwen3.5-4B. A
qwen35_4b_lora_32ksubclass ran with two B200 trainer GPUs, one sampler replica and nolm_headtarget, because the stock RL example passestrain_unembed=False. Images and adapter layout are the recipe's own. It was deployed withSPINDLE_TORCH_PROFILE_STEP=2.scripts/wandb_rl_example.py, 5 stepsrank0.trace.json.gzandrank0.key_averages.txtwritten. Step 3'sforward_backwardtook 0.61 s, against 0.56 s for step 2.compute_logprobsand 0.006 vs trainerforward(492 tokens)target_tokens[47, 3], [47, 1], [47, 5]seed=1234twice, one step eachforwardandcompute_logprobs; 0.37 vsseed=4321in_proj_q/k/v/z/a/bandout_projadapters on all 24 linear-attention layersNemotron 3.5 Super VL 120B-A12B.
nemotron35-super-vl-120b-a12b-lora-64kran as shipped: an 8×H200 trainer and a 4×H200 sampler, capped at one sampler replica. The recipe doesn't trainlm_head, so the stock example'strain_unembed=Falsematches. Text only.scripts/wandb_rl_example.py, 3 stepscompute_logprobsand 0.035 vs trainerforward(458 tokens)spindle stopstopped both deployments afterwards.Not run: the shared server, full fine-tuning and DFlash recipes.