Skip to content

[trainer throughput] Build the logits gradient once per micro-batch - #317

Merged
Micah Yong (micahtyong) merged 4 commits into
mainfrom
micah/logits-split-per-micro-batch
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 4 commits into
mainfrom
micah/logits-split-per-micro-batch

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

After the forward pass, the trainer scores each row's response tokens. Miles takes each row's response logits by slicing them out of the micro-batch's logits. In the backward pass, every slice writes a zero tensor the size of the whole micro-batch's logits, plus that row's gradient. A micro-batch of N rows therefore writes N full logits gradients. With a 248K-token vocabulary, each one is several GB.

This PR splits the micro-batch's logits once into the per-row pieces. The values and gradients are identical, and the backward pass writes the logits gradient once per micro-batch. It applies to every recipe on the default Miles images. A micro-batch with one row is unchanged.

E2E results

Setup: stock Tinker SDK, LoRA rank 32, 128 rows of about 1,600 tokens per step, one forward_backward and one optim_step per step. Times are the trainer's forward_backward median. Per-token log probabilities are bit-identical on Qwen3.5-4B and within run-to-run noise on Nemotron.

Recipe Before WandB After WandB forward_backward (s) Peak GB
qwen35-4b-lora-32k, 4x H100 Before After 11.51 → 3.67 35.5 → 33.2
nemotron35-lightning-30b-a3b-lora-64k, 2x H200, with PRs #244–#247 Before After 8.46 → 7.70 107.5 → 103.3

On Qwen3.5-4B, forward_backward used to climb from 6.0 s at step 1 to about 12 s by step 20. It now stays at 3.7 s.

Miles slices each row's response logits out of the micro-batch's logits.
Each slice's backward writes a full-size zero tensor plus the row's
gradient, so a micro-batch of N rows writes N full logits gradients. One
split per micro-batch gives the same values and gradient.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	src/spindle/backends/miles_runtime/actor.py
# Conflicts:
#	CHANGELOG.md
#	src/spindle/backends/miles_runtime/actor.py
@micahtyong
Micah Yong (micahtyong) merged commit f969ddc into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/logits-split-per-micro-batch branch November 17, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant