Repository navigation
[trainer memory] Score full fine-tuning log-probs in token chunks - #335
Merged
Merged
Conversation
Qwen3.5's 248K vocabulary makes full FP32 logits for a 64K-token micro-batch 60 GiB, and cross entropy keeps a second copy for backward, so one 64K row cannot train without tensor or context parallelism. logprob_chunk_tokens computes the LM head and log-softmax a chunk at a time and recomputes each chunk in backward. It requires tensor parallel size 1. A 64K row now peaks at 58 GiB on one GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask the model for unconverted logits whenever the output layer is deferred, so a bf16 hidden is never upcast before it meets the bf16 weight; cast the hidden to the weight dtype on the non-FP32 path. Refuse the setting on the LoRA backend, with MuP logit scaling, or when no output layer is found. Share one logits helper for the chunked and replay paths and test the output-layer keyword contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
marked this pull request as ready for review
November 17, 2026 05:21
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 05:21
This was referenced Nov 17, 2026
Merged
Micah Yong (micahtyong)
deleted the
micah/q4fft/b200-chunked-logprobs
branch
November 17, 2026 17:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
To score a row, the trainer multiplies every token's hidden state by the output layer and takes a log-softmax over the vocabulary. Qwen3.5 has a 248K-token vocabulary, so a 64K-token row needs a [65,536 × 248,320] logits matrix. In fp32 that is 65 GB, and the backward pass needs about as much again for its gradient. This is why a 64K row runs out of memory on a B200 unless the recipe splits it across GPUs with tensor or context parallelism, and that splitting slows every step down.
This PR adds a
megatron_cfgsetting,logprob_chunk_tokens. When it is set, the full fine-tuning backend scores tokens in chunks of that size. Each chunk's logits are freed after use and recomputed in the backward pass, so memory no longer grows with row length. The math is the same as the unchunked path, including fp32 scoring whenfp32_lm_headis on. Each chunk peaks at roughly 2–3 × chunk × vocabulary × 4 bytes; 8,192 tokens is about 16–24 GB for Qwen3.5.The setting needs tensor and pipeline parallel size 1, and LoRA recipes reject it. Replay losses (
mis,score_centered,score_centered_mis) still build the full logits. No bundled recipe turns it on in this PR.E2E results
Setup:
qwen35-4b-fft-64kon 4×B200 with tensor parallel 1 (pure data parallel), stock Tinker SDK, 4 rows of 16,384 tokens per step with cross entropy, 4 steps. The paired runs differ only inlogprob_chunk_tokens(unset vs 8192). The 64K row is one 65,535-token row per GPU.fp32_lm_headwith TF32: Before W&B · After W&B · 64K row W&Bfp32_lm_headoff: Before W&B · After W&B · 64K row W&BRows that already fit get about 8% slower, because the LM-head matmul runs again in the backward pass. The setting is for recipes where it lets a long row fit without tensor or context parallelism.
🤖 Generated with Claude Code