Skip to content

[trainer memory] Score full fine-tuning log-probs in token chunks - #335

Merged
Micah Yong (micahtyong) merged 4 commits into
mainfrom
micah/q4fft/b200-chunked-logprobs
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 4 commits into
mainfrom
micah/q4fft/b200-chunked-logprobs

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

To score a row, the trainer multiplies every token's hidden state by the output layer and takes a log-softmax over the vocabulary. Qwen3.5 has a 248K-token vocabulary, so a 64K-token row needs a [65,536 × 248,320] logits matrix. In fp32 that is 65 GB, and the backward pass needs about as much again for its gradient. This is why a 64K row runs out of memory on a B200 unless the recipe splits it across GPUs with tensor or context parallelism, and that splitting slows every step down.

This PR adds a megatron_cfg setting, logprob_chunk_tokens. When it is set, the full fine-tuning backend scores tokens in chunks of that size. Each chunk's logits are freed after use and recomputed in the backward pass, so memory no longer grows with row length. The math is the same as the unchunked path, including fp32 scoring when fp32_lm_head is on. Each chunk peaks at roughly 2–3 × chunk × vocabulary × 4 bytes; 8,192 tokens is about 16–24 GB for Qwen3.5.

The setting needs tensor and pipeline parallel size 1, and LoRA recipes reject it. Replay losses (mis, score_centered, score_centered_mis) still build the full logits. No bundled recipe turns it on in this PR.

E2E results

Setup: qwen35-4b-fft-64k on 4×B200 with tensor parallel 1 (pure data parallel), stock Tinker SDK, 4 rows of 16,384 tokens per step with cross entropy, 4 steps. The paired runs differ only in logprob_chunk_tokens (unset vs 8192). The 64K row is one 65,535-token row per GPU.

fp32_lm_head with TF32: Before W&B · After W&B · 64K row W&B
fp32_lm_head off: Before W&B · After W&B · 64K row W&B

Before After
One 64K row, peak memory per GPU out of memory (155.8 GiB in use when it asked for 30.3 GiB more) 58.1 GiB (fp32 head) / 57.0 GiB
Step time, 4 × 16K rows (fp32 head) 1.34 s 1.46 s
Step time, 4 × 16K rows (bf16 head) 1.42 s 1.52 s
Per-token logprobs, step 0 mean |Δ| 2.0e-7, max 1.9e-6
Loss, steps 0–3 within 0.004%
Gradient norm, steps 0–3 within 0.25%

Rows that already fit get about 8% slower, because the LM-head matmul runs again in the backward pass. The setting is for recipes where it lets a long row fit without tensor or context parallelism.

🤖 Generated with Claude Code

Qwen3.5's 248K vocabulary makes full FP32 logits for a 64K-token
micro-batch 60 GiB, and cross entropy keeps a second copy for backward,
so one 64K row cannot train without tensor or context parallelism.
logprob_chunk_tokens computes the LM head and log-softmax a chunk at a
time and recomputes each chunk in backward. It requires tensor parallel
size 1. A 64K row now peaks at 58 GiB on one GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask the model for unconverted logits whenever the output layer is
deferred, so a bf16 hidden is never upcast before it meets the bf16
weight; cast the hidden to the weight dtype on the non-FP32 path. Refuse
the setting on the LoRA backend, with MuP logit scaling, or when no output
layer is found. Share one logits helper for the chunked and replay paths
and test the output-layer keyword contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) merged commit e000026 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/q4fft/b200-chunked-logprobs branch November 17, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant