Repository navigation
[trainer throughput] Train Qwen3.5-4B FFT 64K data-parallel on four B300s or B200s - #337
Merged
Merged
Conversation
Qwen3.5's 248K vocabulary makes full FP32 logits for a 64K-token micro-batch 60 GiB, and cross entropy keeps a second copy for backward, so one 64K row cannot train without tensor or context parallelism. logprob_chunk_tokens computes the LM head and log-softmax a chunk at a time and recomputes each chunk in backward. It requires tensor parallel size 1. A 64K row now peaks at 58 GiB on one GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask the model for unconverted logits whenever the output layer is deferred, so a bf16 hidden is never upcast before it meets the bf16 weight; cast the hidden to the weight dtype on the non-FP32 path. Refuse the setting on the LoRA backend, with MuP logit scaling, or when no output layer is found. Share one logits helper for the chunked and replay paths and test the output-layer keyword contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the recipe from four H100s with TP2 + CP2 to four B200s with pure data parallelism. Chunked LM-head log-probs keep a 65,535-token row at 58 GiB per GPU without tensor or context parallelism, TF32 runs the FP32 LM head on tensor cores with the same products, and expandable segments avoid fragmentation on long-lived trainers. The sampler moves to B200 too. The 9B child keeps its current H200 TP2 + CP2 layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…00 layout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
marked this pull request as ready for review
November 17, 2026 17:38
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 17:38
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qwen35-4b-fft-64ktrains on eight B200s with tensor parallel 2 and context parallel 4, so all eight GPUs share one copy of the model. That split was there to fit a 64K-token row in memory. It makes every step pay for cross-GPU traffic and many small kernel launches, and it needs an RDMA allocation, which can take a while to schedule.This PR runs the trainer on four B300s, or four B200s when no B300 is free, with no tensor or context parallelism. Each GPU holds the whole model and trains its own share of the batch. Three settings make that fit and run fast:
logprob_chunk_tokens: 8192scores the output layer in chunks, so a 64K row needs 58 GiB instead of running out of memory ([trainer memory] Score full fine-tuning log-probs in token chunks #335).TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1runs the fp32 output layer on tensor cores. Its forward inputs are bf16 values, which TF32 holds exactly. The backward rounds gradients to TF32, which is still finer than the bf16 they are stored in. Without it, that one matmul took about 3.7 s per step.PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truestays on, so memory doesn't fragment.The sampler is unchanged.
qwen35-9b-fft-64kinherits from this recipe and keeps its own H200 layout.Training states saved on the old layout can't be resumed with
load_state_with_optimizer, because the optimizer state is sharded for the old layout.load_statestill loads their weights.The step times below rely on #333, #334 and #336, which are on main.
E2E results
Setup: stock Tinker SDK,
forward_backwardwith cross entropy plusoptim_stepon fixed random batches of ~256K tokens. 64 × 4K = 64 rows of ~4K tokens, 8 × 32K = 8 rows of ~32K tokens, 4 × 64K = four 65,535-token rows. Medians after step 0. Cost is trainer GPUs × list price × step time. Loss stays within 0.014% between the old layout with and without the code fixes, and within 0.06% between the old and new layouts.The old layout's 8-GPU RDMA request queued for 24 minutes in us-east, so those runs ran in eu-north and include about 1.5–2.5 s per step of extra network time. The new layout ran in us-east.
Old layout: as is W&B (32K, 64K) · with #333–#336 W&B (32K, 64K) · New layout: B300 W&B (32K, 64K) · B200 W&B (32K, 64K)
As is, the old layout's steps swing between 8 and 145 s while kernels recompile, and its first step takes about 8 minutes. On trainer time alone, which leaves out the network difference, the old layout with the fixes takes 7.5 s on 64 × 4K rows and this PR takes 4.3 s on half the GPUs.
🤖 Generated with Claude Code