Repository navigation
[trainer throughput] Train Qwen3.5/3.6-35B-A3B FFT 64K with expert parallelism only, on B300 or B200 - #340
Merged
Micah Yong (micahtyong) merged 8 commits intoNov 17, 2026
Conversation
Qwen3.5's 248K vocabulary makes full FP32 logits for a 64K-token micro-batch 60 GiB, and cross entropy keeps a second copy for backward, so one 64K row cannot train without tensor or context parallelism. logprob_chunk_tokens computes the LM head and log-softmax a chunk at a time and recomputes each chunk in backward. It requires tensor parallel size 1. A 64K row now peaks at 58 GiB on one GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask the model for unconverted logits whenever the output layer is deferred, so a bf16 hidden is never upcast before it meets the bf16 weight; cast the hidden to the weight dtype on the non-FP32 path. Refuse the setting on the LoRA backend, with MuP logit scaling, or when no output layer is found. Share one logits helper for the chunked and replay paths and test the output-layer keyword contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…00 or B300 The recipes split each layer over 4 GPUs and each sequence over 2 on 8xH200, with experts spread over all 8. On Blackwell each GPU holds the dense weights plus its eighth of the experts, so the GPUs now train their own sequences and only the experts are split. Log-probs are computed in 8,192-token chunks of the LM head, and full recompute keeps a 65,535-token row at 128 GiB per GPU, which fits B200 as well. On 8xB200 or 8xB300 the trainer spends 4.8 s on a step of 64 x 4K tokens (25.0 s on 8xH200 before, 23.1 s with the old layout on 8xB300) and 8.7-9.1 s on 4 x 64K (12.2 s on 8xH200). B200 matches B300 here and costs less, so it comes first. qwen36-35b-a3b-fft-64k inherits the layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
marked this pull request as ready for review
November 17, 2026 17:39
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 17:39
Micah Yong (micahtyong)
deleted the
micah/q4fft/others-qwen35-35b-a3b-fft-64k
branch
November 17, 2026 17:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qwen35-35b-a3b-fft-64ktrained on eight H200s with tensor parallel 4, context parallel 2 and expert parallel 8. Most of this model's weights are experts, which expert parallelism already spreads over the eight GPUs. The tensor and context parallelism mostly added communication and small kernel launches.This PR keeps expert parallel 8 and drops tensor and context parallelism, so each GPU holds the shared layers whole and trains its own share of the batch. Chunked log-probs (#335) keep a 64K row's output layer in memory, and full recompute replaces selective recompute. That keeps peak memory at 128 GiB for cross entropy, close to the old layout. The recipe lists eight B300s first and eight B200s as a fallback. The replay losses (
mis,score_centered,score_centered_mis) still build full-vocabulary logits, so a 65,535-token replay row needs 185 GiB per GPU: it fits on B300 (W&B) but not on B200, where replay rows fit up to about 48K tokens (W&B).qwen36-35b-a3b-fft-64kinherits this layout.The step times below also need #333, #334 and #336, so this should merge after them. Without #334, attention recompiles its backward on every call and steps are slower than on the old layout. Training states saved on the old layout can't be resumed with
load_state_with_optimizer;load_statestill loads their weights.E2E results
Setup: stock Tinker SDK,
forward_backwardwith cross entropy plusoptim_stepon fixed random batches of ~256K tokens. 64 × 4K = 64 rows of ~4K tokens, 4 × 64K = four 65,535-token rows. Medians of steps 1–11. The Blackwell trainers ran outside us-east, so the table shows trainer time per step, which doesn't include the extra network hop. Loss stays within 0.02% of the H200 run.Before W&B · Old layout on B300 with the code fixes W&B · After W&B (B200) · After W&B (B300)
Four 64K rows on eight data-parallel ranks leave half the GPUs idle, so that shape gains less.
🤖 Generated with Claude Code