Skip to content

[trainer throughput] Train Qwen3.5/3.6-35B-A3B FFT 64K with expert parallelism only, on B300 or B200 - #340

Merged
Micah Yong (micahtyong) merged 8 commits into
mainfrom
micah/q4fft/others-qwen35-35b-a3b-fft-64k
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 8 commits into
mainfrom
micah/q4fft/others-qwen35-35b-a3b-fft-64k

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

qwen35-35b-a3b-fft-64k trained on eight H200s with tensor parallel 4, context parallel 2 and expert parallel 8. Most of this model's weights are experts, which expert parallelism already spreads over the eight GPUs. The tensor and context parallelism mostly added communication and small kernel launches.

This PR keeps expert parallel 8 and drops tensor and context parallelism, so each GPU holds the shared layers whole and trains its own share of the batch. Chunked log-probs (#335) keep a 64K row's output layer in memory, and full recompute replaces selective recompute. That keeps peak memory at 128 GiB for cross entropy, close to the old layout. The recipe lists eight B300s first and eight B200s as a fallback. The replay losses (mis, score_centered, score_centered_mis) still build full-vocabulary logits, so a 65,535-token replay row needs 185 GiB per GPU: it fits on B300 (W&B) but not on B200, where replay rows fit up to about 48K tokens (W&B). qwen36-35b-a3b-fft-64k inherits this layout.

The step times below also need #333, #334 and #336, so this should merge after them. Without #334, attention recompiles its backward on every call and steps are slower than on the old layout. Training states saved on the old layout can't be resumed with load_state_with_optimizer; load_state still loads their weights.

E2E results

Setup: stock Tinker SDK, forward_backward with cross entropy plus optim_step on fixed random batches of ~256K tokens. 64 × 4K = 64 rows of ~4K tokens, 4 × 64K = four 65,535-token rows. Medians of steps 1–11. The Blackwell trainers ran outside us-east, so the table shows trainer time per step, which doesn't include the extra network hop. Loss stays within 0.02% of the H200 run.

Before W&B · Old layout on B300 with the code fixes W&B · After W&B (B200) · After W&B (B300)

Before (8×H200) Old layout on 8×B300, with #333–#336 After (8×B200) After (8×B300)
Trainer step, 64 × 4K rows 25.0 s 11.7 s 4.9 s 4.8 s
Trainer step, 4 × 64K rows 12.2 s 8.8 s 9.1 s 8.7 s
Peak memory per GPU 123 GiB 123 GiB 128 GiB 128 GiB
Trainer cost per step, 64 × 4K rows $0.252 $0.185 $0.068 $0.076

Four 64K rows on eight data-parallel ranks leave half the GPUs idle, so that shape gains less.

🤖 Generated with Claude Code

Qwen3.5's 248K vocabulary makes full FP32 logits for a 64K-token
micro-batch 60 GiB, and cross entropy keeps a second copy for backward,
so one 64K row cannot train without tensor or context parallelism.
logprob_chunk_tokens computes the LM head and log-softmax a chunk at a
time and recomputes each chunk in backward. It requires tensor parallel
size 1. A 64K row now peaks at 58 GiB on one GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask the model for unconverted logits whenever the output layer is
deferred, so a bf16 hidden is never upcast before it meets the bf16
weight; cast the hidden to the weight dtype on the non-FP32 path. Refuse
the setting on the LoRA backend, with MuP logit scaling, or when no output
layer is found. Share one logits helper for the chunked and replay paths
and test the output-layer keyword contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…00 or B300

The recipes split each layer over 4 GPUs and each sequence over 2 on
8xH200, with experts spread over all 8. On Blackwell each GPU holds the
dense weights plus its eighth of the experts, so the GPUs now train
their own sequences and only the experts are split. Log-probs are
computed in 8,192-token chunks of the LM head, and full recompute keeps
a 65,535-token row at 128 GiB per GPU, which fits B200 as well.

On 8xB200 or 8xB300 the trainer spends 4.8 s on a step of 64 x 4K
tokens (25.0 s on 8xH200 before, 23.1 s with the old layout on 8xB300)
and 8.7-9.1 s on 4 x 64K (12.2 s on 8xH200). B200 matches B300 here and
costs less, so it comes first. qwen36-35b-a3b-fft-64k inherits the
layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong Micah Yong (micahtyong) changed the title [trainer throughput] Train Qwen3.5/3.6-35B-A3B FFT 64K with expert parallelism only, on B200 or B300 [trainer throughput] Train Qwen3.5/3.6-35B-A3B FFT 64K with expert parallelism only, on B300 or B200 Nov 17, 2026
Base automatically changed from micah/q4fft/b200-chunked-logprobs to main November 17, 2026 17:33
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 17:39
@micahtyong
Micah Yong (micahtyong) merged commit 0073999 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/q4fft/others-qwen35-35b-a3b-fft-64k branch November 17, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant