Skip to content

[trainer throughput] Train Qwen3.5-4B FFT 64K data-parallel on four B300s or B200s - #337

Merged
Micah Yong (micahtyong) merged 10 commits into
mainfrom
micah/q4fft/b200-recipe-dp4
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 10 commits into
mainfrom
micah/q4fft/b200-recipe-dp4

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

qwen35-4b-fft-64k trains on eight B200s with tensor parallel 2 and context parallel 4, so all eight GPUs share one copy of the model. That split was there to fit a 64K-token row in memory. It makes every step pay for cross-GPU traffic and many small kernel launches, and it needs an RDMA allocation, which can take a while to schedule.

This PR runs the trainer on four B300s, or four B200s when no B300 is free, with no tensor or context parallelism. Each GPU holds the whole model and trains its own share of the batch. Three settings make that fit and run fast:

  • logprob_chunk_tokens: 8192 scores the output layer in chunks, so a 64K row needs 58 GiB instead of running out of memory ([trainer memory] Score full fine-tuning log-probs in token chunks #335).
  • TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 runs the fp32 output layer on tensor cores. Its forward inputs are bf16 values, which TF32 holds exactly. The backward rounds gradients to TF32, which is still finer than the bf16 they are stored in. Without it, that one matmul took about 3.7 s per step.
  • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True stays on, so memory doesn't fragment.

The sampler is unchanged. qwen35-9b-fft-64k inherits from this recipe and keeps its own H200 layout.

Training states saved on the old layout can't be resumed with load_state_with_optimizer, because the optimizer state is sharded for the old layout. load_state still loads their weights.

The step times below rely on #333, #334 and #336, which are on main.

E2E results

Setup: stock Tinker SDK, forward_backward with cross entropy plus optim_step on fixed random batches of ~256K tokens. 64 × 4K = 64 rows of ~4K tokens, 8 × 32K = 8 rows of ~32K tokens, 4 × 64K = four 65,535-token rows. Medians after step 0. Cost is trainer GPUs × list price × step time. Loss stays within 0.014% between the old layout with and without the code fixes, and within 0.06% between the old and new layouts.

The old layout's 8-GPU RDMA request queued for 24 minutes in us-east, so those runs ran in eu-north and include about 1.5–2.5 s per step of extra network time. The new layout ran in us-east.

Old layout: as is W&B (32K, 64K) · with #333–#336 W&B (32K, 64K) · New layout: B300 W&B (32K, 64K) · B200 W&B (32K, 64K)

Old layout, 8×B200 Old layout + #333–#336 This PR, 4×B300 This PR, 4×B200
Step, 64 × 4K rows 26.7 s 9.0 s 4.4 s 4.8 s
Step, 8 × 32K rows 25.9 s 7.9 s 4.5 s 4.7 s
Step, 4 × 64K rows 9.1 s 9.8 s 5.0 s 5.1 s
Peak memory per GPU 58 GiB 58 GiB
Cost per step, 64 × 4K rows $0.371 $0.125 $0.035 $0.033

As is, the old layout's steps swing between 8 and 145 s while kernels recompile, and its first step takes about 8 minutes. On trainer time alone, which leaves out the network difference, the old layout with the fixes takes 7.5 s on 64 × 4K rows and this PR takes 4.3 s on half the GPUs.

🤖 Generated with Claude Code

Qwen3.5's 248K vocabulary makes full FP32 logits for a 64K-token
micro-batch 60 GiB, and cross entropy keeps a second copy for backward,
so one 64K row cannot train without tensor or context parallelism.
logprob_chunk_tokens computes the LM head and log-softmax a chunk at a
time and recomputes each chunk in backward. It requires tensor parallel
size 1. A 64K row now peaks at 58 GiB on one GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask the model for unconverted logits whenever the output layer is
deferred, so a bf16 hidden is never upcast before it meets the bf16
weight; cast the hidden to the weight dtype on the non-FP32 path. Refuse
the setting on the LoRA backend, with MuP logit scaling, or when no output
layer is found. Share one logits helper for the chunked and replay paths
and test the output-layer keyword contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the recipe from four H100s with TP2 + CP2 to four B200s with pure
data parallelism. Chunked LM-head log-probs keep a 65,535-token row at
58 GiB per GPU without tensor or context parallelism, TF32 runs the FP32
LM head on tensor cores with the same products, and expandable segments
avoid fragmentation on long-lived trainers. The sampler moves to B200 too.

The 9B child keeps its current H200 TP2 + CP2 layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from micah/q4fft/b200-chunked-logprobs to main November 17, 2026 17:33
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 17:38
@micahtyong
Micah Yong (micahtyong) merged commit 70ab297 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/q4fft/b200-recipe-dp4 branch November 17, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant