Skip to content

[trainer throughput] Keep all experts on every GPU for Nemotron-3.5-Lightning 64K - #246

Merged
Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/trainer-3-expert-parallel-1
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/trainer-3-expert-parallel-1

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Merge after #257. With all experts on every GPU, each later forward_backward before an optim_step doubles the expert gradients already waiting (four identical calls give 15 times the single-call gradient, not 4). #257 fixes that for every layout; with it, the same test gives 2.01 and 4.00 on this layout. One call per optim_step is correct either way, which is what the runs below use.

Summary

Nemotron-3.5-Lightning is a mixture-of-experts model with 128 experts per layer. The recipe split them across its two GPUs with expert_model_parallel_size 2, so each GPU held 64. In every layer each GPU then has to send tokens to the GPU that holds their expert and get the results back. A profile of one training step shows those exchanges, plus the gathers that go with them, take about 17% of GPU kernel time.

The split is not needed for memory on H200. This PR sets expert_model_parallel_size to 1 in nemotron35_lightning_30b_a3b_lora_64k, so every GPU holds all 128 experts and no tokens move between GPUs. The 256k recipe sets its own value and is unchanged.

Memory goes up. On the full stack, four clients each training a 64K-token row peak at 130 GB allocated on a 140 GB H200.

E2E results

Setup: 2x H200, one client, stock Tinker SDK, LoRA rank 32. Short rows are 128 rows of about 1,600 tokens per step, long rows are 8 rows of about 16,000 tokens, and 64K rows are 2 rows of 64K tokens. Per-token log probabilities match the previous PR to within run-to-run noise.

Batch Before WandB After WandB forward_backward (s)
Short rows Before After 12.41 → 9.06
Long rows Before After 7.01 → 6.79
64K rows Before After 8.32 → 6.54

@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/trainer-3-expert-parallel-1 branch from eff6b35 to 91861f5 Compare November 17, 2026 02:22
Base automatically changed from micah/nemotron-perf/trainer-2-log-probs-chunk to main November 17, 2026 02:33
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/trainer-3-expert-parallel-1 branch from 266c2f3 to edc6695 Compare November 17, 2026 02:33
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 02:42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/trainer-3-expert-parallel-1 branch from b4320f2 to 8a1d78a Compare November 17, 2026 02:58
@micahtyong
Micah Yong (micahtyong) merged commit 36cae7e into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/nemotron-perf/trainer-3-expert-parallel-1 branch November 17, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant