Skip to content

[trainer throughput] Train Nemotron-3.5-Lightning 64K on two B300s - #332

Merged
Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/lightning-64k-b300
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/lightning-64k-b300

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

nemotron35-lightning-30b-a3b-lora-64k trained on two H200s with each layer split across both GPUs and every layer's activations recomputed in the backward pass, because one H200 can't hold the model and a 64K-token row otherwise. A B300 has 288 GB, which is enough for each GPU to hold the whole model and train its own rows, and for the last 16 of 52 layers to keep their activations instead of recomputing them.

This PR moves the recipe to that layout on two B300s: one model copy per GPU, 65,536 tokens per micro-batch, and recompute for the first 36 layers only. Because no expert or layer is split across GPUs, the gradient reduce leaves kept gradients unchanged, so the gradient copy from #257 does not run.

The previous H200 layout moves to a new recipe, nemotron35-lightning-30b-a3b-lora-64k-h200, for when B300s are not available. It splits the experts across both GPUs again, which also avoids #257's copy; the layout on main today (all experts on both GPUs) needs a 12.85 GB copy per GPU on every call. The 256K recipe and the shared-server recipe now inherit from the H200 recipe, so their trainers don't change. The sampler settings are the same for both.

This replaces #321 and #331.

E2E results

Setup: stock Tinker SDK, LoRA rank 32, trainer, frontend and client in us-east. The B300 runs used this layout with #317–#320. Per-token log probabilities match Tinker's to the same level as the H200 recipe (0.036–0.042).

Batch H200 WandB B300 WandB Step (s) forward_backward (s) Tinker step (s)
128 rows of 1,600 tokens H200 B300 8.25 → 4.79 7.61 → 3.50 3.20
8 rows of 16,000 tokens H200 B300 5.32 → 2.80 4.74 → 2.26 2.90

Four clients each training a 64K-token row peak at 234 GB allocated of 288 GB (run).

…ecipe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 03:36
@micahtyong
Micah Yong (micahtyong) merged commit adf4578 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/nemotron-perf/lightning-64k-b300 branch November 17, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant