Repository navigation
[trainer throughput] Train Nemotron-3.5-Lightning 64K on two B300s - #332
Merged
Micah Yong (micahtyong) merged 1 commit intoNov 17, 2026
Merged
Conversation
…ecipe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 17, 2026
Micah Yong (micahtyong)
marked this pull request as ready for review
November 17, 2026 03:36
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 03:36
Micah Yong (micahtyong)
deleted the
micah/nemotron-perf/lightning-64k-b300
branch
November 17, 2026 03:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nemotron35-lightning-30b-a3b-lora-64ktrained on two H200s with each layer split across both GPUs and every layer's activations recomputed in the backward pass, because one H200 can't hold the model and a 64K-token row otherwise. A B300 has 288 GB, which is enough for each GPU to hold the whole model and train its own rows, and for the last 16 of 52 layers to keep their activations instead of recomputing them.This PR moves the recipe to that layout on two B300s: one model copy per GPU, 65,536 tokens per micro-batch, and recompute for the first 36 layers only. Because no expert or layer is split across GPUs, the gradient reduce leaves kept gradients unchanged, so the gradient copy from #257 does not run.
The previous H200 layout moves to a new recipe,
nemotron35-lightning-30b-a3b-lora-64k-h200, for when B300s are not available. It splits the experts across both GPUs again, which also avoids #257's copy; the layout on main today (all experts on both GPUs) needs a 12.85 GB copy per GPU on every call. The 256K recipe and the shared-server recipe now inherit from the H200 recipe, so their trainers don't change. The sampler settings are the same for both.This replaces #321 and #331.
E2E results
Setup: stock Tinker SDK, LoRA rank 32, trainer, frontend and client in us-east. The B300 runs used this layout with #317–#320. Per-token log probabilities match Tinker's to the same level as the H200 recipe (0.036–0.042).
Four clients each training a 64K-token row peak at 234 GB allocated of 288 GB (run).