Repository navigation
[sampler throughput] Sample Nemotron-3.5-Lightning with its native MTP layer - #323
Merged
Merged
Conversation
Micah Yong (micahtyong)
force-pushed
the
micah/nemotron-perf/sampler-6-mtp
branch
from
November 17, 2026 02:27
51a1e95 to
0df5025
Compare
Micah Yong (micahtyong)
marked this pull request as ready for review
November 17, 2026 02:29
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 02:29
Micah Yong (micahtyong)
force-pushed
the
micah/nemotron-perf/sampler-6-mtp
branch
from
November 17, 2026 02:39
a6a5c74 to
bab36d9
Compare
Micah Yong (micahtyong)
force-pushed
the
micah/nemotron-perf/sampler-6-mtp
branch
from
November 17, 2026 02:43
bab36d9 to
7808de9
Compare
Micah Yong (micahtyong)
force-pushed
the
micah/nemotron-perf/sampler-6-mtp
branch
from
November 17, 2026 02:45
7808de9 to
bd72335
Compare
Micah Yong (micahtyong)
force-pushed
the
micah/nemotron-perf/sampler-6-mtp
branch
from
November 17, 2026 02:46
bd72335 to
8674a17
Compare
Base automatically changed from
micah/nemotron-perf/sampler-5-lora-virtual-experts
to
main
November 17, 2026 03:29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
force-pushed
the
micah/nemotron-perf/sampler-6-mtp
branch
from
November 17, 2026 03:29
8674a17 to
6ee109e
Compare
Micah Yong (micahtyong)
deleted the
micah/nemotron-perf/sampler-6-mtp
branch
November 17, 2026 03:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nemotron-3.5-Lightning ships one multi-token prediction (MTP) layer. SGLang can use it as a draft (
NEXTN): the MTP layer proposes the next few tokens and the main model checks them all in one step. On H200 a decode step is limited by reading the expert weights from memory, so checking four tokens costs about as much as generating one, and every accepted token is nearly free. On these evals the sampler accepts 2.4 tokens per step on IFEval and 3.4 on MATH.This PR turns MTP on in
nemotron35-lightning-30b-a3b-lora-64k, the same wayqwen35-4b-lora-32kuses its MTP layer. It works with LoRA adapters loaded; the draft runs without the adapter and the main model checks with it. MTP keeps extra Mamba state per request, so the Mamba cache drops to 448 slots and the sampler runs 89 requests at a time instead of 128.inference_target_concurrencymoves to 89 to match, so a second replica is added when the first is full. The 256k recipe inherits it.The sampler's Mamba state stays in fp32. Sampled log probabilities match the trainer's as closely as without MTP: mean difference 0.012–0.013 per token with or without MTP, base and adapter (without MTP, with MTP).
E2E results
Setup: the stock recipe from this branch, one warm H200 sampler replica, stock Tinker SDK. IFEval is 541 prompts, greedy, 1,280 max tokens. MATH-500 x4 is 500 problems with 4 samples each, temperature 0.6, 2,048 max tokens. The adapter is rank 32. Eval scores are unchanged and no run had a client error. Times before are from the previous PR in this stack.
On one stream the sampler generates about 229 tokens per second, with or without an adapter.