Skip to content

[sampler throughput] Sample Nemotron-3.5-Lightning with its native MTP layer - #323

Merged
Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/sampler-6-mtp
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/nemotron-perf/sampler-6-mtp

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Nemotron-3.5-Lightning ships one multi-token prediction (MTP) layer. SGLang can use it as a draft (NEXTN): the MTP layer proposes the next few tokens and the main model checks them all in one step. On H200 a decode step is limited by reading the expert weights from memory, so checking four tokens costs about as much as generating one, and every accepted token is nearly free. On these evals the sampler accepts 2.4 tokens per step on IFEval and 3.4 on MATH.

This PR turns MTP on in nemotron35-lightning-30b-a3b-lora-64k, the same way qwen35-4b-lora-32k uses its MTP layer. It works with LoRA adapters loaded; the draft runs without the adapter and the main model checks with it. MTP keeps extra Mamba state per request, so the Mamba cache drops to 448 slots and the sampler runs 89 requests at a time instead of 128. inference_target_concurrency moves to 89 to match, so a second replica is added when the first is full. The 256k recipe inherits it.

The sampler's Mamba state stays in fp32. Sampled log probabilities match the trainer's as closely as without MTP: mean difference 0.012–0.013 per token with or without MTP, base and adapter (without MTP, with MTP).

E2E results

Setup: the stock recipe from this branch, one warm H200 sampler replica, stock Tinker SDK. IFEval is 541 prompts, greedy, 1,280 max tokens. MATH-500 x4 is 500 problems with 4 samples each, temperature 0.6, 2,048 max tokens. The adapter is rank 32. Eval scores are unchanged and no run had a client error. Times before are from the previous PR in this stack.

Eval Before WandB After WandB Wall time (s)
MATH-500 x4, every prompt at once Before After 336–358 → 280–283
MATH-500 x4 with a rank-32 adapter Before After 392 → 248
IFEval, every prompt at once Before After 48.3 → 44.6
IFEval, 64 requests in flight Before After 52.5 → 39.6
IFEval, one request at a time (16 prompts) – After 48.9 → 31.0

On one stream the sampler generates about 229 tokens per second, with or without an adapter.

@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/sampler-6-mtp branch from 51a1e95 to 0df5025 Compare November 17, 2026 02:27
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 02:29
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/sampler-6-mtp branch from a6a5c74 to bab36d9 Compare November 17, 2026 02:39
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/sampler-6-mtp branch from bab36d9 to 7808de9 Compare November 17, 2026 02:43
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/sampler-6-mtp branch from 7808de9 to bd72335 Compare November 17, 2026 02:45
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/sampler-6-mtp branch from bd72335 to 8674a17 Compare November 17, 2026 02:46
Base automatically changed from micah/nemotron-perf/sampler-5-lora-virtual-experts to main November 17, 2026 03:29
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/sampler-6-mtp branch from 8674a17 to 6ee109e Compare November 17, 2026 03:29
@micahtyong
Micah Yong (micahtyong) merged commit 6c59ca1 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/nemotron-perf/sampler-6-mtp branch November 17, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant