Skip to content

[trainer throughput] Fuse Nemotron's squared-ReLU expert activation - #320

Closed
Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/nemotron-perf/gap-2-fused-squared-relu
Closed

Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/nemotron-perf/gap-2-fused-squared-relu

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

Nemotron-3.5-Lightning's experts use a squared-ReLU activation, multiplied by the router's probability for each token. Megatron runs this as several separate elementwise kernels, with a round trip through fp32, and takes 0.86 s of a training step on short rows. Megatron already has a fused kernel for exactly this, use_fused_weighted_squared_relu, but Miles does not pass the setting through.

This PR turns it on from Spindle when FUSED_WEIGHTED_SQUARED_RELU=1, which the 64k Lightning recipe sets. The 256k recipe sets it off because it hasn't been measured there.

E2E results

Setup: nemotron35-lightning-30b-a3b-lora-64k with the earlier PRs in this stack, 2x H200, stock Tinker SDK, LoRA rank 32. Times are the trainer's forward_backward median. Per-token log probabilities are within run-to-run noise.

Batch Before WandB After WandB forward_backward (s)
128 rows of 1,600 tokens Before After 8.99 → 8.46
8 rows of 16,000 tokens Before After 5.40 → 4.92

The activation itself goes from 0.86 to 0.03 s per step.

@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/gap-2-fused-squared-relu branch from ab2ebc6 to f52fcfd Compare November 17, 2026 02:22
@micahtyong
Micah Yong (micahtyong) changed the base branch from micah/nemotron-perf/gap-1-expert-grouped-gemm to main November 17, 2026 02:46
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/nemotron-perf/gap-2-fused-squared-relu branch 2 times, most recently from 0a4960d to 5676d17 Compare November 17, 2026 03:07
Each Nemotron expert activation ran as separate kernels: the squared ReLU,
a multiply by the router probability that promoted the activation to fp32,
and a cast back to bf16, with the same pattern in backward. Megatron has one
fused kernel for this behind use_fused_weighted_squared_relu, but Miles does
not pass that setting through, so Spindle sets it on the model provider when
a recipe asks with FUSED_WEIGHTED_SQUARED_RELU=1. The Nemotron 3.5 Lightning
64K recipe does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong

Copy link
Copy Markdown
Collaborator Author

Closing. This turns on Megatron's fused squared-ReLU expert activation by patching the Miles model provider, which upstream Miles doesn't do; the measured gain was about 5% of step time on short rows and not clearly positive on long rows. If it's worth having, the cleaner route is a small upstream Miles change that passes use_fused_weighted_squared_relu through to the Megatron config, picked up with a Miles pin bump.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant