Skip to content

[trainer throughput] Give the MoE token permute a faster backward - #318

Merged
Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/moe-permute-scatter-backward
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/moe-permute-scatter-backward

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

In a mixture-of-experts layer, the trainer reorders tokens so each expert's tokens sit together. With permute fusion off, which LoRA on expert weights requires, Megatron does this with index_select. Its backward pass is a BF16 index_add_, which is slow: 2.5–3.1 ms per call on H200.

This PR gives the permute a scatter_add_ backward, which computes the same gradient in about 0.6 ms. It applies on the default Miles images when the pinned Megatron permute matches a known source hash and nothing else has wrapped it. Otherwise it leaves Megatron alone and logs that it skipped. The miles-v0.2.0 image is unaffected.

E2E results

Setup: stock Tinker SDK, LoRA rank 32, 128 rows of about 1,600 tokens per step. Times are the trainer's forward_backward median. Per-token log probabilities are within run-to-run noise.

Recipe Before WandB After WandB forward_backward (s)
nemotron35-lightning-30b-a3b-lora-64k, 2x H200, with PRs #244–#247 and #317 Before After 7.70 → 7.37

On Nemotron the time in token reordering goes from 0.69 to 0.41 s per step. gpt-oss-20b on two H200s also gets faster: 9.35 → 8.61 s together with the logits change in #317 (before, after).

Megatron's unfused MoE permute gathers tokens with index_select, whose
backward is a BF16 index_add_. A scatter_add_ backward computes the same
gradient faster. The patch applies only when the pinned Megatron permute's
source hash matches and nothing has wrapped it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	src/spindle/backends/miles_runtime/actor.py
@micahtyong
Micah Yong (micahtyong) merged commit 50798ce into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/moe-permute-scatter-backward branch November 17, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant