Repository navigation
[trainer correctness] Restart Mamba state at each row packed into a micro-batch - #244
Merged
Micah Yong (micahtyong) merged 6 commits intoNov 17, 2026
Conversation
Megatron's Mamba mixer restarts its convolution and state at packed sequence boundaries only when PackedSeqParams carries seq_idx, which it derives from total_tokens. Miles does not set total_tokens, so every row after the first in a micro-batch was computed as a continuation of the row before it. Set total_tokens when the trainer runs without context parallelism, behind MAMBA_SEQUENCE_RESET, and turn it on for the Nemotron 3.5 Lightning recipes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Drop the MAMBA_SEQUENCE_RESET flag and set the sequence indices for every packed batch. At the pinned Megatron they are read during training by the Mamba mixer and by the fused gated-delta-net path, which is off by default and derives the same indices itself, so models without Mamba layers are unchanged. Under context parallelism the mixer gathers every rank's slice back into the whole micro-batch before its convolution and scan, so packed rows shared state there too. Each rank holds an equal slice, so the micro-batch length is the local length times the context-parallel size. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rainer-1-mamba-sequence-reset # Conflicts: # src/spindle/backends/miles_runtime/actor.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 17, 2026
Micah Yong (micahtyong)
marked this pull request as ready for review
November 17, 2026 02:17
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 02:17
Micah Yong (micahtyong)
deleted the
micah/nemotron-perf/trainer-1-mamba-sequence-reset
branch
November 17, 2026 02:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The trainer packs several short rows into one micro-batch so the GPU runs the model once for all of them. Attention layers keep packed rows apart with a mask. Mamba layers carry a running state along the sequence, and they need to be told where each row starts so the state restarts there. Megatron works that out from
total_tokensonPackedSeqParams. Miles does not set it, so the Mamba layers treat the whole micro-batch as one long sequence, and every row after the first starts from the previous row's state.The effect shows up in the per-token log probabilities. Compared with the same step on Tinker, only the first row of each micro-batch matches. The other rows are off by 0.4 to 0.85 on average, most at their first tokens.
Miles v0.2.0 sets
total_tokensitself, the same way, so themiles-v0.2.0image already gets this right. This PR adds a small wrapper aroundget_packed_seq_paramsthat does the same on the other Miles images. Like upstream, it is always on. Only Mamba layers read the field, so only models with Mamba layers change, which today means the Nemotron recipes. It also covers context parallelism, where each GPU holds a slice of the micro-batch and the Mamba layers gather the slices back to full length before they run.Training results on the Nemotron recipes change with this PR, because rows after the first in a micro-batch were computed from the wrong state before. A batch with one row per micro-batch is unchanged.
E2E results
Setup: one client, stock Tinker SDK, LoRA rank 32. Short rows are 128 rows of about 1,600 tokens per step. Long rows are 8 rows of about 16,000 tokens. The gap is the mean absolute difference in per-token log probability against the same step on Tinker. Two runs of one Spindle config differ by about 0.04, and a row counts as matching when its gap is at that level. Tinker does not serve Nemotron 3 Nano, so that row compares against the same rows trained one per micro-batch.
Gradients were checked separately at the Mamba layer in float32: with the reset, packed rows give the same outputs and gradients as the same rows run one at a time, and one row's loss has zero gradient with respect to another row's inputs. The Nemotron Super and Ultra recipes use the same code path and were not run.