Skip to content

[trainer correctness] Restart Mamba state at each row packed into a micro-batch - #244

Merged
Micah Yong (micahtyong) merged 6 commits into
mainfrom
micah/nemotron-perf/trainer-1-mamba-sequence-reset
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 6 commits into
mainfrom
micah/nemotron-perf/trainer-1-mamba-sequence-reset

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The trainer packs several short rows into one micro-batch so the GPU runs the model once for all of them. Attention layers keep packed rows apart with a mask. Mamba layers carry a running state along the sequence, and they need to be told where each row starts so the state restarts there. Megatron works that out from total_tokens on PackedSeqParams. Miles does not set it, so the Mamba layers treat the whole micro-batch as one long sequence, and every row after the first starts from the previous row's state.

The effect shows up in the per-token log probabilities. Compared with the same step on Tinker, only the first row of each micro-batch matches. The other rows are off by 0.4 to 0.85 on average, most at their first tokens.

Miles v0.2.0 sets total_tokens itself, the same way, so the miles-v0.2.0 image already gets this right. This PR adds a small wrapper around get_packed_seq_params that does the same on the other Miles images. Like upstream, it is always on. Only Mamba layers read the field, so only models with Mamba layers change, which today means the Nemotron recipes. It also covers context parallelism, where each GPU holds a slice of the micro-batch and the Mamba layers gather the slices back to full length before they run.

Training results on the Nemotron recipes change with this PR, because rows after the first in a micro-batch were computed from the wrong state before. A batch with one row per micro-batch is unchanged.

E2E results

Setup: one client, stock Tinker SDK, LoRA rank 32. Short rows are 128 rows of about 1,600 tokens per step. Long rows are 8 rows of about 16,000 tokens. The gap is the mean absolute difference in per-token log probability against the same step on Tinker. Two runs of one Spindle config differ by about 0.04, and a row counts as matching when its gap is at that level. Tinker does not serve Nemotron 3 Nano, so that row compares against the same rows trained one per micro-batch.

Recipe Batch Before WandB After WandB Rows matching Gap forward_backward (s)
Lightning 64k, 2x H200 Short rows Before After 4 → 128 of 128 0.690 → 0.044 12.25 → 12.32
Lightning 256k, 8x H200, context parallel 4 Short rows Before After 1 → 128 of 128 0.821 → 0.043 6.74 → 6.56
Lightning 256k, 8x H200, context parallel 4 Long rows Before After 1 → 8 of 8 0.452 → 0.036 2.83 → 2.90
Lightning 256k, 8x H200, context parallel 4 One 131K-token row Before After 1 → 1 of 1 0.036 → 0.036 3.59 → 3.59
Nano 64k, 2x H200 Short rows Before After 4 → 128 of 128 0.996 → 0.068 12.10 → 12.23

Gradients were checked separately at the Mamba layer in float32: with the reset, packed rows give the same outputs and gradients as the same rows run one at a time, and one row's loss has zero gradient with respect to another row's inputs. The Nemotron Super and Ultra recipes use the same code path and were not run.

Megatron's Mamba mixer restarts its convolution and state at packed sequence
boundaries only when PackedSeqParams carries seq_idx, which it derives from
total_tokens. Miles does not set total_tokens, so every row after the first in
a micro-batch was computed as a continuation of the row before it.

Set total_tokens when the trainer runs without context parallelism, behind
MAMBA_SEQUENCE_RESET, and turn it on for the Nemotron 3.5 Lightning recipes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Drop the MAMBA_SEQUENCE_RESET flag and set the sequence indices for every
packed batch. At the pinned Megatron they are read during training by the
Mamba mixer and by the fused gated-delta-net path, which is off by default
and derives the same indices itself, so models without Mamba layers are
unchanged.

Under context parallelism the mixer gathers every rank's slice back into the
whole micro-batch before its convolution and scan, so packed rows shared
state there too. Each rank holds an equal slice, so the micro-batch length is
the local length times the context-parallel size.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rainer-1-mamba-sequence-reset

# Conflicts:
#	src/spindle/backends/miles_runtime/actor.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) marked this pull request as ready for review November 17, 2026 02:17
@micahtyong
Micah Yong (micahtyong) merged commit 48b1972 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/nemotron-perf/trainer-1-mamba-sequence-reset branch November 17, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant