Repository navigation
[trainer throughput] Pass max_seqlen as an int so FlashAttention 4 compiles its backward once - #334
Merged
Micah Yong (micahtyong) merged 3 commits intoNov 17, 2026
Conversation
Without context parallelism, Transformer Engine runs attention through FlashAttention 4 on Blackwell. Its backward pass caches compiled kernels under a key derived from max_seqlen. A tensor max_seqlen puts a tensor in that key, so the cache never hits and every backward call recompiles, about 3.4 s of host time per attention layer per micro-batch. Passing the packed length as a Python int lets the first compile be reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 04:21
This was referenced Nov 17, 2026
Merged
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
deleted the
micah/q4fft/b300-fa4-int-max-seqlen
branch
November 17, 2026 17:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a full fine-tuning recipe runs without context parallelism, Transformer Engine runs attention with FlashAttention 4. FlashAttention 4 compiles its backward kernel and caches it under a key built from the call's arguments, including one derived from
max_seqlen.Spindle passed
max_seqlenas a CUDA tensor. A tensor never compares equal to the cached key, so every backward call compiled the kernel again. That cost about 3.4 s of host time per attention layer on every step, while the GPU sat idle.This PR passes
max_seqlenas a plain Python int, which is the type Megatron and Transformer Engine declare for it. The backward now compiles once. The value is the same as before, and runs with context parallelism are unaffected. The forward-only and log-prob paths share this code, so they get the fix too.E2E results
Setup:
qwen35-4b-fft-64kon 4×B300 without tensor or context parallelism (pure data parallel, full recompute), stock Tinker SDK, four 65,535-token rows per step with cross entropy. Trainerforward_backwardtime per step, median after step 0. Loss is within 0.016% of the recipe's default layout.Before W&B · After W&B
forward_backward, 4 × 64K rows🤖 Generated with Claude Code