Skip to content

[trainer throughput] Pass max_seqlen as an int so FlashAttention 4 compiles its backward once - #334

Merged
Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/q4fft/b300-fa4-int-max-seqlen
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/q4fft/b300-fa4-int-max-seqlen

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

When a full fine-tuning recipe runs without context parallelism, Transformer Engine runs attention with FlashAttention 4. FlashAttention 4 compiles its backward kernel and caches it under a key built from the call's arguments, including one derived from max_seqlen.

Spindle passed max_seqlen as a CUDA tensor. A tensor never compares equal to the cached key, so every backward call compiled the kernel again. That cost about 3.4 s of host time per attention layer on every step, while the GPU sat idle.

This PR passes max_seqlen as a plain Python int, which is the type Megatron and Transformer Engine declare for it. The backward now compiles once. The value is the same as before, and runs with context parallelism are unaffected. The forward-only and log-prob paths share this code, so they get the fix too.

E2E results

Setup: qwen35-4b-fft-64k on 4×B300 without tensor or context parallelism (pure data parallel, full recompute), stock Tinker SDK, four 65,535-token rows per step with cross entropy. Trainer forward_backward time per step, median after step 0. Loss is within 0.016% of the recipe's default layout.

Before W&B · After W&B

Before After
forward_backward, 4 × 64K rows 31.7 s 4.4 s
Host time per attention backward call 3.3–3.5 s about 0.06 s

🤖 Generated with Claude Code

Without context parallelism, Transformer Engine runs attention through
FlashAttention 4 on Blackwell. Its backward pass caches compiled kernels under
a key derived from max_seqlen. A tensor max_seqlen puts a tensor in that key,
so the cache never hits and every backward call recompiles, about 3.4 s of host
time per attention layer per micro-batch. Passing the packed length as a Python
int lets the first compile be reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) merged commit 95de4b6 into main Nov 17, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/q4fft/b300-fa4-int-max-seqlen branch November 17, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant