Repository navigation
feat(megatron): add --te-disable-flash-attn-4 so TE can fall back to FlashAttention 2 - #3662
Open
EvilSwordman wants to merge 3 commits into
Open
EvilSwordman wants to merge 3 commits into
EvilSwordman wants to merge 3 commits into
Conversation
…FlashAttention 2 Transformer Engine prefers FlashAttention 4 whenever it is installed. On sm_120 with head_dim 256 (the full-attention layers of Qwen3.5/3.6/3.8), FA4 fails to compile and cuDNN has no fused kernel for the shape, so TE is left with the unfused kernel, whose memory grows with the square of the sequence length. The flash_attn 2 in the image handles this shape, but TE has no switch for FA4 alone (NVTE_FLASH_ATTN=0 disables every version). The new flag flips TE's own FA4 detection in every training process, from MegatronTrainRayActor.init, before any model is built.
EvilSwordman
requested review from
Shi-Dong,
Zhichenzzz,
fzyzcjy,
guapisolo,
jybsuper,
maocheng23,
yueming-yuan and
yushengsu-thu
as code owners
September 24, 2026 08:26
Keeps the change clear of the import block that main reworked, so the branch merges cleanly; actor.py already imports this way in several methods.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Transformer Engine prefers FlashAttention 4 whenever
flash-attn-4is installed. The miles image ships it next toflash_attn2.7.4.post1, with TE 2.17.This breaks one hardware case: sm_120 GPUs (the RTX PRO 6000 / RTX 6000D class) with head_dim 256, which is what the full-attention layers of Qwen3.5/3.6/3.8 use. There, FA4 fails to compile (
MLIRError: Operation creation failed), and cuDNN has no fused kernel for the shape.--attention-backend flash, training crashes on the first forward.The installed
flash_attn2.7.4 is built for sm_120 and handles head_dim 256 in both forward and backward. The trouble is that TE has no switch for FA4 alone:NVTE_FLASH_ATTN=0disables every FlashAttention version.Change
--te-disable-flash-attn-4, off by default.maybe_hide_te_flash_attn_4(args)inmegatron_utils/misc_utils.py. It sets TE's ownFlashAttentionUtils.v4_is_installed = False, so TE's backend selection picks FlashAttention 2.MegatronTrainRayActor.init, right aftermonkey_patch_torch_dist(). That means it runs in every training process before any model, bridge or native, is built.tests/fast/backends/megatron_utils/test_te_flash_attn_4.py. It runs the helper against a stand-in TE module, so it needs neither TE nor a GPU.Measurements (RTX 6000D, sm_120)
The first table covers one full-attention layer of Qwen3.8-27B at TP2 (12 query heads, 2 KV heads, head_dim 256, causal, thd), forward plus backward, with the TE backend forced:
FA2's outputs and gradients match an fp32 reference at least as closely as a plain bf16 implementation does. Its worst max-abs error is 0.85x that of bf16.
End to end, we ran Qwen3.8-27B multi-LoRA through the Tinker gateway with TP2, full recompute, and one 2-GPU group per sample. The longest trainable single sample was:
--log-probs-chunk-size 4096, and without building the checkpoint's untrained MTP block (the change fix: Multi-LoRA correctness fixes across models #3584 makes).Testing
test_te_flash_attn_4.pyand the existingtest_bridge_mtp_detachment.pyboth pass: 9 passed.MLIRError.