Skip to content

[trainer throughput] Train Qwen3.5-4B FFT on flash-linear-attention 0.6.0 - #352

Draft
Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/q4fft/fla-0.6
Draft

Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/q4fft/fla-0.6

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds trainer image megatron-2d1fa8d-fla06: megatron-2d1fa8d with flash-linear-attention 0.6.0 instead of 0.4.2. qwen35-4b-fft-64k moves to it; other Megatron configs stay on the old image until each gets its own run.

FLA 0.6.0 fixes the causal_conv1d recompiles upstream, so #333's pin now skips those kernels and only pins l2norm. kernels.py is unchanged because configs on FLA 0.4.2 still need both pins. A build step imports the FLA functions Megatron uses, so an incompatible release fails the build, not the trainer.

E2E

qwen35-4b-fft-64k on main a3fd861, 4×B200, stock Tinker SDK, cross entropy on fixed random batches. Medians of steps 1–11, two fresh trainers per side. Loss matches within 0.012% on every step; publish, sample and KL are unchanged.

FLA 0.4.2 FLA 0.6.0
Step, 64 × 4K rows 6.7 / 6.9 s 5.3 / 6.5 s
Step, 4 × 64K rows 7.0 / 7.0 s 5.6 / 6.8 s

Run-to-run spread is as large as the gap, so treat speed as unchanged. The first trainer on the new image spends ~5 minutes compiling kernels once.

W&B: 4K before, after · 64K before, after · publish + sample before, after

Not tested: other Megatron configs on the new image.

Add megatron-2d1fa8d-fla06, the megatron-2d1fa8d image with fla-core and
flash-linear-attention 0.6.0, and move qwen35-4b-fft-64k to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant