Repository navigation
[trainer throughput] Train Qwen3.5-4B FFT on flash-linear-attention 0.6.0 - #352
Draft
Micah Yong (micahtyong) wants to merge 1 commit into
Draft
Micah Yong (micahtyong) wants to merge 1 commit into
Micah Yong (micahtyong) wants to merge 1 commit into
Conversation
Add megatron-2d1fa8d-fla06, the megatron-2d1fa8d image with fla-core and flash-linear-attention 0.6.0, and move qwen35-4b-fft-64k to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds trainer image
megatron-2d1fa8d-fla06:megatron-2d1fa8dwith flash-linear-attention 0.6.0 instead of 0.4.2.qwen35-4b-fft-64kmoves to it; other Megatron configs stay on the old image until each gets its own run.FLA 0.6.0 fixes the
causal_conv1drecompiles upstream, so #333's pin now skips those kernels and only pinsl2norm.kernels.pyis unchanged because configs on FLA 0.4.2 still need both pins. A build step imports the FLA functions Megatron uses, so an incompatible release fails the build, not the trainer.E2E
qwen35-4b-fft-64kon main a3fd861, 4×B200, stock Tinker SDK, cross entropy on fixed random batches. Medians of steps 1–11, two fresh trainers per side. Loss matches within 0.012% on every step; publish, sample and KL are unchanged.Run-to-run spread is as large as the gap, so treat speed as unchanged. The first trainer on the new image spends ~5 minutes compiling kernels once.
W&B: 4K before, after · 64K before, after · publish + sample before, after
Not tested: other Megatron configs on the new image.