Repository navigation
[trainer throughput] Balance rows across data-parallel ranks and pad with short micro-batches - #336
Merged
Merged
Conversation
Packing all sequences first and dealing bins round robin left some ranks with one more 64K micro-batch than others, and ranks short of a batch ran a full copy of a real one as a no-loss dummy. Split sequences across ranks longest-first by token count before packing each rank's share, and pad the shorter ranks with 128-token dummies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…trims Balancing sequences before packing can add a micro-batch when a batch nearly fills every rank. Also balance whole packed bins and keep the plan with the lighter busiest rank. Make the dummy length required, size it from sequence_padding_multiple, and test the split across DP ranks with CP2 and sequence parallelism. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 17, 2026 05:51
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With data parallelism, each GPU trains its own share of the batch, and all of them wait for the slowest at the end of the step. Two things made that wait long in the full fine-tuning backend.
First, rows were packed into micro-batches before they were split across GPUs, so one GPU could get much more work than another. Second, every GPU must run the same number of micro-batches. A GPU with fewer got padding micro-batches, and each one was a full copy of a real micro-batch, up to 64K tokens, that trained to nothing. In a 64-row batch on four GPUs, three GPUs were running a 64K-token copy while the fourth did real work.
This PR splits rows across GPUs by length first (largest first, each to the least-loaded GPU), then packs each GPU's rows. It also compares that plan with splitting whole packed micro-batches, and keeps whichever needs fewer micro-batches, then whichever gives the busiest GPU less work. Padding micro-batches are now as short as the layout allows (128 tokens on this recipe) and carry no loss. Replay losses (
mis,score_centered,score_centered_mis) still pad with a full copy, because their loss needs the copied rows' sampler supports.Every bundled Megatron recipe runs with one data-parallel rank today, so nothing changes for them until a recipe uses more.
E2E results
Setup:
qwen35-4b-fft-64kon 4×B200 with tensor and context parallel 1 (four data-parallel ranks), chunked log-probs (#335), TF32 and #333, stock Tinker SDK, cross entropy on fixed random batches of ~256K tokens. Times are trainer time per step, median after step 0. 64 × 4K = 64 rows of ~4K tokens, 8 × 32K = 8 rows of ~32K tokens, 4 × 64K = four 65,535-token rows.Short padding, with FlashAttention 2 on both sides: Before W&B · After W&B
Balanced split, with #334 on both sides: Before W&B (4K, 32K, 64K) · After W&B (4K, 32K, 64K)
These timings ran before the last commit, which adds the comparison with splitting whole micro-batches. On the final commit, together with #333, #334 and #335, trainer time is 4.32 s (64 × 4K), 4.33 s (8 × 32K) and 4.69 s (4 × 64K) (4K W&B, 32K W&B, 64K W&B).
A check with context parallel 2 and two data-parallel ranks (5 rows of 40,000 tokens, so one rank needs a padding micro-batch) ran the 128-token padding through the context-parallel split and attention without errors. Loss fell normally, and on the 64 × 4K batch it matched the four-rank run (W&B, 64 × 4K W&B).
🤖 Generated with Claude Code