Skip to content

[trainer throughput] Balance rows across data-parallel ranks and pad with short micro-batches - #336

Merged
Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/q4fft/b200-balanced-dp
Nov 17, 2026
Merged

Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/q4fft/b200-balanced-dp

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

With data parallelism, each GPU trains its own share of the batch, and all of them wait for the slowest at the end of the step. Two things made that wait long in the full fine-tuning backend.

First, rows were packed into micro-batches before they were split across GPUs, so one GPU could get much more work than another. Second, every GPU must run the same number of micro-batches. A GPU with fewer got padding micro-batches, and each one was a full copy of a real micro-batch, up to 64K tokens, that trained to nothing. In a 64-row batch on four GPUs, three GPUs were running a 64K-token copy while the fourth did real work.

This PR splits rows across GPUs by length first (largest first, each to the least-loaded GPU), then packs each GPU's rows. It also compares that plan with splitting whole packed micro-batches, and keeps whichever needs fewer micro-batches, then whichever gives the busiest GPU less work. Padding micro-batches are now as short as the layout allows (128 tokens on this recipe) and carry no loss. Replay losses (mis, score_centered, score_centered_mis) still pad with a full copy, because their loss needs the copied rows' sampler supports.

Every bundled Megatron recipe runs with one data-parallel rank today, so nothing changes for them until a recipe uses more.

E2E results

Setup: qwen35-4b-fft-64k on 4×B200 with tensor and context parallel 1 (four data-parallel ranks), chunked log-probs (#335), TF32 and #333, stock Tinker SDK, cross entropy on fixed random batches of ~256K tokens. Times are trainer time per step, median after step 0. 64 × 4K = 64 rows of ~4K tokens, 8 × 32K = 8 rows of ~32K tokens, 4 × 64K = four 65,535-token rows.

Short padding, with FlashAttention 2 on both sides: Before W&B · After W&B

Full-copy padding Short padding
64 × 4K rows 7.79 s 4.62 s

Balanced split, with #334 on both sides: Before W&B (4K, 32K, 64K) · After W&B (4K, 32K, 64K)

Short padding only Short padding + balanced split
64 × 4K rows 4.37 s 4.07 s
8 × 32K rows 5.81 s 4.13 s
4 × 64K rows 4.65 s 4.56 s

These timings ran before the last commit, which adds the comparison with splitting whole micro-batches. On the final commit, together with #333, #334 and #335, trainer time is 4.32 s (64 × 4K), 4.33 s (8 × 32K) and 4.69 s (4 × 64K) (4K W&B, 32K W&B, 64K W&B).

A check with context parallel 2 and two data-parallel ranks (5 rows of 40,000 tokens, so one rank needs a padding micro-batch) ran the 128-token padding through the context-parallel split and attention without errors. Loss fell normally, and on the 64 × 4K batch it matched the four-rank run (W&B, 64 × 4K W&B).

🤖 Generated with Claude Code

Packing all sequences first and dealing bins round robin left some
ranks with one more 64K micro-batch than others, and ranks short of a
batch ran a full copy of a real one as a no-loss dummy. Split sequences
across ranks longest-first by token count before packing each rank's
share, and pad the shorter ranks with 128-token dummies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…trims

Balancing sequences before packing can add a micro-batch when a batch
nearly fills every rank. Also balance whole packed bins and keep the plan
with the lighter busiest rank. Make the dummy length required, size it
from sequence_padding_multiple, and test the split across DP ranks with
CP2 and sequence parallelism.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant