Skip to content

[feat] Optimize distributed weight loading in multi-node training - #572

Merged
mergify[bot] merged 11 commits into
mainfrom
revert-571-will/revert_distributed_load
May 21, 2026
Merged

mergify[bot] merged 11 commits into
mainfrom
revert-571-will/revert_distributed_load

Conversation

@Edenzzzz

@Edenzzzz Edenzzzz commented Jun 29, 2025 •

Copy link
Copy Markdown
Collaborator

A draft PR to fix the multi-node issue in #470

Results on H100 and 14B model (Wan-AI/Wan2.1-T2V-14B-Diffusers)
python examples/inference/basic/basic.py

Method 2 GPUs 4 GPUs
From disk 16 s 21 s
Broadcast (sync) 14 s 15 s
Broadcast (async) 11 s 14 s

TODO

@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 02:09 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 02:09 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 02:09 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 02:09 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 02:09 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 02:09 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 04:37 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 04:37 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 04:37 — with GitHub Actions Inactive
@Edenzzzz
Edenzzzz temporarily deployed to runpod-runners June 29, 2025 04:37 — with GitHub Actions Inactive
@Edenzzzz Edenzzzz added the go label Nov 6, 2025
@Edenzzzz

Edenzzzz commented Dec 8, 2025

Copy link
Copy Markdown
Collaborator Author

Seems to hang in pytest -vs --pdb fastvideo/tests/ssim/test_inference_similarity.py, not sure why yet

@SolitaryThinker

Copy link
Copy Markdown
Collaborator

are you testing this on runpod? CUDA will sometimes hang randomly on runpod machines. I don't know why

@zhisbug

zhisbug commented Feb 8, 2026

Copy link
Copy Markdown
Collaborator

@Edenzzzz are you still working on this PR? if it is too stale and you do not plan to upstream it -- let's close?

@Edenzzzz

Edenzzzz commented Feb 8, 2026

Copy link
Copy Markdown
Collaborator Author

It causes hang in some cases (pytest -vs --pdb fastvideo/tests/ssim/test_inference_similarity.py). I don't have bandwidth now but someone else can try debugging it

@Eigensystem Eigensystem removed the go label Mar 28, 2026
@mergify

mergify Bot commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

Pre-commit checks failed

Hi @Edenzzzz, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

  • yapf: yapf -i <file> (formatting)
  • ruff: ruff check --fix <file> (linting)
  • codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

@SolitaryThinker
SolitaryThinker force-pushed the revert-571-will/revert_distributed_load branch from 2556524 to 4569ee0 Compare April 14, 2026 23:19
@mergify mergify Bot removed the needs-rebase PR has merge conflicts label Apr 14, 2026
@mergify

mergify Bot commented Apr 14, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 PR merge requirements

Waiting for:

  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model)\]
This rule is failing.
  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model)\]
  • check-success=full-suite-passed
  • check-success~=pre-commit

@mergify

mergify Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Pre-commit checks failed

Hi @Edenzzzz, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

  • yapf: yapf -i <file> (formatting)
  • ruff: ruff check --fix <file> (linting)
  • codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

@SolitaryThinker

Copy link
Copy Markdown
Collaborator

/merge

@github-actions github-actions Bot added the ready PR is ready to merge label Apr 14, 2026
@SolitaryThinker SolitaryThinker changed the title Fix distributed weight loading in multi-node training [feat] Optimize distributed weight loading in multi-node training Apr 15, 2026
@mergify mergify Bot added the type: feat New feature or capability label Apr 15, 2026
@mergify

mergify Bot commented Apr 15, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 PR merge requirements

Waiting for:

  • check-success=fastcheck-passed
  • check-success=full-suite-passed
This rule is failing.
  • check-success=fastcheck-passed
  • check-success=full-suite-passed
  • #approved-reviews-by>=1
  • check-success~=pre-commit
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model)\]

SolitaryThinker and others added 10 commits May 20, 2026 15:29
Squashed from: revert-571-will/revert_distributed_load
Original commits: fix dist load device, fix fsdp to cpu, fix group, fix typos, remove ckpt

Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
- Fix dtype mismatch: non-rank-0 torch.empty now uses correct dtype from
  safetensors metadata instead of defaulting to float32
- Fix async broadcast handle accumulation across safetensors files
- Add _NODE cleanup to destroy_model_parallel() to prevent stale state
- Guard init_node_group() to skip redundant collective on re-entry
- Preserve to_cpu=True for FSDP training path in fsdp_load.py
- Add weight loading benchmark scripts for measuring broadcast speedup


Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
safetensors_weights_iterator and pt_weights_iterator called
get_node_group() unconditionally at function entry. On single-rank
paths (pytest collection, single-GPU inference) the node group is
uninitialized and the call asserts. Guard with
torch.distributed.is_initialized() + _NODE is not None; fall through
to the legacy device-only path when uninitialized.

Fixes the SSIM regression hang reported in PR #572 comments 2025-12-08
and 2026-02-08.
PR #572's original change flipped the to_cpu default in
safetensors_weights_iterator and pt_weights_iterator from True to False
to enable cross-rank broadcasting on GPU. This was a silent ABI break
for any caller relying on the default (load to CPU, no NCCL).

Restore to_cpu=True default; add explicit broadcast=False parameter
that opt-in to the new cross-rank broadcast path. Behavior for
default-call sites is identical to pre-PR.
PR #572 flipped is_the_same_node_as from list[bool] (per-rank boolean)
to list[int] (rank IDs of co-located ranks). Function name and prior
type signature implied boolean — silent miscount risk for downstream
callers reading the result as a boolean mask.

Option A: added explicit get_same_node_ranks() returning list[int];
is_the_same_node_as marked deprecated as boolean alias.
The 146-line distill_dmd_t2v_5B_lora.sh is unrelated to distributed weight loading. It was bundled with PR #572 but belongs in a separate PR with its own review (and the WANDB_API_KEY placeholder line needs scrubbing before commit). Drop here; resurrect in a follow-up if desired (commit was preserved on the branch's pre-drop HEAD).
The RMSNorm.forward_native DTensor->local-tensor unwrap (with the author's own TODO admitting it's a workaround for an unreported FSDP CPU-offload bug) is unrelated to distributed weight loading. Drop from this PR; the fix belongs in a separate PR with a real reproducer (FSDP version + model + code path) and either an upstream issue link or an owner for the upstream fix.
@SolitaryThinker
SolitaryThinker force-pushed the revert-571-will/revert_distributed_load branch from 4683746 to b7c3382 Compare May 21, 2026 15:43
@SolitaryThinker

Copy link
Copy Markdown
Collaborator

Hi @Edenzzzz — this is a maintainer push from one of @SolitaryThinker's AI agents (Gob). This PR has been open ~11 months and was in CONFLICTING merge state; per @SolitaryThinker's call, I've rebased the branch onto current main and applied fixes for the must-fix issues flagged in my prior review. I run these to help triage PRs but @SolitaryThinker hasn't personally verified every change — please ping @SolitaryThinker if anything below doesn't match your intent or the PR's design.

What I did (force-with-lease push: 46837462e → b7c338210)

  • Rebased onto current origin/main (resolved one conflict in fastvideo/pipelines/training/__init__.py)
  • Fix 1 (cff639f06): guard get_node_group() lookups in weight iterators. safetensors_weights_iterator and pt_weights_iterator previously called get_node_group() unconditionally; on single-rank paths (pytest / single-GPU) the node group is uninitialized and the call asserted. Now guarded with is_initialized() and _NODE is not None; falls through to a non-distributed path when uninitialized. This is the SSIM hang fix (the hang you flagged on 2025-12-08 and 2026-02-08).
  • Fix 2 (d06156ee0): restore to_cpu=True default; gate broadcast on opt-in flag. Your original change flipped the to_cpu default True → False to enable cross-rank broadcasting. That was a silent ABI break for any caller using the default. Restored to_cpu=True; added explicit broadcast=False parameter. Default callers behave identically to pre-PR (no NCCL); the perf path is opt-in via broadcast=True, to_cpu=False.
  • Fix 3 (c2b66087d): clarify is_the_same_node_as contract. Added explicit get_same_node_ranks() → list[int] function. Kept is_the_same_node_as() as a deprecated alias with docstring warning. Updated the only in-tree caller (init_node_group).
  • Dropped (29733ce68, b7c338210): bundled-scope items — the 146-line distill_dmd_t2v_5B_lora.sh SLURM script and the layernorm.py DTensor unwrap (which your TODO admitted was a workaround for an unreported FSDP bug). Both are unrelated to weight loading and belong in their own PRs. Cherry-pick from c2b66087d if you want to ship either separately.

Verification

  • SSIM hang resolved — pytest fastvideo/tests/ssim/test_wan_t2v_similarity.py -k "1.3B-Diffusers-FLASH_ATTN" on 2× B200 ran in 71 seconds without hanging (failed only at reference comparison because B200 reference videos haven't been seeded — unrelated to PR).
  • Pre-commit clean on all changed files (yapf + ruff + codespell + mypy).
  • Module smoke — both iterators load tensors with no distributed init initialized.

Perf benchmark on 2× B200 (independent vs broadcast)

Branch Independent Sync Broadcast Async Broadcast Speedup
PR pre-fix (46837462e) 1.29s 0.90s 0.89s 1.44x
my-fix (b7c338210) 1.30s 1.11s 1.12s 1.18x

The broadcast hot path is ~22% slower on my fix branch than on your original. I traced the diff and couldn't identify a clean mechanical cause (the only parallel_state.py logic change for the 2-GPU single-node case is the deprecated-alias indirection; for node_ranks the result is [0, 1] either way). The independent-read path is unchanged. Since no current in-tree caller passes broadcast=True (all use the new to_cpu=True default), the regression has no production impact — but a future PR that opts callers into broadcast would see the smaller speedup. If you want to investigate, profile starting from parallel_state.py:1011.

What this push invalidates

  • @SolitaryThinker's prior approval (it was on e98d7ee, now stale)
  • CI checks (will retrigger on new HEAD)

Re-request review when you're ready. Sorry for the long delay on this one — happy to coordinate further if you'd rather take it back over.


Reviews above are from @SolitaryThinker's agent Gob (an AI reviewer). Ping @SolitaryThinker if any finding is off, contradicts your intent, or applies to a stale rebase. Full review (including S3 / discussion items) is archived locally and available on request.

@mergify

mergify Bot commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 PR merge requirements

Wonderful, this rule succeeded.
  • #approved-reviews-by>=1
  • check-success=fastcheck-passed
  • check-success=full-suite-passed
  • check-success~=pre-commit
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

Flips safetensors_weights_iterator and pt_weights_iterator defaults to to_cpu=False, broadcast=True so in-tree inference callers (component_loader.py) get the cross-rank broadcast perf path automatically. The S1a guard already makes the function robust to single-rank callers (falls through to independent-GPU-read). fsdp_load.py is unaffected (already passes to_cpu=True explicitly because FSDP requires CPU staging).

Benchmark (2x B200, Wan2.1-T2V-1.3B-Diffusers, 6 alternating runs): sync broadcast 0.898-0.943s vs independent 0.976-1.068s ≈ 1.13x speedup on warm-cache; ~1.4x on cold-disk.
@SolitaryThinker

Copy link
Copy Markdown
Collaborator

Hi @Edenzzzz — quick follow-up from @SolitaryThinker's Gob.

After we discussed the perf path, I ran a more careful benchmark (alternating runs to control for disk-page-cache state) and found that the "22% broadcast regression" I'd flagged in my prior comment was actually variance — when I controlled for cache state across 6 alternating runs, the broadcast hot path is essentially identical between this branch and the original PR (both ~0.9s sync; my-fix is occasionally slightly faster).

Given that, I've shipped one more commit:

  • d5a62f326 [perf]: enable distributed broadcast weight loading by default — flips safetensors_weights_iterator and pt_weights_iterator defaults from to_cpu=True, broadcast=False to to_cpu=False, broadcast=True. In-tree callers in component_loader.py pass to_cpu explicitly but not broadcast, so they now get the broadcast path automatically for multi-rank inference when CPU offload is disabled. fsdp_load.py is unaffected (already passes explicit to_cpu=True because FSDP requires CPU staging).

Net effect: this restores the PR's intended default behavior — the broadcast perf path is on by default for the multi-GPU inference scenario you originally targeted, while the S1a guard from earlier keeps single-rank and pytest paths safe.

Updated branch state (force-push on revert-571-will/revert_distributed_load was earlier; this is a plain fast-forward on top):

d5a62f326 [perf]: enable distributed broadcast weight loading by default   ← new
b7c338210 [misc]: drop layernorm.py DTensor unwrap from this PR
29733ce68 [misc]: drop unrelated 5B LoRA/DMD SLURM script from this PR
c2b66087d [bugfix]: clarify is_the_same_node_as contract
d06156ee0 [bugfix]: restore to_cpu default; gate broadcast flag
cff639f06 [bugfix]: guard get_node_group() lookups in weight iterators

Note that d06156ee0 restored the to_cpu=True default and d5a62f326 flips it back — leaving them both in history rather than rebasing because each commit was a deliberate iteration and the audit trail seemed more valuable than a clean linear story. Squash-merge will collapse them naturally when the time comes.


Reviews above are from @SolitaryThinker's agent Gob (an AI reviewer). Ping @SolitaryThinker if any finding is off, contradicts your intent, or applies to a stale rebase. Full review (including S3 / discussion items) is archived locally and available on request.

@mergify
mergify Bot merged commit af2ee9c into main May 21, 2026
23 checks passed
@mergify
mergify Bot deleted the revert-571-will/revert_distributed_load branch May 21, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready PR is ready to merge scope: distributed SP, FSDP, USP, multi-node scope: model Model architecture (DiTs, encoders, VAEs) scope: training Training pipeline, methods, configs type: feat New feature or capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants