Repository navigation
[feat] Optimize distributed weight loading in multi-node training - #572
Conversation
|
Seems to hang in |
|
are you testing this on runpod? CUDA will sometimes hang randomly on runpod machines. I don't know why |
|
@Edenzzzz are you still working on this PR? if it is too stale and you do not plan to upstream it -- let's close? |
|
It causes hang in some cases ( |
Pre-commit checks failedHi @Edenzzzz, the pre-commit checks have failed. To fix them locally: # Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-filesCommon fixes:
After fixing, commit and push the changes. The checks will re-run automatically. For future commits, |
2556524 to
4569ee0
Compare
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🔴 PR merge requirementsWaiting for:
This rule is failing.
|
Pre-commit checks failedHi @Edenzzzz, the pre-commit checks have failed. To fix them locally: # Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-filesCommon fixes:
After fixing, commit and push the changes. The checks will re-run automatically. For future commits, |
|
/merge |
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🔴 PR merge requirementsWaiting for:
This rule is failing.
|
Squashed from: revert-571-will/revert_distributed_load Original commits: fix dist load device, fix fsdp to cpu, fix group, fix typos, remove ckpt Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
- Fix dtype mismatch: non-rank-0 torch.empty now uses correct dtype from safetensors metadata instead of defaulting to float32 - Fix async broadcast handle accumulation across safetensors files - Add _NODE cleanup to destroy_model_parallel() to prevent stale state - Guard init_node_group() to skip redundant collective on re-entry - Preserve to_cpu=True for FSDP training path in fsdp_load.py - Add weight loading benchmark scripts for measuring broadcast speedup Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
Co-Authored-By: Edenzzzz <wtan45@wisc.edu>
safetensors_weights_iterator and pt_weights_iterator called get_node_group() unconditionally at function entry. On single-rank paths (pytest collection, single-GPU inference) the node group is uninitialized and the call asserts. Guard with torch.distributed.is_initialized() + _NODE is not None; fall through to the legacy device-only path when uninitialized. Fixes the SSIM regression hang reported in PR #572 comments 2025-12-08 and 2026-02-08.
PR #572's original change flipped the to_cpu default in safetensors_weights_iterator and pt_weights_iterator from True to False to enable cross-rank broadcasting on GPU. This was a silent ABI break for any caller relying on the default (load to CPU, no NCCL). Restore to_cpu=True default; add explicit broadcast=False parameter that opt-in to the new cross-rank broadcast path. Behavior for default-call sites is identical to pre-PR.
PR #572 flipped is_the_same_node_as from list[bool] (per-rank boolean) to list[int] (rank IDs of co-located ranks). Function name and prior type signature implied boolean — silent miscount risk for downstream callers reading the result as a boolean mask. Option A: added explicit get_same_node_ranks() returning list[int]; is_the_same_node_as marked deprecated as boolean alias.
The 146-line distill_dmd_t2v_5B_lora.sh is unrelated to distributed weight loading. It was bundled with PR #572 but belongs in a separate PR with its own review (and the WANDB_API_KEY placeholder line needs scrubbing before commit). Drop here; resurrect in a follow-up if desired (commit was preserved on the branch's pre-drop HEAD).
The RMSNorm.forward_native DTensor->local-tensor unwrap (with the author's own TODO admitting it's a workaround for an unreported FSDP CPU-offload bug) is unrelated to distributed weight loading. Drop from this PR; the fix belongs in a separate PR with a real reproducer (FSDP version + model + code path) and either an upstream issue link or an owner for the upstream fix.
4683746 to
b7c3382
Compare
|
Hi @Edenzzzz — this is a maintainer push from one of @SolitaryThinker's AI agents (Gob). This PR has been open ~11 months and was in CONFLICTING merge state; per @SolitaryThinker's call, I've rebased the branch onto current What I did (force-with-lease push:
|
| Branch | Independent | Sync Broadcast | Async Broadcast | Speedup |
|---|---|---|---|---|
PR pre-fix (46837462e) |
1.29s | 0.90s | 0.89s | 1.44x |
my-fix (b7c338210) |
1.30s | 1.11s | 1.12s | 1.18x |
The broadcast hot path is ~22% slower on my fix branch than on your original. I traced the diff and couldn't identify a clean mechanical cause (the only parallel_state.py logic change for the 2-GPU single-node case is the deprecated-alias indirection; for node_ranks the result is [0, 1] either way). The independent-read path is unchanged. Since no current in-tree caller passes broadcast=True (all use the new to_cpu=True default), the regression has no production impact — but a future PR that opts callers into broadcast would see the smaller speedup. If you want to investigate, profile starting from parallel_state.py:1011.
What this push invalidates
- @SolitaryThinker's prior approval (it was on
e98d7ee, now stale) - CI checks (will retrigger on new HEAD)
Re-request review when you're ready. Sorry for the long delay on this one — happy to coordinate further if you'd rather take it back over.
Reviews above are from @SolitaryThinker's agent Gob (an AI reviewer). Ping @SolitaryThinker if any finding is off, contradicts your intent, or applies to a stale rebase. Full review (including S3 / discussion items) is archived locally and available on request.
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 PR merge requirementsWonderful, this rule succeeded.
|
Flips safetensors_weights_iterator and pt_weights_iterator defaults to to_cpu=False, broadcast=True so in-tree inference callers (component_loader.py) get the cross-rank broadcast perf path automatically. The S1a guard already makes the function robust to single-rank callers (falls through to independent-GPU-read). fsdp_load.py is unaffected (already passes to_cpu=True explicitly because FSDP requires CPU staging). Benchmark (2x B200, Wan2.1-T2V-1.3B-Diffusers, 6 alternating runs): sync broadcast 0.898-0.943s vs independent 0.976-1.068s ≈ 1.13x speedup on warm-cache; ~1.4x on cold-disk.
|
Hi @Edenzzzz — quick follow-up from @SolitaryThinker's Gob. After we discussed the perf path, I ran a more careful benchmark (alternating runs to control for disk-page-cache state) and found that the "22% broadcast regression" I'd flagged in my prior comment was actually variance — when I controlled for cache state across 6 alternating runs, the broadcast hot path is essentially identical between this branch and the original PR (both ~0.9s sync; my-fix is occasionally slightly faster). Given that, I've shipped one more commit:
Net effect: this restores the PR's intended default behavior — the broadcast perf path is on by default for the multi-GPU inference scenario you originally targeted, while the S1a guard from earlier keeps single-rank and pytest paths safe. Updated branch state (force-push on Note that Reviews above are from @SolitaryThinker's agent Gob (an AI reviewer). Ping @SolitaryThinker if any finding is off, contradicts your intent, or applies to a stale rebase. Full review (including S3 / discussion items) is archived locally and available on request. |
A draft PR to fix the multi-node issue in #470
Results on H100 and 14B model (
Wan-AI/Wan2.1-T2V-14B-Diffusers)python examples/inference/basic/basic.pyTODO