Repository navigation
fix(peft): sum multi-LoRA gradients across tensor-parallel ranks - #6513
Open
micahtyong wants to merge 2 commits into
Open
micahtyong wants to merge 2 commits into
micahtyong wants to merge 2 commits into
Conversation
MultiLoRALinear gathers (column-parallel base) or reduces (row-parallel base) the rank-r intermediate so every TP rank holds the full A·x, then multiplies it by its own shard of B. In backward each rank only computed its own B_r^T·dy_r term of the intermediate's gradient, so lora_A and the adapter's input gradient trained on partial sums. All-reduce the intermediate's gradient (copy_to_tensor_model_parallel_region after the gather/reduce), and, for a column-parallel base without sequence parallelism, the adapter input's gradient, mirroring what ColumnParallelLinear does for its input in the single-LoRA ParallelLinearAdapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Micah Yong <micahtyong@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Micah Yong <micahtyong@gmail.com>
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Fixes partial gradients in
MultiLoRALinearunder tensor parallelism:lora_Aand the adapter's input gradient were missing the other TP ranks' contributions, so multi-LoRA training at TP > 1 silently trained on wrong gradients.Problem
For a TP-sharded (non-replicated) adapter,
MultiLoRALinear.forwardcomputes the rank-rintermediatemid = A·x, gathers it (column-parallel base) or all-reduces it (row-parallel base) so every rank holds the fullmid, and then multiplies it by the rank's own shard oflora_B.The forward is correct. In backward, the true gradient of
midis a sum over ranks,Σ_r B_rᵀ · dy_r, but each rank only computes its own term.gather_from_tensor_model_parallel_regionsplits in backward andreduce_from_tensor_model_parallel_regionis the identity in backward, so nothing sums it. As a result:lora_Agradients are wrong at any TP > 1, for both column- and row-parallel bases.d mid, and for a column-parallel base without sequence parallelism becauseAᵀ · d midis also rank-sharded and is never summed.lora_Bgradients and the forward output are correct, which is why this does not show up in forward-only checks or at step 0 with zero-initialized B.The single-LoRA path (
ParallelLinearAdapter) does not have this problem, because itslinear_outis aColumnParallelLinearwhose input goes throughcopy_to_tensor_model_parallel_region(identity forward, all-reduce backward), and its column-parallellinear_inall-reduces its input gradient unless sequence parallelism is on. The fused multi-LoRA path skipped both.We hit this with multi-LoRA training in Miles at TP 2.
Fix
Mirror
ColumnParallelLinearin the fused path:mid, applycopy_to_tensor_model_parallel_region(mid)so its gradient is all-reduced.copy_to_tensor_model_parallel_region(x)to the adapter input. Under sequence parallelism the existing sequence gather's reduce-scatter already sums it, and a base with suppressed TP communication (shared experts undermoe_shared_expert_overlap) leaves the reduction to its caller, so neither case changes.Replicated adapters are unchanged. The cost is one all-reduce of
[tokens, rank]in backward per adapted linear, plus one of[tokens, hidden]for column-parallel bases without SP, which is the same communication the single-LoRA adapter already does.Changelog
peft/multi_lora_layers.py: all-reduce the gradient of the rank-rintermediate for TP-sharded adapters, and the adapter input's gradient for column-parallel bases without sequence parallelism.tests/unit_tests/peft/test_multi_lora_tp_distributed.py(new): two-rank test that wraps aColumnParallelLinearand aRowParallelLinearbase inMultiLoRALinear(two slots, uneven token spans) and checks the output, the input gradient and every rank'slora_A/lora_Bgradient shard against an unsharded reference.L0_Launch_peft_replicated_adapters.sh: run the new test next to the existing two-rank PEFT test.test_multi_lora_layers.py: the CPU grouped-MM fallback test, which already stubs the TP gather, now also stubscopy_to_tensor_model_parallel_region, since its backward needs a process group.Test evidence
2×H100, fp32 with TF32 disabled, relative error
‖actual − reference‖ / ‖reference‖, max over both ranks:lora_Agrad (slot 0 / slot 1)lora_Agrad (slot 0 / slot 1)lora_Bgradmain(both parametrizations) and passes with the fix.test_multi_lora_layers.py,test_multi_lora.py,test_multi_lora_moe.py(93 passed);test_replicated_peft_distributed.py+ the new test,test_multi_lora_seeded_slot_distributed.pyandtest_shared_expert_lora_distributed.pyall pass on two ranks.TELayerNormColumnParallelLinear→TERowParallelLinearMLP with multi-LoRA on both linears (TE 2.17),lora_Agradients were 72–84% off before the fix and matched TP 1 to fp32 noise after.Additional Information
Suggested labels:
bug,area:peft,needs-review.Under sequence parallelism, the adapter's input gradient for LayerNorm-fused targets is also dropped, by Transformer Engine's
return_layernorm_output_gatheredpath. That is a separate bug, tracked in #6453 / #6491, and is not changed here.#5635 edits the same function for shared experts under
moe_shared_expert_overlap(output-side zero-embedding). The two changes touch different lines and compose; themidgradient all-reduce added here is needed in that configuration too.GitHub Actions CI
See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.
Before your PR is "Ready for review"
Pre checks:
🤖 Generated with Claude Code