Skip to content

fix(peft): sum multi-LoRA gradients across tensor-parallel ranks - #6513

Open
micahtyong wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
micahtyong:fix/multi-lora-tp-grad-allreduce
Open

micahtyong wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
micahtyong:fix/multi-lora-tp-grad-allreduce

Conversation

@micahtyong

@micahtyong micahtyong commented Oct 16, 2026 •

Copy link
Copy Markdown

What does this PR do ?

Fixes partial gradients in MultiLoRALinear under tensor parallelism: lora_A and the adapter's input gradient were missing the other TP ranks' contributions, so multi-LoRA training at TP > 1 silently trained on wrong gradients.

Problem

For a TP-sharded (non-replicated) adapter, MultiLoRALinear.forward computes the rank-r intermediate mid = A·x, gathers it (column-parallel base) or all-reduces it (row-parallel base) so every rank holds the full mid, and then multiplies it by the rank's own shard of lora_B.

The forward is correct. In backward, the true gradient of mid is a sum over ranks, Σ_r B_rᵀ · dy_r, but each rank only computes its own term. gather_from_tensor_model_parallel_region splits in backward and reduce_from_tensor_model_parallel_region is the identity in backward, so nothing sums it. As a result:

  • lora_A gradients are wrong at any TP > 1, for both column- and row-parallel bases.
  • The gradient flowing into the layer input through the adapter is wrong as well: for a row-parallel base because it is computed from the partial d mid, and for a column-parallel base without sequence parallelism because Aᵀ · d mid is also rank-sharded and is never summed.
  • lora_B gradients and the forward output are correct, which is why this does not show up in forward-only checks or at step 0 with zero-initialized B.

The single-LoRA path (ParallelLinearAdapter) does not have this problem, because its linear_out is a ColumnParallelLinear whose input goes through copy_to_tensor_model_parallel_region (identity forward, all-reduce backward), and its column-parallel linear_in all-reduces its input gradient unless sequence parallelism is on. The fused multi-LoRA path skipped both.

We hit this with multi-LoRA training in Miles at TP 2.

Fix

Mirror ColumnParallelLinear in the fused path:

  • After the gather/reduce of mid, apply copy_to_tensor_model_parallel_region(mid) so its gradient is all-reduced.
  • For a column-parallel base without sequence parallelism, apply copy_to_tensor_model_parallel_region(x) to the adapter input. Under sequence parallelism the existing sequence gather's reduce-scatter already sums it, and a base with suppressed TP communication (shared experts under moe_shared_expert_overlap) leaves the reduction to its caller, so neither case changes.

Replicated adapters are unchanged. The cost is one all-reduce of [tokens, rank] in backward per adapted linear, plus one of [tokens, hidden] for column-parallel bases without SP, which is the same communication the single-LoRA adapter already does.

Changelog

  • peft/multi_lora_layers.py: all-reduce the gradient of the rank-r intermediate for TP-sharded adapters, and the adapter input's gradient for column-parallel bases without sequence parallelism.
  • tests/unit_tests/peft/test_multi_lora_tp_distributed.py (new): two-rank test that wraps a ColumnParallelLinear and a RowParallelLinear base in MultiLoRALinear (two slots, uneven token spans) and checks the output, the input gradient and every rank's lora_A/lora_B gradient shard against an unsharded reference.
  • L0_Launch_peft_replicated_adapters.sh: run the new test next to the existing two-rank PEFT test.
  • test_multi_lora_layers.py: the CPU grouped-MM fallback test, which already stubs the TP gather, now also stubs copy_to_tensor_model_parallel_region, since its backward needs a process group.

Test evidence

2×H100, fp32 with TF32 disabled, relative error ‖actual − reference‖ / ‖reference‖, max over both ranks:

Base Quantity Before After
column-parallel lora_A grad (slot 0 / slot 1) 0.863 / 0.900 < 1e-6
column-parallel input grad 0.828 < 1e-6
row-parallel lora_A grad (slot 0 / slot 1) 0.743 / 0.591 < 1e-6
row-parallel input grad 0.683 < 1e-6
both output, lora_B grad < 1e-6 < 1e-6
  • The new test fails on main (both parametrizations) and passes with the fix.
  • With the fix: test_multi_lora_layers.py, test_multi_lora.py, test_multi_lora_moe.py (93 passed); test_replicated_peft_distributed.py + the new test, test_multi_lora_seeded_slot_distributed.py and test_shared_expert_lora_distributed.py all pass on two ranks.
  • Separately, in a TP 2 vs TP 1 comparison of a TELayerNormColumnParallelLinear → TERowParallelLinear MLP with multi-LoRA on both linears (TE 2.17), lora_A gradients were 72–84% off before the fix and matched TP 1 to fp32 noise after.

Additional Information

Suggested labels: bug, area:peft, needs-review.

Under sequence parallelism, the adapter's input gradient for LayerNorm-fused targets is also dropped, by Transformer Engine's return_layernorm_output_gathered path. That is a separate bug, tracked in #6453 / #6491, and is not changed here.

#5635 edits the same function for shared experts under moe_shared_expert_overlap (output-side zero-embedding). The two changes touch different lines and compose; the mid gradient all-reduce added here is needed in that configuration too.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation? (not applicable: bug fix with no user-facing change)
  • Does the PR affect components that are optional to install? No.

🤖 Generated with Claude Code

MultiLoRALinear gathers (column-parallel base) or reduces (row-parallel
base) the rank-r intermediate so every TP rank holds the full A·x, then
multiplies it by its own shard of B. In backward each rank only computed
its own B_r^T·dy_r term of the intermediate's gradient, so lora_A and the
adapter's input gradient trained on partial sums.

All-reduce the intermediate's gradient (copy_to_tensor_model_parallel_region
after the gather/reduce), and, for a column-parallel base without sequence
parallelism, the adapter input's gradient, mirroring what ColumnParallelLinear
does for its input in the single-LoRA ParallelLinearAdapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Micah Yong <micahtyong@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Micah Yong <micahtyong@gmail.com>
@micahtyong

Copy link
Copy Markdown
Author

Hi @yaoyu-33, would you be able to review this PR? I hit this when trying to get multi-LoRA working with DS v4.1 Flash, and looking at prior PRs (ex: #6068), you might have the most relevant context here.

@cuichenx cuichenx added bug Something isn't working area:peft Parameter-efficient fine-tuning (LoRA, adapters) needs-review PR is ready for code review and waiting on a reviewer labels Oct 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:peft Parameter-efficient fine-tuning (LoRA, adapters) bug Something isn't working community-request needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants