Skip to content

fix(peft): keep adapter input gradient under sequence parallelism - #6491

Open
adityavavreNVDA wants to merge 1 commit into
mainfrom
avavre/lora_sp_fix
Open

adityavavreNVDA wants to merge 1 commit into
mainfrom
avavre/lora_sp_fix

Conversation

@adityavavreNVDA

Copy link
Copy Markdown
Contributor

What does this PR do ?

Fixes #6453: under sequence parallelism, adapters on TELayerNormColumnParallelLinear (LoRA, DoRA, CanonicalLoRA) lost their part of the input gradient. This PR stops using TE's gathered LayerNorm output so the gradient is kept.

Changelog

  • peft/utils.py: get_adapter_attributes_from_linear now always sets return_layernorm_output_gathered = False. TE's LayerNormLinear backward drops the gradient of a gathered LayerNorm output, so the adapter's input-gradient term was never added.
  • peft/utils.py: removed the block that set disable_sequence_parallel_comm = True. The adapter now gets only its local slice of the sequence and does its own all-gather. In the backward pass, that gather sends the adapter's gradient back into the input gradient.
  • peft/utils.py: removed the packaging and importlib.metadata.version imports, which only that block used.
  • peft/utils.py: added a docstring note that sequence_parallel_input_regather no longer changes this function's output. The argument is kept so existing callers don't break.
  • peft/utils.py: added a TODO to bring back the gathered-output path once TE adds the missing gradient.
  • tests/unit_tests/peft/test_utils.py: the default path now expects return_layernorm_output_gathered=False and disable_sequence_parallel_comm=False. Removed the TE version patch the test no longer needs.
  • Cost: one extra all-gather in forward and one extra reduce-scatter in backward for each adapted layer.

🤖 Generated with Claude Code

Signed-off-by: adityavavreNVDA <avavre@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:peft Parameter-efficient fine-tuning (LoRA, adapters) bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] LoRA under sequence parallelism drops the LoRA-A input gradient (default sequence_parallel_input_regather=False)

2 participants