Skip to content

[fix] Accumulate synchronized multi-LoRA gradients per request - #4480

Merged
yueming-yuan merged 1 commit into
mainfrom
yueming/multilora-grad-accumulation
Oct 28, 2026
Merged

yueming-yuan merged 1 commit into
mainfrom
yueming/multilora-grad-accumulation

Conversation

@yueming-yuan

Copy link
Copy Markdown
Collaborator

What

Multi-LoRA keeps each slot's gradients in the DDP buffers across forward_backward requests until that slot's optim_step. The old reset_grad_metadata_keep_grads reset only the DDP bookkeeping, so each request's gradient sync reduced the whole buffer again, including the gradients retained from earlier requests.

That is harmless for dense parameters: the all-reduce averages DP replicas that already hold identical values. It is wrong for expert parameters. Megatron's expert DDP reduces over expert-DP but scales by 1 / dp_cp_size, the full EP × expert-DP world. Every re-sync therefore multiplies the retained expert gradients by 1 / EP. With EP2, two identical requests accumulate 1.5 g instead of 2 g, a 25% error.

accumulate_gradients replaces it:

  1. Save the accumulated buffers, then zero_grad_buffer().
  2. Run and sync this request's gradients alone.
  3. Add the saved buffers back.

If the request raises, the saved buffers are restored instead, so a failed request leaves no partial gradient. zero_grad_buffer() also does what the old helper did: it resets grad_added_to_main_grad (with the same TE CUDA graph exception) and the bucket groups.

The fix applies to any multi-LoRA run that trains expert adapters with EP > 1. On main that is the per-expert layout; no shared-outer support is needed to hit it.

Cost: during each request's forward/backward, one extra copy of all slots' LoRA gradient buffers per rank (4 bytes per trainable LoRA parameter with grad_reduce_in_fp32), freed after the request. Peak memory was not measured.

Split out of #3899, which stacks on this PR and keeps the shared-outer EP-replica sync.

Validation

tests/e2e/lora/test_multi_lora_grad_accumulation.py builds per-expert MultiLoRAGroupedExpertLinear fc1/fc2 with two slots under EP2 and checks that two identical requests leave exactly twice one request's main_grad, bitwise.

On 2× B200 with Megatron-Bridge 8cd3466d (the pinned version) and Megatron-LM miles-main 2fbe25a44:

tree result
this branch passes on both ranks
main d4ff2ba, same test driven through reset_grad_metadata_keep_grads fails on both ranks: Mismatched elements: 512 / 512 (100.0%), Greatest relative difference: 0.25

The 0.25 is the predicted error: 1.5 g vs 2 g.

Pre-commit passes. GPU CI scope: run-ci-multi-lora.

🤖 Generated with Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@HJSang HJSang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice catch on the 1/EP rescale of retained expert grads.

@yueming-yuan yueming-yuan changed the title Accumulate synchronized multi-LoRA gradients per request [fix] Accumulate synchronized multi-LoRA gradients per request Oct 28, 2026
@yueming-yuan
yueming-yuan force-pushed the yueming/multilora-grad-accumulation branch from 981bfda to bc087f8 Compare October 28, 2026 22:11
@yueming-yuan
yueming-yuan merged commit 1f9fdb8 into main Oct 28, 2026
12 of 16 checks passed
@yueming-yuan
yueming-yuan deleted the yueming/multilora-grad-accumulation branch October 28, 2026 22:12
micahtyong added a commit to modal-labs/spindle that referenced this pull request Nov 16, 2026
Merge main and replace the pre-divide and startup check with
radixark/miles#4480's accumulate_gradients, which the miles-v0.2.0 image
already runs: set kept gradients aside for each training pass so its reduce
only sees the new gradients, then add them back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants