Repository navigation
[fix] Accumulate synchronized multi-LoRA gradients per request - #4480
Merged
Merged
Conversation
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
yueming-yuan
requested review from
Shi-Dong,
Zhichenzzz and
yushengsu-thu
as code owners
October 27, 2026 22:10
yueming-yuan
force-pushed
the
yueming/multilora-grad-accumulation
branch
from
October 28, 2026 05:37
afb1a8b to
981bfda
Compare
HJSang
approved these changes
Oct 28, 2026
HJSang
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. Nice catch on the 1/EP rescale of retained expert grads.
yueming-yuan
force-pushed
the
yueming/multilora-grad-accumulation
branch
from
October 28, 2026 22:11
981bfda to
bc087f8
Compare
micahtyong
added a commit
to modal-labs/spindle
that referenced
this pull request
Nov 16, 2026
Merge main and replace the pre-divide and startup check with radixark/miles#4480's accumulate_gradients, which the miles-v0.2.0 image already runs: set kept gradients aside for each training pass so its reduce only sees the new gradients, then add them back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Multi-LoRA keeps each slot's gradients in the DDP buffers across
forward_backwardrequests until that slot'soptim_step. The oldreset_grad_metadata_keep_gradsreset only the DDP bookkeeping, so each request's gradient sync reduced the whole buffer again, including the gradients retained from earlier requests.That is harmless for dense parameters: the all-reduce averages DP replicas that already hold identical values. It is wrong for expert parameters. Megatron's expert DDP reduces over expert-DP but scales by
1 / dp_cp_size, the full EP × expert-DP world. Every re-sync therefore multiplies the retained expert gradients by1 / EP. With EP2, two identical requests accumulate1.5 ginstead of2 g, a 25% error.accumulate_gradientsreplaces it:zero_grad_buffer().If the request raises, the saved buffers are restored instead, so a failed request leaves no partial gradient.
zero_grad_buffer()also does what the old helper did: it resetsgrad_added_to_main_grad(with the same TE CUDA graph exception) and the bucket groups.The fix applies to any multi-LoRA run that trains expert adapters with EP > 1. On
mainthat is the per-expert layout; no shared-outer support is needed to hit it.Cost: during each request's forward/backward, one extra copy of all slots' LoRA gradient buffers per rank (4 bytes per trainable LoRA parameter with
grad_reduce_in_fp32), freed after the request. Peak memory was not measured.Split out of #3899, which stacks on this PR and keeps the shared-outer EP-replica sync.
Validation
tests/e2e/lora/test_multi_lora_grad_accumulation.pybuilds per-expertMultiLoRAGroupedExpertLinearfc1/fc2 with two slots under EP2 and checks that two identical requests leave exactly twice one request'smain_grad, bitwise.On 2× B200 with Megatron-Bridge 8cd3466d (the pinned version) and Megatron-LM
miles-main2fbe25a44:maind4ff2ba, same test driven throughreset_grad_metadata_keep_gradsMismatched elements: 512 / 512 (100.0%),Greatest relative difference: 0.25The 0.25 is the predicted error:
1.5 gvs2 g.Pre-commit passes. GPU CI scope:
run-ci-multi-lora.🤖 Generated with Claude Code