Skip to content

feat(peft): seed multi-LoRA slot re-initialization in init_adapter_slot - #6068

Merged
huvunvidia merged 4 commits into
NVIDIA-NeMo:mainfrom
radixark:upstream-pr/multilora-seeded-slot-init
Sep 16, 2026
Merged

huvunvidia merged 4 commits into
NVIDIA-NeMo:mainfrom
radixark:upstream-pr/multilora-seeded-slot-init

Conversation

@yushengsu-thu

@yushengsu-thu yushengsu-thu commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds an optional seed to the model-level init_adapter_slot() so a multi-LoRA adapter slot can be (re)initialised reproducibly across ranks and runs.

Changelog

  • init_adapter_slot(model, idx, rank, alpha, seed=None): with seed, snapshot the CUDA RNG-tracker states, rebuild every stream from the seed via tracker.add (so byte-tensor and graph-safe generator states both work), call reset_adapter(idx) then init_adapter_slot(idx, rank, alpha) on every multi-LoRA layer in model order, and restore the saved states in a finally. Without seed behaviour is unchanged.
  • Tests: CPU tests for the unseeded (weights kept) and seeded (every layer re-drawn, B zeroed) paths in TestMultiLoRAModelHelpers; a GPU test in TestMultiLoRALinearGPU that the same seed reproduces identical weights, a different seed does not, and the tracker streams are restored afterwards.

Motivation: in disaggregated multi-LoRA serving/training a slot is reused for many adapters. Today its weights come from construction or from the previous clear_adapter_slot() re-init, which is DP-consistent but not reproducible; RL pipelines that restart an adapter need a deterministic init.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) — no

Additional Information

Validation

  • tests/unit_tests/peft: 446 passed, 7 skipped (baseline 443). New tests PASSED by name: TestMultiLoRAModelHelpers::test_init_adapter_slot_unseeded_keeps_slot_weights, TestMultiLoRAModelHelpers::test_init_adapter_slot_seeded_reinitializes_every_layer, TestMultiLoRALinearGPU::test_init_adapter_slot_seeded_is_reproducible (GPU, NCCL + model-parallel init).
  • GPU: rx devbox 2× H200 (h200-sci-k8s), image nvcr.io/nvidia/pytorch:26.08-py3 (the upstream CI base image), TE 2.18.0+27486e03 (matches upstream uv.lock), Megatron-Core at .main.commit c9b53d0a8, Megatron-Bridge editable. Command mirrors CI's Launch_Unit_Tests_Core.sh: CUDA_VISIBLE_DEVICES=0,1 pytest -m "not pleasefixme". Baseline upstream/main@2eae3c971 on the same box: tests/unit_tests/peft 443 passed / 7 skipped, the three model-bridge export files 181 passed, test_qwen3_next_bridge.py 23 passed. ruff check, ruff check --select I, ruff format --check (ruff 0.9.9) pass on the changed files.

Revision 2 (review follow-up, 5c77076)

  • Reseed through model_parallel_cuda_manual_seed(seed + 100 * pp_rank) instead of a rank-independent seed + offset per stream, so model-parallel-rng keeps +2718+tp_rank, expert-parallel-rng keeps +1024+100*ep_rank+etp_rank and data-parallel-rng stays equal across replicas; TP/EP/ETP shards draw distinct values while DP replicas match, exactly as the base weights do.
  • reset() / add() moved inside the try; finally restores the tracker states and the default CUDA generator.
  • New two-rank module tests/unit_tests/peft/test_multi_lora_seeded_slot_distributed.py (torch.distributed.run --nproc_per_node=2): TP=2 shards distinct and reproducible per rank, DP=2 identical, EP=2 expert stream distinct while the data-parallel stream matches. CPU tests stub the topology-aware reseed.
  • Validation (2× RTX 4090, nvcr.io/nvidia/pytorch:26.08-py3, TE 2.18.0+27486e03, Megatron-Core at .main.commit): fail-before with the previous multi_lora_layers.py → 2 failed, 1 passed on the new module (TP=2 shards collapse); pass-after → 3 passed on both ranks; tests/unit_tests/peft single-process 446 passed / 10 skipped with no new failures against main (443 / 7); the three seeded tests PASSED by name.

🤖 Generated with Claude Code

Slot weights come from construction or from the previous
``clear_adapter_slot`` re-init, so creating an adapter in a slot is
DP-consistent but not reproducible across runs. ``init_adapter_slot``
gains an optional ``seed``: when given, the RNG-tracker streams are
rebuilt from the seed (through ``tracker.add``, so byte-tensor and
graph-safe generator states both work), every layer's slot is re-
initialised in order, and the original streams are restored. A given
seed therefore reproduces the same adapter weights on every rank and
every run; without ``seed`` behaviour is unchanged.

Ported from #34 (3adbed3) onto upstream main.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:peft Parameter-efficient fine-tuning (LoRA, adapters) feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer labels Sep 12, 2026
yaoyu-33
yaoyu-33 previously approved these changes Sep 12, 2026
@yaoyu-33 yaoyu-33 added ready-to-merge PR is approved, current, and only waiting for CI to pass before merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Sep 12, 2026
# tracker's native state representation (byte tensor or graph-safe generator).
tracker.reset()
for offset, name in enumerate(sorted(saved_states)):
tracker.add(name, seed + offset + 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking correctness concern: reseeding every named stream from the same rank-independent seed removes the topology offsets that Megatron-Core deliberately applies to the model-parallel and expert-parallel streams. Non-replicated TP shards and EP/ETP expert shards can therefore be initialized identically instead of reconstructing the intended full weight, while only DP replicas should match. Please preserve the TP/EP/ETP-dependent seed semantics and add a TP=2 / EP=2 test that checks DP equality without collapsing distinct shards. Also, tracker.reset() and tracker.add() should be covered by the try/finally so a failed add cannot leave the tracker cleared.

@yushengsu-thu yushengsu-thu Sep 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — agreed on both points, fixed in 5c77076 (rebased on your merge of main).

…A slot

Review feedback on the seeded ``init_adapter_slot`` path: reseeding every
tracker stream from the same rank-independent value dropped the offsets
Megatron-Core applies per topology, so tensor-parallel and expert-parallel
shards of a slot were initialised identically instead of drawing distinct
values, and ``tracker.reset()`` / ``tracker.add()`` ran outside the
try/finally.

Rebuild the streams through ``model_parallel_cuda_manual_seed`` instead, so
``model-parallel-rng`` keeps ``+2718+tp_rank``, ``expert-parallel-rng`` keeps
``+1024+100*ep_rank+etp_rank`` and ``data-parallel-rng`` stays equal across
replicas; offset pipeline stages by ``100*pp_rank`` like
``_set_random_seed`` does. The reseed now runs inside the try/finally, and the
default CUDA generator (reseeded by Megatron-Core as a side effect) is restored
alongside the tracker states.

Tests: a two-rank ``torch.distributed.run`` module covering TP=2 (shards
distinct and reproducible per rank), DP=2 (replicas identical) and EP=2
(expert stream distinct, data-parallel stream identical); the CPU tests stub
the topology-aware reseed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor

/ok to test 5c77076

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The latest changes address the review comments and add coverage for the affected boundary.

@huvunvidia

Copy link
Copy Markdown
Contributor

/ok to test a256856

This branch was previously deployed

1 inactive deployment
test — a2568560 Deployed Sep 14, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #21811
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:peft Parameter-efficient fine-tuning (LoRA, adapters) community-request feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge ready-to-merge PR is approved, current, and only waiting for CI to pass before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants