Repository navigation
feat(peft): seed multi-LoRA slot re-initialization in init_adapter_slot - #6068
huvunvidia merged 4 commits into
Conversation
Slot weights come from construction or from the previous ``clear_adapter_slot`` re-init, so creating an adapter in a slot is DP-consistent but not reproducible across runs. ``init_adapter_slot`` gains an optional ``seed``: when given, the RNG-tracker streams are rebuilt from the seed (through ``tracker.add``, so byte-tensor and graph-safe generator states both work), every layer's slot is re- initialised in order, and the original streams are restored. A given seed therefore reproduces the same adapter weights on every rank and every run; without ``seed`` behaviour is unchanged. Ported from #34 (3adbed3) onto upstream main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
| # tracker's native state representation (byte tensor or graph-safe generator). | ||
| tracker.reset() | ||
| for offset, name in enumerate(sorted(saved_states)): | ||
| tracker.add(name, seed + offset + 1) |
There was a problem hiding this comment.
Blocking correctness concern: reseeding every named stream from the same rank-independent seed removes the topology offsets that Megatron-Core deliberately applies to the model-parallel and expert-parallel streams. Non-replicated TP shards and EP/ETP expert shards can therefore be initialized identically instead of reconstructing the intended full weight, while only DP replicas should match. Please preserve the TP/EP/ETP-dependent seed semantics and add a TP=2 / EP=2 test that checks DP equality without collapsing distinct shards. Also, tracker.reset() and tracker.add() should be covered by the try/finally so a failed add cannot leave the tracker cleared.
There was a problem hiding this comment.
Thanks — agreed on both points, fixed in 5c77076 (rebased on your merge of main).
…A slot Review feedback on the seeded ``init_adapter_slot`` path: reseeding every tracker stream from the same rank-independent value dropped the offsets Megatron-Core applies per topology, so tensor-parallel and expert-parallel shards of a slot were initialised identically instead of drawing distinct values, and ``tracker.reset()`` / ``tracker.add()`` ran outside the try/finally. Rebuild the streams through ``model_parallel_cuda_manual_seed`` instead, so ``model-parallel-rng`` keeps ``+2718+tp_rank``, ``expert-parallel-rng`` keeps ``+1024+100*ep_rank+etp_rank`` and ``data-parallel-rng`` stays equal across replicas; offset pipeline stages by ``100*pp_rank`` like ``_set_random_seed`` does. The reseed now runs inside the try/finally, and the default CUDA generator (reseeded by Megatron-Core as a side effect) is restored alongside the tracker states. Tests: a two-rank ``torch.distributed.run`` module covering TP=2 (shards distinct and reproducible per rank), DP=2 (replicas identical) and EP=2 (expert stream distinct, data-parallel stream identical); the CPU tests stub the topology-aware reseed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
|
/ok to test 5c77076 |
yaoyu-33
left a comment
There was a problem hiding this comment.
LGTM. The latest changes address the review comments and add coverage for the affected boundary.
|
/ok to test a256856 |
What does this PR do ?
Adds an optional
seedto the model-levelinit_adapter_slot()so a multi-LoRA adapter slot can be (re)initialised reproducibly across ranks and runs.Changelog
init_adapter_slot(model, idx, rank, alpha, seed=None): withseed, snapshot the CUDA RNG-tracker states, rebuild every stream from the seed viatracker.add(so byte-tensor and graph-safe generator states both work), callreset_adapter(idx)theninit_adapter_slot(idx, rank, alpha)on every multi-LoRA layer in model order, and restore the saved states in afinally. Withoutseedbehaviour is unchanged.TestMultiLoRAModelHelpers; a GPU test inTestMultiLoRALinearGPUthat the same seed reproduces identical weights, a different seed does not, and the tracker streams are restored afterwards.Motivation: in disaggregated multi-LoRA serving/training a slot is reused for many adapters. Today its weights come from construction or from the previous
clear_adapter_slot()re-init, which is DP-consistent but not reproducible; RL pipelines that restart an adapter need a deterministic init.GitHub Actions CI
See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.
Before your PR is "Ready for review"
Pre checks:
Additional Information
Validation
tests/unit_tests/peft: 446 passed, 7 skipped (baseline 443). New tests PASSED by name:TestMultiLoRAModelHelpers::test_init_adapter_slot_unseeded_keeps_slot_weights,TestMultiLoRAModelHelpers::test_init_adapter_slot_seeded_reinitializes_every_layer,TestMultiLoRALinearGPU::test_init_adapter_slot_seeded_is_reproducible(GPU, NCCL + model-parallel init).rx devbox2× H200 (h200-sci-k8s), imagenvcr.io/nvidia/pytorch:26.08-py3(the upstream CI base image), TE 2.18.0+27486e03 (matches upstreamuv.lock), Megatron-Core at.main.commitc9b53d0a8, Megatron-Bridge editable. Command mirrors CI'sLaunch_Unit_Tests_Core.sh:CUDA_VISIBLE_DEVICES=0,1 pytest -m "not pleasefixme". Baselineupstream/main@2eae3c971on the same box:tests/unit_tests/peft443 passed / 7 skipped, the three model-bridge export files 181 passed,test_qwen3_next_bridge.py23 passed.ruff check,ruff check --select I,ruff format --check(ruff 0.9.9) pass on the changed files.Revision 2 (review follow-up, 5c77076)
model_parallel_cuda_manual_seed(seed + 100 * pp_rank)instead of a rank-independentseed + offsetper stream, somodel-parallel-rngkeeps+2718+tp_rank,expert-parallel-rngkeeps+1024+100*ep_rank+etp_rankanddata-parallel-rngstays equal across replicas; TP/EP/ETP shards draw distinct values while DP replicas match, exactly as the base weights do.reset()/add()moved inside thetry;finallyrestores the tracker states and the default CUDA generator.tests/unit_tests/peft/test_multi_lora_seeded_slot_distributed.py(torch.distributed.run --nproc_per_node=2): TP=2 shards distinct and reproducible per rank, DP=2 identical, EP=2 expert stream distinct while the data-parallel stream matches. CPU tests stub the topology-aware reseed.nvcr.io/nvidia/pytorch:26.08-py3, TE 2.18.0+27486e03, Megatron-Core at.main.commit): fail-before with the previousmulti_lora_layers.py→2 failed, 1 passedon the new module (TP=2 shards collapse); pass-after →3 passedon both ranks;tests/unit_tests/peftsingle-process 446 passed / 10 skipped with no new failures againstmain(443 / 7); the three seeded tests PASSED by name.🤖 Generated with Claude Code