Skip to content

[RL] Stream LoRA adapter weights through the weight-update session - #41044

Open
yueming-yuan wants to merge 7 commits into
mainfrom
yueming/rl-lora-streamed-updates
Open

yueming-yuan wants to merge 7 commits into
mainfrom
yueming/rl-lora-streamed-updates

Conversation

@yueming-yuan

@yueming-yuan yueming-yuan commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

RL training that fine-tunes LoRA adapters needs to push adapter weights into the rollout engine on every step, alongside or instead of base weights, without a second transport, without a full reload, and without serving a half-written adapter. On main the only route is load_lora_adapter_from_tensors, which is a separate whole-adapter IPC call outside the weight-update session.

Modifications

  • Streamed adapter updates. POST /register_lora_adapter (also Engine.register_lora_adapter) creates or refreshes an adapter's identity and config with zeroed weights. The weights then travel as {lora_name}:{hf_key}-prefixed tensors in the ordinary update_weights_from_{tensor,distributed} stream: the scheduler stashes them for the session and applies each adapter whole at end_weight_update, verifying an optional expected_lora_checksums manifest (name set + sha256). begin_weight_update(sync_base=False) opens an adapter-only session that skips the base unpack/finalize and rejects base tensors.
  • Staged publication. register_lora_adapter(defer_publish=True) keeps a fresh name unservable until its session commits; such sessions take the reader lock and run alongside generation, end_weight_update(abort=True) discards the stash and the pending identities, and requests may carry lora_backfill_paths so a fresh engine serves a published adapter from disk.
  • Lease lifecycle. Every terminal path of a request releases its LoRA lease exactly once (_finalize_lora_lease), the registry distinguishes reloadable (disk-backed) from wire-only adapters, and over-cap eviction is one helper (_evict_lru_over_cap).
  • weight_updater.py: the session struct gains sync_base and pending_version; the version reported by buckets is recorded only when the session commits.
  • Tests: test_lora_stream_update.py, test_lora_staged_publication.py, test_lora_lease.py, test_lora_upsert.py (CPU units) and test_lora_load_from_tensor.py (e2e, now through the stream).

Squash of sglang-miles #27268, #31759, #30913, #33299, #36384, #38165, rebased onto the _WeightUpdateSession design of #40777 (authors credited as co-authors). Not carried over: load_lora_adapter_from_distributed / upsert (the intermediate API on sglang-miles, superseded by the stream). main's load_lora_adapter_from_tensors is unchanged.

Deliberately different from sglang-miles: an explicit unload_lora_adapter keeps the adapter's entry in the reload catalog (lora_ref_cache). sglang-miles made it a delete; main's dynamic-update contract (test_lora_update) is that a later request for an unloaded disk-backed adapter reloads it implicitly, and wire-only adapters are already refused by the reloadable check.

Testing

pre-commit clean; file runs via /rerun-test on the LoRA unit tests, the LoRA e2e set, and the weight-update e2e tests.


CI States

Latest PR Test (Base): ❌ Run #35969117738
Latest PR Test (Extra): ❌ Run #35969117453
Latest PR Test (AMD ROCm 10): ❌ Run #35969117682

Port the sglang-miles LoRA weight-update line to main as its final state:

- Streamed adapter updates: register_lora_adapter creates or refreshes an
  adapter's identity and config (weights zeroed); the bytes then arrive as
  "{lora_name}:{hf_key}"-prefixed tensors in the ordinary
  update_weights_from_{tensor,distributed} stream, are stashed for the
  session, and are applied whole (optionally checksum-verified) at
  end_weight_update. begin_weight_update(sync_base=False) opens an
  adapter-only session that skips the base unpack/finalize and rejects base
  tensors.
- Staged publication: a defer_publish registration keeps the name unservable
  until its session commits; such sessions run under the reader lock,
  concurrent with generation, and an abort discards the stash and the
  pending identities. Requests may carry lora_backfill_paths so a fresh
  engine can serve a published adapter from disk.
- Multi-LoRA lease redesign: every terminal path releases a request's LoRA
  lease exactly once through _finalize_lora_lease; the registry tracks
  reloadable vs wire-only adapters and the over-cap eviction is one helper.
- load_lora_adapter_from_distributed / upsert (the intermediate API on
  sglang-miles) is superseded by the stream and not added; main's
  load_lora_adapter_from_tensors stays.

Squash of sglang-miles #27268, #31759, #30913, #33299, #36384, #38165,
rebased onto the _WeightUpdateSession design of #40777.

Co-authored-by: Yusheng Su <yushengsu.thu@gmail.com>
Co-authored-by: gongyisheng <yishenggong9437@gmail.com>
Co-authored-by: Mathew Han <mathewjhan@gmail.com>
@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/tag-run-ci-label

@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/lora/test_lora_stream_update.py test/registered/unit/lora/test_lora_staged_publication.py test/registered/unit/lora/test_lora_lease.py test/registered/unit/lora/test_lora_upsert.py test/registered/unit/lora/test_eviction_policy.py test/registered/unit/lora/test_lora_manager_tied_lm_head.py test/registered/unit/lora/test_lora_moe_inplace_unit.py test/registered/unit/lora/test_uno_inactive_lora_batch.py test/registered/unit/managers/test_msgpack_ipc_roundtrip.py test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/managers/test_tokenizer_manager_rid_cleanup.py test/registered/unit/managers/test_tokenizer_config_updates.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py test/registered/rl/test_lora_load_from_tensor.py test/registered/rl/test_update_weights_from_tensor.py test/registered/rl/test_update_weights_from_distributed.py test/registered/rl/test_weight_checker_e2e.py test/registered/lora/test_lora_update.py test/registered/lora/test_lora_eviction.py test/registered/lora/test_lora_overlap_loading.py test/registered/lora/test_lora_drainer.py test/registered/lora/test_lora_openai_api.py test/registered/lora/test_lora_qwen3.py test/registered/lora/test_lora_tp.py test/registered/lora/test_multi_lora_backend.py test/registered/lora/test_lora_spec_decoding.py test/registered/lora/test_lora_hf_sgl_logprob_diff.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/lora/test_lora_stream_update.py test/registered/unit/lora/test_lora_staged_publication.py test/registered/unit/lora/test_lora_lease.py test/registered/unit/lora/test_lora_upsert.py test/registered/unit/lora/test_eviction_policy.py test/registered/unit/lora/test_lora_manager_tied_lm_head.py test/registered/unit/lora/test_lora_moe_inplace_unit.py test/registered/unit/lora/test_uno_inactive_lora_batch.py test/registered/unit/managers/test_msgpack_ipc_roundtrip.py test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/managers/test_tokenizer_manager_rid_cleanup.py test/registered/unit/managers/test_tokenizer_config_updates.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py test/registered/rl/test_lora_load_from_tensor.py test/registered/rl/test_update_weights_from_tensor.py test/registered/rl/test_update_weights_from_distributed.py test/registered/rl/test_weight_checker_e2e.py test/registered/lora/test_lora_update.py test/registered/lora/test_lora_eviction.py test/registered/lora/test_lora_overlap_loading.py test/registered/lora/test_lora_drainer.py test/registered/lora/test_lora_openai_api.py test/registered/lora/test_lora_qwen3.py test/registered/lora/test_lora_tp.py test/registered/lora/test_multi_lora_backend.py test/registered/lora/test_lora_spec_decoding.py test/registered/lora/test_lora_hf_sgl_logprob_diff.py:

🚀 ubuntu-latest (13 tests): ❌ View workflow run

cd test/ && python3 registered/unit/lora/test_lora_stream_update.py
cd test/ && python3 registered/unit/lora/test_lora_staged_publication.py
cd test/ && python3 registered/unit/lora/test_lora_lease.py
cd test/ && python3 registered/unit/lora/test_lora_upsert.py
cd test/ && python3 registered/unit/lora/test_eviction_policy.py
cd test/ && python3 registered/unit/lora/test_lora_manager_tied_lm_head.py
cd test/ && python3 registered/unit/lora/test_lora_moe_inplace_unit.py
cd test/ && python3 registered/unit/lora/test_uno_inactive_lora_batch.py
cd test/ && python3 registered/unit/managers/test_msgpack_ipc_roundtrip.py
cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py
cd test/ && python3 registered/unit/managers/test_tokenizer_manager_rid_cleanup.py
cd test/ && python3 registered/unit/managers/test_tokenizer_config_updates.py
cd test/ && python3 registered/lora/test_lora_openai_api.py

🚀 1-gpu-h100 (6 tests): ❌ View workflow run

cd test/ && python3 registered/rl/test_lora_load_from_tensor.py
cd test/ && python3 registered/rl/test_weight_checker_e2e.py
cd test/ && python3 registered/lora/test_lora_update.py
cd test/ && python3 registered/lora/test_lora_overlap_loading.py
cd test/ && python3 registered/lora/test_multi_lora_backend.py
cd test/ && python3 registered/lora/test_lora_spec_decoding.py

🚀 1-gpu-5090 (4 tests): ✅ View workflow run

cd test/ && python3 registered/rl/test_update_weights_from_tensor.py
cd test/ && python3 registered/lora/test_lora_eviction.py
cd test/ && python3 registered/lora/test_lora_drainer.py
cd test/ && python3 registered/lora/test_lora_hf_sgl_logprob_diff.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/rl/test_update_weights_from_distributed.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/lora/test_lora_tp.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

⛔ test/registered/lora/test_lora_qwen3.py: test/registered/lora/test_lora_qwen3.py is registered for AMD (suite stage-b-test-1-gpu-small-amd), not for CUDA or CPU; rerun-test.yml has no AMD job. Rerun it with /rerun-failed-ci, or dispatch the AMD workflow manually.

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 24, 2026
… reads

unload_lora_adapter checks --enable-lora and dp_size through get_lora() /
get_parallel(), not server_args, and rejects names awaiting publication, so
the bare TokenizerManager needs the context override and an empty
_pending_lora_publications.
@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/lora/test_lora_lease.py test/registered/unit/lora/test_lora_stream_update.py test/registered/unit/lora/test_lora_staged_publication.py test/registered/unit/lora/test_lora_upsert.py test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/lora/test_lora_lease.py test/registered/unit/lora/test_lora_stream_update.py test/registered/unit/lora/test_lora_staged_publication.py test/registered/unit/lora/test_lora_upsert.py test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py:

🚀 ubuntu-latest (5 tests): ❌ View workflow run

cd test/ && python3 registered/unit/lora/test_lora_lease.py
cd test/ && python3 registered/unit/lora/test_lora_stream_update.py
cd test/ && python3 registered/unit/lora/test_lora_staged_publication.py
cd test/ && python3 registered/unit/lora/test_lora_upsert.py
cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

sglang-miles turned an explicit unload into a DELETE of the lora_ref_cache
entry. main's dynamic-update contract (test_lora_update) is that a request
for an unloaded disk-backed adapter reloads it implicitly, so the catalog
entry stays; wire-only adapters are already refused by the reloadable check.
Also hand the version-tracking fixture an iterable weight list, since the
distributed path now splits the received tensors before loading.
@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/lora/test_lora_lease.py test/registered/unit/lora/test_lora_stream_update.py test/registered/unit/lora/test_lora_staged_publication.py test/registered/unit/lora/test_lora_upsert.py test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py test/registered/lora/test_lora_update.py test/registered/lora/test_lora_drainer.py test/registered/lora/test_lora_openai_api.py test/registered/lora/test_lora_qwen3.py test/registered/lora/test_multi_lora_backend.py test/registered/lora/test_lora_spec_decoding.py test/registered/lora/test_lora_hf_sgl_logprob_diff.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/lora/test_lora_lease.py test/registered/unit/lora/test_lora_stream_update.py test/registered/unit/lora/test_lora_staged_publication.py test/registered/unit/lora/test_lora_upsert.py test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py test/registered/lora/test_lora_update.py test/registered/lora/test_lora_drainer.py test/registered/lora/test_lora_openai_api.py test/registered/lora/test_lora_qwen3.py test/registered/lora/test_multi_lora_backend.py test/registered/lora/test_lora_spec_decoding.py test/registered/lora/test_lora_hf_sgl_logprob_diff.py:

🚀 ubuntu-latest (6 tests): ❌ View workflow run

cd test/ && python3 registered/unit/lora/test_lora_lease.py
cd test/ && python3 registered/unit/lora/test_lora_stream_update.py
cd test/ && python3 registered/unit/lora/test_lora_staged_publication.py
cd test/ && python3 registered/unit/lora/test_lora_upsert.py
cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py
cd test/ && python3 registered/lora/test_lora_openai_api.py

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/lora/test_lora_update.py
cd test/ && python3 registered/lora/test_multi_lora_backend.py
cd test/ && python3 registered/lora/test_lora_spec_decoding.py

🚀 1-gpu-5090 (2 tests): ✅ View workflow run

cd test/ && python3 registered/lora/test_lora_drainer.py
cd test/ && python3 registered/lora/test_lora_hf_sgl_logprob_diff.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

⛔ test/registered/lora/test_lora_qwen3.py: test/registered/lora/test_lora_qwen3.py is registered for AMD (suite stage-b-test-1-gpu-small-amd), not for CUDA or CPU; rerun-test.yml has no AMD job. Rerun it with /rerun-failed-ci, or dispatch the AMD workflow manually.

@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py:

🚀 ubuntu-latest (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

update_weights_from_* inside a session hold the version they carry until
end_weight_update, so a version never names a half-applied refit; the
tracking test asserted the pre-session immediate record.
@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py test/registered/unit/lora/test_lora_stream_update.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py test/registered/unit/lora/test_lora_stream_update.py:

🚀 ubuntu-latest (2 tests): ❌ View workflow run

cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py
cd test/ && python3 registered/unit/lora/test_lora_stream_update.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

The fixture opened a session for every path; inside one the version is
held until commit, which is not the checkpoint-engine IPC flow.
@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py:

🚀 ubuntu-latest (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py:

🚀 ubuntu-latest (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@yueming-yuan

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/unit/managers/test_scheduler_weight_version_tracking.py test/registered/unit/rl/test_distributed_weight_update_spec_worker.py:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/managers/test_scheduler_weight_version_tracking.py

⛔ test/registered/unit/rl/test_distributed_weight_update_spec_worker.py: File not found: test/registered/unit/rl/test_distributed_weight_update_spec_worker.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lora run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant