Skip to content

[sglang-miles] Add allowlisted external weight-update receivers - #42271

Open
ajcasagrande wants to merge 35 commits into
sgl-project:sglang-milesfrom
ajcasagrande:acasagrande/external-weight-receiver
Open

ajcasagrande wants to merge 35 commits into
sgl-project:sglang-milesfrom
ajcasagrande:acasagrande/external-weight-receiver

Conversation

@ajcasagrande

@ajcasagrande ajcasagrande commented Oct 2, 2026 •

Copy link
Copy Markdown

Summary

Add a generic, allowlisted hook that lets an external weight-update receiver own the transfer for a distributed weight-update group, in place of SGLang's torch process group. RL frameworks can then bring their own transport, such as an NCCL M2N reshard managed by a control plane, without SGLang plugins and without encoding control data in group names. With no receivers configured, the default path is unchanged.

init_weights_update_group(receiver=path, receiver_init_payload=…)
  → allowlist check (exact match, before import) → factory(WeightUpdateReceiverContext)
begin_weight_update
  → update_weights_from_distributed(names=[], receiver_payload=…) → receiver.receive(payload)   (× N rounds)
end_weight_update
destroy_weights_update_group → receiver.destroy()
  • Allowlist: --weight-update-receivers lists the dotted module.function factory paths an engine may load. A path is matched exactly before anything is imported. With no flag, every receiver is rejected.
  • Context: the factory receives a WeightUpdateReceiverContext(model, device, tp_rank, tp_size, group_name, master_address, master_port, world_size, rank_offset, init_payload) and returns an object with receive(payload) and destroy(). A malformed receiver is rejected with a TypeError; if its destroy() also fails, that failure is chained as the cause.
  • Fail-closed rounds: a receiver round is rejected before receive() is called, with nothing recorded, when:
    • a weight cache or derived weight cache is active (the same guards the load paths use);
    • a speculative draft model exists, keyed on the group being a receiver group, because the receiver writes only the target model;
    • the round names tensors;
    • the round sets load_format;
    • the round runs outside a begin_weight_update/end_weight_update session.
  • Ownership: receiver groups and torch groups cannot shadow each other's names.

The full contract, including the lifecycle and session side effects, is documented in the RL guide (docs/docs/advanced_features/sglang_for_rl.mdx, "External Weight-Update Receivers"), and the new request fields are added to that page's request tables. The first consumer is ModelExpress's NCCL M2N receiver for Miles.

Also in this PR

  • engine.py ruff format: removes one blank line. The pinned ruff-format hook (v0.15.1) rewrites the file whenever it is touched.
  • Scheduler weight-version test repairs: six tests in test/registered/unit/managers/test_scheduler_weight_version_tracking.py already failed on sglang-miles @ 6c94e57f9f (6 failed / 14 passed).
    • The base stages pending_version inside a session and records it at end_weight_update, but the old tests asserted immediate recording.
    • They also left receive_weights_from_distributed unstubbed.
    • The repairs assert the current session semantics, and the file now passes 21/21. Happy to split these into a separate prep PR.

Test and Validation

  • Unit tests: receiver and forwarding tests, scheduler weight-version, io_struct, the derived-cache guard suites, and the msgpack round trip: 168 passed, 2 skipped. Both skips already exist in test_io_struct.py.
  • Forwarding coverage: forwarding tests pin every hop, including a msgpack round trip: Engine, HTTP structs, tokenizer, scheduler, TP worker, model runner.
  • Lint: ruff 0.15.1, isort 7.0.0 and codespell are clean.
  • No GPU run in this PR: the hook has been exercised by ModelExpress's receiver tests against this tree. A GPU end-to-end refit runs with the Miles integration.

CI States

Latest PR Test (Base): ❌ Run #37078371728
Latest PR Test (Extra): ❌ Run #37078371380
Latest PR Test (AMD ROCm 10): ❌ Run #37078371700

Kangyan-Zhou and others added 30 commits September 29, 2026 17:30
…nd pin SentencePiece for InternVL (sgl-project#41777) (sgl-project#41788)

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
…ty check to a warning (sgl-project#41681) (sgl-project#41802)

Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ject#22623)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lication (sgl-project#27268, sgl-project#31759, sgl-project#30913, sgl-project#30912, sgl-project#33299, sgl-project#36384, sgl-project#38165)

Register adapter identities, stream their tensors within weight-update sessions, and publish checksum-verified fresh versions alongside generation. Preserve per-rank failure cleanup and request-prefix cancellation for adapter lifecycle management.

Co-authored-by: Mathew Han <mathewjhan@gmail.com>
Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…ism (sgl-project#21278, sgl-project#22663, sgl-project#33478)

Expose rank topology for P2P weight mapping and create replicas using scoped runtime-context overrides. Restore the previous configuration on exit, including failed context entry.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…RL weight checks (sgl-project#29339)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…30499, sgl-project#37466, sgl-project#37887)

Co-authored-by: fzyzcjy <ch271828n@outlook.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…gl-project#27723)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…tils (sgl-project#35127)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Shi Dong <shi.dong@radixark.ai>
…weight checks (sgl-project#37649)

Co-authored-by: Zhichenzzz <zczeng@uw.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.