Repository navigation
[sglang-miles] Allocate packed weight receive buffers directly from metadata - #39265
Merged
yueming-yuan merged 2 commits intoOct 2, 2026
Merged
yueming-yuan merged 2 commits into
yueming-yuan merged 2 commits into
Conversation
Zhichenzzz
force-pushed
the
sglang-miles
branch
from
September 21, 2026 06:58
c9660a7 to
5a8da8c
Compare
7 of 15 tasks
ziang-and
force-pushed
the
ziang/packed-receiver-empty-miles
branch
from
September 27, 2026 05:41
151fed7 to
bc934b6
Compare
zianglih
marked this pull request as ready for review
September 27, 2026 06:46
zianglih
requested review from
Fridge003,
Ying1123,
hnyls2002,
ispobock and
merrymercy
as code owners
September 27, 2026 06:46
guapisolo
force-pushed
the
sglang-miles
branch
from
October 2, 2026 06:45
14a1fa7 to
6c94e57
Compare
guapisolo
requested review from
CatherineSue,
HaiShaw,
JustinTong0323,
Qiaolin-Yu,
b8zhong,
ch-wan,
fzyzcjy,
key4ng,
kpham-sgl,
lifuhuang,
mickqian,
mmangkad,
slin1237,
xiezhq-hermann,
yctseng0211,
yhyang201 and
yuan-luo
as code owners
October 2, 2026 06:45
guapisolo
requested review from
1am9trash,
AgainstEntropy,
Alisehen,
AniZpZ,
ByronHsu,
DarkSharpness,
Duyi-Wang,
HydraQYH,
OrangeRedeng,
ShangmingCai,
YAMY1234,
celve,
hebiao064,
hubertlu-tw,
kevin-mii,
kkHuang-amd,
niehen6174,
ping1jing2,
rainj-me,
sogalin,
yichiche and
yizhang2077
as code owners
October 2, 2026 06:45
ziang-and
force-pushed
the
ziang/packed-receiver-empty-miles
branch
2 times, most recently
from
October 2, 2026 18:41
bc934b6 to
0b09ee5
Compare
2 of 5 tasks
nv-hwoo
pushed a commit
to nv-hwoo/sglang
that referenced
this pull request
Oct 20, 2026
…y from metadata (sgl-project#39265) Co-authored-by: yueming-yuan <yym022502@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
@HumansAnd
Packed weight updates allocate every destination tensor and concatenate their uninitialized bytes immediately before overwriting that buffer with a broadcast. Allocate the receive buffer directly from tensor metadata to remove the discarded concatenation and temporary allocations.
This PR targets
sglang-milesat6c94e57f9ff5b84ba9746f5983ff13ce7e84c68b; current head0b09ee5c8cf5b582d417586c5afe30199c203738. Full-model weight-sync benefit is unmeasured.Modifications
FlattenedTensorBucket.empty(names, dtypes, shapes, device): retain byte offsets and allocate one contiguous uint8 buffer. Reconstruction returns the original typed shared-storage views.bc934b6d9eda8f1fd7aed7c73ea51311cb07308donto the current target, leaving exactly one commit above it. Retain upstream's removal of the duplicate update-and-load helper and its keyword-only receive API. The factory and six registered tests are unchanged; old target-branch ancestry is excluded.Accuracy Tests
Hosted CI at this rebase: the PR is not a draft, but the CI gate fails because the required
run-cilabel is absent. The inherited maintenance check also requires main commit8854857a991e, which is not an ancestor of the selected latestsglang-milestarget. The PR intentionally remains exactly one feature commit above that target; no main ancestry was added to bypass this gate.Rebase (2026-10-02): ready for review; rebased onto
6c94e57f9ff5b84ba9746f5983ff13ce7e84c68b. The local results below validate the current head within their stated CPU scope; hosted CI is tracked separately.Current head
0b09ee5c8cf5b582d417586c5afe30199c203738: macOS arm64 CPU, Python 3.12.12, PyTorch 2.14.1. No container or GPU used for this rebase.sglang/__init__.pybootstrap; production bucket, CI registration and tests are unchanged. Full runtime import/GPU/NCCL/model-loader integration was not rerun.git diff --checkpassed for all changed files; the current registered-test placement/registration and bare-pytest lint checks also passed.Registered regression command in an installed SGLang development environment:
For the bounded CPU check without top-level serving imports, run from the checkout with pytest/PyTorch installed:
Previously executed real NCCL protocol test: two B300 GPUs, Python3.12.3, PyTorch2.13.0+cu130, CUDA13.0, NCCL2.29.7. Exact image:
us-docker.pkg.dev/hai-infra/hai/hai-miles@sha256:202dbc57befbd2d8abd626f480dae284fc1989f3826070071eb1298050f8a9ee.The actual private candidate factory and receive-only method passed four distinct versions each of a100-byte toy and339,751,136-byte GLM16-expert fixture. Both ranks checked every byte, name, dtype, shape and shared-storage address. Dtype objects/strings alternate. This is historical pre-rebase evidence; it does not qualify the current upstream head or full-model loader.
# Standalone harness with its frozen source and manifest files present. CUDA_VISIBLE_DEVICES=0,1 timeout --signal=TERM --kill-after=10s 240s \ python test_receiver_nccl.py --output receiver-nccl-cuda-001Both workers and parent exited0. Exact owned-process identities were absent and all eight GPUs were idle afterward; no models or serving endpoints were created. Source manifest SHA256
46b5a99af146ad0e6113638f5feeed569fe8fefa85484c3c94d4a181443cbc60; helper0890d9e9b2a2ca8ba31f415895551a788f30b00a311cd6b11f35e3e33909b80e; full protocol/cleanup receipta1cea72082c20756c6a8d50efa0123a3c14d83e23f071e1120fc23f647ce8b11.Speed Tests and Profiling
Previously measured on one B300 in the same image:2 warmups and10 alternating trials per variant. Both variants include an identical local device overwrite and typed-view reconstruction. Completed wall time includes device synchronization. Peak allocation is live PyTorch tensor allocation above the preallocated payload, not RSS or reserved memory.
Original construction has one
aten::cat; the metadata-only path has none. Local completed-wall reduction is0.050–0.769ms per bucket and temporary allocation is approximately halved. This earlier construction benchmark used an equivalent metadata-only function; the subsequent real NCCL test used the actual production factory. Neither measures full-model loading/finalization or end-to-end synchronization benefit. Unflatten host work and per-name model loading remain unchanged. The larger NVFP4-vs-MXFP8 sync gap is not attributed to this small result.Construction helper
d6c92e241a30f16563a7955777f8a59f7bf805c5b716a03f263423eeaa6d8561; raw result1bdac967966b958f80fd760732b769b2ba1b5aa4e71afec4fa8a2c70e5265cb4; analysisc8e229ef147056ca490101f1d6c5b1c0cd03d7755fde360349e420011dfd6260. Complete raw timings, six traces, source/AST and cleanup evidence are preserved in project-local validation artifacts.Full repository pre-commit and GPU/model benchmarks were not rerun; the current CPU checks above are separate from the historical results.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ⏳ Run #37077483582
Latest PR Test (Extra): ⏳ Run #37077483115
Latest PR Test (AMD ROCm 10): ⏳ Run #37077483331