Skip to content

[sglang-miles] Allocate packed weight receive buffers directly from metadata - #39265

Merged
yueming-yuan merged 2 commits into
sgl-project:sglang-milesfrom
zianglih:ziang/packed-receiver-empty-miles
Oct 2, 2026
Merged

yueming-yuan merged 2 commits into
sgl-project:sglang-milesfrom
zianglih:ziang/packed-receiver-empty-miles

Conversation

@zianglih

@zianglih zianglih commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

@HumansAnd

Packed weight updates allocate every destination tensor and concatenate their uninitialized bytes immediately before overwriting that buffer with a broadcast. Allocate the receive buffer directly from tensor metadata to remove the discarded concatenation and temporary allocations.

This PR targets sglang-miles at 6c94e57f9ff5b84ba9746f5983ff13ce7e84c68b; current head 0b09ee5c8cf5b582d417586c5afe30199c203738. Full-model weight-sync benefit is unmeasured.

Modifications

  • Add FlattenedTensorBucket.empty(names, dtypes, shapes, device): retain byte offsets and allocate one contiguous uint8 buffer. Reconstruction returns the original typed shared-storage views.
  • Use the factory in the current receive-only path. Preserve upstream's separate begin/load/end lifecycle, single broadcast, shared-storage views and error propagation. Sender packing, mixed-dtype wire format, quantization, collective group and finalization are unchanged.
  • Fail on inconsistent metadata lengths, negative shapes, empty metadata or offsets incompatible with dtype alignment. No padding or dtype promotion.
  • Rebased only the final feature commit bc934b6d9eda8f1fd7aed7c73ea51311cb07308d onto the current target, leaving exactly one commit above it. Retain upstream's removal of the duplicate update-and-load helper and its keyword-only receive API. The factory and six registered tests are unchanged; old target-branch ancestry is excluded.

Accuracy Tests

  • Hosted CI at this rebase: the PR is not a draft, but the CI gate fails because the required run-ci label is absent. The inherited maintenance check also requires main commit 8854857a991e, which is not an ancestor of the selected latest sglang-miles target. The PR intentionally remains exactly one feature commit above that target; no main ancestry was added to bypass this gate.

  • Rebase (2026-10-02): ready for review; rebased onto 6c94e57f9ff5b84ba9746f5983ff13ce7e84c68b. The local results below validate the current head within their stated CPU scope; hosted CI is tracked separately.

Current head 0b09ee5c8cf5b582d417586c5afe30199c203738: macOS arm64 CPU, Python 3.12.12, PyTorch 2.14.1. No container or GPU used for this rebase.

  • Six registered tests cover mixed uint8/FP8/FP32/BF16 bytes, dtype objects/strings, scalar/empty shapes, signed zero/infinity/NaN payloads, reconstruction aliases and invalid metadata. Test contents are unchanged.
  • Bounded project-local checks compile and execute the actual receiver method bodies with only the collective intercepted, using four payload versions. They check byte/shape/dtype equality, shared storage and the branch's receive-only or model-delivery behavior. These checks do not import or qualify the complete serving runtime.
  • The CPU run bypassed only sglang/__init__.py bootstrap; production bucket, CI registration and tests are unchanged. Full runtime import/GPU/NCCL/model-loader integration was not rerun.
  • Ruff F401/F821/F823/UP037, Ruff format, Python compile and git diff --check passed for all changed files; the current registered-test placement/registration and bare-pytest lint checks also passed.
......                                                                   [100%]
6 passed in 0.01s
CPU_ACTUAL_RECEIVER_METHODS_PASS (4 cases; collective intercepted, no serving runtime)
Python 3.12.12, PyTorch 2.14.1, CUDA None
All checks passed!
3 files already formatted
compile: exit 0
git diff --check: exit 0

Registered regression command in an installed SGLang development environment:

CUDA_VISIBLE_DEVICES= PYTHONPATH=python python -m pytest -q \
  --confcutdir=test/registered/rl -p no:cacheprovider \
  test/registered/rl/test_tensor_bucket.py

For the bounded CPU check without top-level serving imports, run from the checkout with pytest/PyTorch installed:

python - <<'CPU'
import pathlib, sys, types
root = pathlib.Path.cwd()
package = types.ModuleType("sglang")
package.__path__ = [str(root / "python/sglang")]
sys.modules["sglang"] = package
import pytest
raise SystemExit(pytest.main([
    "-q", "-p", "no:cacheprovider", "--confcutdir=test/registered/rl",
    "test/registered/rl/test_tensor_bucket.py",
]))
CPU

Previously executed real NCCL protocol test: two B300 GPUs, Python3.12.3, PyTorch2.13.0+cu130, CUDA13.0, NCCL2.29.7. Exact image:
us-docker.pkg.dev/hai-infra/hai/hai-miles@sha256:202dbc57befbd2d8abd626f480dae284fc1989f3826070071eb1298050f8a9ee.

The actual private candidate factory and receive-only method passed four distinct versions each of a100-byte toy and339,751,136-byte GLM16-expert fixture. Both ranks checked every byte, name, dtype, shape and shared-storage address. Dtype objects/strings alternate. This is historical pre-rebase evidence; it does not qualify the current upstream head or full-model loader.

# Standalone harness with its frozen source and manifest files present.
CUDA_VISIBLE_DEVICES=0,1 timeout --signal=TERM --kill-after=10s 240s \
  python test_receiver_nccl.py --output receiver-nccl-cuda-001
TWO_GPU_NCCL_PROTOCOL_PASS

Both workers and parent exited0. Exact owned-process identities were absent and all eight GPUs were idle afterward; no models or serving endpoints were created. Source manifest SHA256 46b5a99af146ad0e6113638f5feeed569fe8fefa85484c3c94d4a181443cbc60; helper 0890d9e9b2a2ca8ba31f415895551a788f30b00a311cd6b11f35e3e33909b80e; full protocol/cleanup receipt a1cea72082c20756c6a8d50efa0123a3c14d83e23f071e1120fc23f647ce8b11.

Speed Tests and Profiling

Previously measured on one B300 in the same image:2 warmups and10 alternating trials per variant. Both variants include an identical local device overwrite and typed-view reconstruction. Completed wall time includes device synchronization. Peak allocation is live PyTorch tensor allocation above the preallocated payload, not RSS or reserved memory.

Tensors Payload bytes Original median wall ms Metadata-only median wall ms Original peak allocation bytes Metadata-only peak allocation bytes
10 21245964 0.144953 0.095179 42493952 21246464
37 84946992 0.408382 0.221236 170936320 85983232
145 339751104 1.403790 0.634502 679526912 339751424

Original construction has one aten::cat; the metadata-only path has none. Local completed-wall reduction is0.050–0.769ms per bucket and temporary allocation is approximately halved. This earlier construction benchmark used an equivalent metadata-only function; the subsequent real NCCL test used the actual production factory. Neither measures full-model loading/finalization or end-to-end synchronization benefit. Unflatten host work and per-name model loading remain unchanged. The larger NVFP4-vs-MXFP8 sync gap is not attributed to this small result.

CUDA_VISIBLE_DEVICES=0 python receiver_construction_bench.py \
  --tensor-bucket receiver-source-001/tensor_bucket.py \
  --weight-updater receiver-source-001/weight_updater.py \
  --output receiver-cuda-001 --device cuda:0 \
  --expert-counts 1 4 16 --warmup 2 --repeats 10 --profile --timeout-seconds 300

Construction helper d6c92e241a30f16563a7955777f8a59f7bf805c5b716a03f263423eeaa6d8561; raw result 1bdac967966b958f80fd760732b769b2ba1b5aa4e71afec4fa8a2c70e5265cb4; analysis c8e229ef147056ca490101f1d6c5b1c0cd03d7755fde360349e420011dfd6260. Complete raw timings, six traces, source/AST and cleanup evidence are preserved in project-local validation artifacts.

Full repository pre-commit and GPU/model benchmarks were not rerun; the current CPU checks above are separate from the historical results.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #37077483582
Latest PR Test (Extra): ⏳ Run #37077483115
Latest PR Test (AMD ROCm 10): ⏳ Run #37077483331

@ziang-and
ziang-and force-pushed the ziang/packed-receiver-empty-miles branch 2 times, most recently from bc934b6 to 0b09ee5 Compare October 2, 2026 18:41
@yueming-yuan
yueming-yuan merged commit 2b7313d into sgl-project:sglang-miles Oct 2, 2026
36 of 45 checks passed
@zianglih
zianglih deleted the ziang/packed-receiver-empty-miles branch October 17, 2026 19:32
nv-hwoo pushed a commit to nv-hwoo/sglang that referenced this pull request Oct 20, 2026
…y from metadata (sgl-project#39265)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants