Skip to content

[Fix] [XPU] weight_sync: stage serialized XPU tensors through host shared memory - #40720

Open
siju-samuel wants to merge 1 commit into
sgl-project:mainfrom
siju-samuel:fix/xpu-weight-sync-shm-fallback
Open

siju-samuel wants to merge 1 commit into
sgl-project:mainfrom
siju-samuel:fix/xpu-weight-sync-shm-fallback

Conversation

@siju-samuel

@siju-samuel siju-samuel commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Engine.update_weights_from_tensor() crashes on Intel XPU, so online RL training (GRPO/RLHF through verl and similar trainers) cannot push weights into a colocated engine after each optimizer.step(). Inference is unaffected.

PyTorch has no XPU tensor IPC: UntypedStorage has _share_cuda_ but no _share_xpu_, and torch.multiprocessing.reductions.reduce_tensor has only CUDA and meta branches (intel/torch-xpu-ops#1678). An XPU tensor falls through to the CPU storage path:

File "torch/multiprocessing/reductions.py", in reduce_storage
    fd, size = storage._share_fd_cpu_()
RuntimeError: _share_fd_: only available on CPU

Under Ray it fails instead with AuthenticationError: digest received was wrong, because fd passing needs an authkey that a trainer actor and an mp.spawn-ed scheduler never share.

Modifications

Transport (python/sglang/srt/utils/xpu_tensor_shm.py). Each serialized XPU tensor is staged through a POSIX shared-memory segment. The producer copies device → host, the payload carries only the segment name (174 B regardless of tensor size), and each scheduler copies host → its own device. There is no fd passing, so no authkey dependency. posix_fallocate turns an exhausted /dev/shm into ENOSPC instead of SIGBUS.

Segment lifetime follows who can observe the readers. Each scheduler reads payload[tp_rank], so a payload has one reader per pipeline stage (DP replicas are already rejected for this API, tokenizer_control_mixin.py:565). Stage 0's TP0 replies only after its TP group has loaded: SchedulerWeightUpdaterManager.update_weights_from_tensor ends on a TP barrier. Other stages may still be loading at that point, and they read in no fixed order: a stage holding a request behind pause_generation relays it onward before dispatching it itself (ingest_requests returns the held requests for relay, since #48992). So neither the producer nor any single stage can tell it is the last reader.

Producer Who unlinks Host copies per call
Engine.update_weights_from_tensor, load_lora_adapter_from_tensors, HttpServerEngineAdapter, PP = 1 the producer, on return (staged_xpu_segments(pp_size=1)) 1 (a tensor serialized once per TP rank is staged once)
the same, PP > 1 the pp_size-th reader, whichever stage that is tp_size
hand-off producers that return before the request does (weight_sync.utils.update_weights on its non-root ranks) the pp_size-th reader one per rank

Reader-owned segments count finished reads in an 8-byte trailer, incremented under flock; the reader that brings the count to pp_size unlinks. Readers never touch multiprocessing's resource tracker — a spawned scheduler shares its Engine's tracker, so a reader that registered and unregistered would race the other readers for one entry and could remove the producer's own, losing the reclaim of a scope whose producer is killed.

Security. _rebuild_xpu_tensor_from_shm is allowlisted in SafeUnpickler.ALLOWED_GLOBALS, and it accepts only the segment-name shape make_shm_name("xputensor") produces. The name arrives in the pickle that POST /update_weights_from_tensor takes. That route is ADMIN_OPTIONAL, so it's unauthenticated by default, and an unchecked name would read and unlink any /dev/shm segment the scheduler's uid owns. That is the channel SafeUnpickler was narrowed for in CVE-2025-10164.

Patch installation. monkey_patch_xpu_tensor_reductions() is installed at engine startup (_set_envs_and_config) and on scope entry. The new call sites install only that patch, not monkey_patch_torch_reductions(), so CUDA producers' payloads are unchanged.

_reduce_tensor_modified fix (patch_torch.py). It rewrote argument index 6 into a device UUID on every reduction, but only the CUDA-IPC payload carries a device there. A CPU tensor reduces to 3 arguments and died with IndexError. A meta tensor reduces to 7, where index 6 is requires_grad, so the UUID was written over it. The rewrite is now keyed off the rebuild function.

Docs. docs/docs/hardware-platforms/xpu.mdx gains an RL weight-sync section covering /dev/shm sizing, cost, what callers that serialize their own payloads must do, and how to clear a leak.

Notes for review

  • A review asked that the synchronous producers own the unlink so --pp-size 2 works. That alone is not safe: the call returns when stage 0 replies, and with the producer unlinking on return, stage 1 found its segment gone and crashed the server. It reproduced deterministically on hardware. A barrier inside the handler can't fix it either, because stage 0 relays the request only after its handler returns. An earlier revision used a "last pipeline stage unlinks" rule instead, which relied on stages reading in order; [Fix] scheduler: Honor pause_generation under PP and release aborted PD rooms and rejected sessions #48992 made a stage relay requests held behind pause_generation before dispatching them, so stage 1 can read first. Counting reads is independent of order.
  • The same review also found that rejected requests leaked their staging, and that a process-wide registry could unlink a bucket the caller had pre-serialized for its next call. Both are fixed by the per-call scope.
  • Behavior on other platforms: no change on CUDA, ROCm or NPU. The new install calls and staged_xpu_segments are no-ops off XPU. The only shared-path change is the _reduce_tensor_modified short-circuit for non-CUDA-IPC payloads. Those payloads previously crashed (CPU) or were corrupted (meta), and the CUDA-IPC rewrite itself is untouched; the existing 2-GPU CUDA case in test_patch_torch.py still covers it.

Known limitations

  • Producer and consumer must be on the same node, as with CUDA IPC.
  • With PP > 1, a request rejected before loading (outside a weight-update session) leaves its staging, since no stage reads it. For hand-off producers, a crash between staging and loading does the same. Both are documented, with rm /dev/shm/sgl_shm_xputensor_* as the remedy.

Accuracy Tests

Not applicable: this is a transport-only change. Weights arrive bit-exact; the unit tests compare raw bytes across bf16, fp8, int64, 0-d, empty and transposed tensors.

Speed Tests and Profiling

Measured on Arc Pro B60, torch 2.14.0.dev+xpu:

  • Staging runs at ~1 GiB/s device → host and ~2.2 GiB/s host → device (256 and 512 MiB tensors). A pinned intermediate buffer was not consistently faster and doubles host memory, so the copy goes straight into the segment.
  • update_weights_from_tensor for one 32 MiB up_proj on Llama-3.2-1B takes 0.041–0.060 s at TP=1, TP=2 and PP=2.
  • At PP=1, a call stages one host copy per tensor instead of tp_size.

Tests

test/registered/utils/test_xpu_tensor_shm.py (registered for stage-b-test-1-gpu-xpu) guards one failure mode per case:

  • round trip is bit-exact (dtype, shape, contiguity, Parameter);
  • a payload survives an unrelated producer process exiting, with no shared authkey;
  • foreign segment names are refused, and the segment survives;
  • one segment serves every TP rank's reader;
  • PP > 1: every stage reads, in either order, and only the second read unlinks;
  • a reader leaves the producer's crash-time cleanup intact (a killed producer's segment is still reclaimed);
  • the scope reclaims payloads nobody read;
  • the scope leaves pre-serialized payloads alone;
  • a tensor written in place between serializations is restaged rather than served stale.

test/registered/rl/test_patch_torch.py adds CPU and meta tensors through the patched reducer, including requires_grad=True leaves.

End-to-end on 8× Arc Pro B60, inside a begin_weight_update() / end_weight_update() session:

Config Model Result
TP=1 Llama-3.2-1B-Instruct (True, 'Success'), readback 1.5, generation OK, 0 segments left
TP=2 Llama-3.2-1B-Instruct same
PP=2 Llama-3.1-8B-Instruct (True, 'Success'). Zeroing layers.31.down_proj, which lives on stage 1, changes greedy output, so stage 1 loaded it; 0 segments left
no session Llama-3.2-1B-Instruct, TP=1 rejected before deserializing, 0 segments left
PP=2, under an in-place pause_generation with a 1500-token generation in flight Llama-3.1-8B-Instruct both stages log Weights were updated, greedy output changes, the paused generation completes after continue_generation, 0 segments left. The call itself returns success=False only because main refuses the default cache flush while the paused request still holds KV (Weights were updated, but the cache was not flushed because requests are still running); pass flush_cache=False when updating under an in-place pause

Re-run on ba3873b (rebased onto 3981de0): TP=1, TP=2 and PP=2 all (True, 'Success') with the last layer's update visible in greedy output, 0 segments left, and no KeyError from multiprocessing's resource tracker in any log — where two concurrent readers on the previous head produced one. Whether a stage actually held the request in the paused run is timing-dependent and not visible in the logs; the order independence itself is pinned deterministically by the unit test.

The previous head of this PR, on the PP=2 case, returned (True, 'Success') and then crashed stage 1 with Shared-memory segment ... is gone.

PP=2 on Llama-3.2-1B crashes during generation with 'PPMissingLayer' object has no attribute 'quant_method'. That also happens on unmodified main with no weight update: tied embeddings put lm_head on stage 0. It is unrelated to this PR, which is why the PP=2 row uses the 8B model.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #43805005497
Latest PR Test (Extra): ❌ Run #43805005431
Latest PR Test (AMD ROCm 10): ❌ Run #43805005391

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 22, 2026
@siju-samuel
siju-samuel force-pushed the fix/xpu-weight-sync-shm-fallback branch from 0753a34 to d6367c3 Compare September 23, 2026 10:56
@Amrutha-M05 Amrutha-M05 mentioned this pull request Oct 8, 2026
4 of 5 tasks
@siju-samuel
siju-samuel marked this pull request as ready for review October 8, 2026 05:35
@siju-samuel

Copy link
Copy Markdown
Contributor Author

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 8, 2026
@siju-samuel
siju-samuel force-pushed the fix/xpu-weight-sync-shm-fallback branch from 15ba90c to 20766ef Compare October 14, 2026 07:22
@siju-samuel siju-samuel changed the title [XPU] Stage serialized XPU tensors through host shared memory [Fix] [XPU] weight_sync: stage serialized XPU tensors through host shared memory Oct 14, 2026
@mintlify

mintlify Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit ba3873b.

Approve preview

@mingfeima

Copy link
Copy Markdown
Collaborator

@siju-samuel The host shared-memory approach looks reasonable, and the new XPU tests did run and pass.

Can you resolve the merge conflicts and get the required CI running on the updated head? One performance question too: staging and reconstruction synchronize once per tensor (producer, consumer). Do you have full-model weight-update latency numbers? It would be good to check whether those waits can be grouped at the payload boundary without changing the buffer-lifetime guarantees.

@siju-samuel
siju-samuel force-pushed the fix/xpu-weight-sync-shm-fallback branch from 838c9ea to 4b06ae0 Compare November 9, 2026 11:04
@siju-samuel

Copy link
Copy Markdown
Contributor Author

Thanks for looking. Rebased onto main and force-pushed — head is now 4b06ae0.

The conflicts were in common.py and patch_torch.py, and both were caused by the module split rather than by overlapping edits: SafeUnpickler moved to utils/serialization.py and is_xpu to utils/device.py, so the allowlist entry and the module's import now point there. common.py is no longer touched by this PR at all. Tests re-run green on the rebased tree (9 cases, 14 subtests), and CI is going on the new head.

On the two per-tensor waits

They are not where the time goes, and grouping them at the payload boundary measured no faster. One Arc Pro B60, torch 2.14.0.dev+xpu.

The copy itself blocks, so the synchronize() after it returns immediately:

transfer copy_() following synchronize()
D2H 512 MiB 282.31 ms 0.026 ms
H2D 512 MiB 156.92 ms 0.010 ms
D2H 112 MiB 44.26 ms 0.008 ms
D2H 4 MiB 3.88 ms 0.021 ms

A copy into unpinned host memory — and a tmpfs mapping is unpinned — is synchronous, so the stream has already drained by the time we wait on it. On an idle stream the call is 0.7 µs. For a 291-tensor 8B payload that is 582 waits × ~15 µs ≈ 9 ms, about 0.03% of the transfer.

I also implemented the grouped variant to be sure — hold every mapping, issue all copies, one wait at the boundary, then close — on an 8B-shaped payload (291 tensors, 15 GiB):

stage (D2H) read (H2D)
per-tensor wait (this PR) 13.13 / 13.51 s 6.46 / 6.88 s
grouped at the boundary 13.19 / 13.49 s 6.93 / 7.00 s

No gain, because there is nothing in flight to overlap; grouped is marginally slower on the read side across repeats. It would also hold all 291 segments mapped until the boundary. So I kept the per-tensor wait and recorded the measurement in a comment at both sites, so the next reader doesn't have to re-derive it.

Full-model latency

model tensors bytes whole-model sync
Llama-3.2-1B-Instruct 146 2.30 GiB 3.68 / 3.87 s (single call)
Llama-3.1-8B-Instruct 291 14.96 GiB 21.93 / 21.93 s (11 × 1.5 GiB buckets)

Effective 0.6–0.7 GiB/s end to end, which is about what the two copies cost plus the load itself.

One thing worth flagging

The 8B row is bucketed because a single call carrying the whole model OOMs a 24 GiB card, and that is inherent to the fallback rather than a tuning problem: CUDA IPC maps the trainer's memory, but a staged payload is copied, so every tensor in one call is live on the device before any of it is loaded. A whole-model call therefore needs free device memory equal to the model, on top of the weights it is replacing. Trainers that bucket (veRL does) are unaffected. I've documented it in the XPU page next to the /dev/shm sizing, since it is the kind of thing that otherwise shows up as a confusing OOM.

If you would still rather the per-tensor waits go away, the honest route is pinned staging buffers, not grouping — I measured those earlier at not consistently faster, and they double host memory, so I left them out.

engine.update_weights_from_tensor() crashed on Intel XPU, so online RL training
(GRPO/RLHF) could not push weights into a colocated engine after
optimizer.step(). Inference was unaffected.

PyTorch has no XPU tensor IPC: UntypedStorage exposes _share_cuda_ but no
_share_xpu_, and torch.multiprocessing.reductions.reduce_tensor has only cuda and
meta branches (intel/torch-xpu-ops#1678). An XPU tensor fell through to the CPU
storage path and died with "RuntimeError: _share_fd_: only available on CPU"; under
Ray the fd fallback fails instead with "AuthenticationError: digest received was
wrong", because a trainer actor and an mp.spawn-ed scheduler share no authkey.

Stage the bytes through a POSIX shared-memory segment. The producer copies device
to host, the payload carries only the segment name, and each scheduler copies host
to its own device. No fd passing, so no authkey dependency; posix_fallocate turns
an exhausted /dev/shm into ENOSPC instead of SIGBUS.

Segment lifetime follows who can observe the readers. Each scheduler reads
payload[tp_rank], so a payload has one reader per pipeline stage (DP replicas are
rejected for this API). Stage 0's TP0 replies only after its TP group has loaded
(the handler ends on a TP barrier), but other stages may still be loading then,
and they read in no fixed order: a stage holding a request behind
pause_generation relays it onward before dispatching it itself.

  * With one pipeline stage, the engine APIs and HttpServerEngineAdapter
    serialize inside staged_xpu_segments(pp_size=1): the producer owns the
    segments and unlinks them on return, a tensor serialized once per TP rank is
    staged once, and a request rejected before deserializing (no weight-update
    session) leaves nothing behind.
  * Otherwise -- pp_size > 1, or a producer that hands a payload off and returns,
    like weight_sync.utils.update_weights on its non-root ranks -- the readers
    unlink: each segment counts finished reads in an 8-byte trailer under
    flock, and the pp_size-th reader unlinks it, whichever stage that is.

Readers never touch multiprocessing's resource_tracker: they attach untracked and
unlink untracked. A spawned scheduler shares its Engine's tracker, so a reader
that registered and unregistered raced the other readers for one entry (KeyError
tracebacks from the tracker) and could remove the producer's own entry, losing
the reclaim of a scope whose producer is killed.

Before this, every reader unlinked what it read, so with --pp-size 2 stage 1 found
stage 0 had removed the segment and the error escaped its scheduler and killed the
server; rejected requests leaked their staging; and the tp_size payloads of one
call staged tp_size host copies.

Allowlist the rebuild callable in SafeUnpickler and accept only the segment-name
shape make_shm_name("xputensor") produces. The name arrives in the pickle that
POST /update_weights_from_tensor takes, which is ADMIN_OPTIONAL and so
unauthenticated by default, and an unchecked name would read and unlink any
/dev/shm segment the scheduler's uid owns -- the channel SafeUnpickler was
narrowed for in CVE-2025-10164.

Install the XPU reduction at engine startup and on scope entry only: the call
sites install monkey_patch_xpu_tensor_reductions(), not
monkey_patch_torch_reductions(), so CUDA producers keep their payloads unchanged.

Fix a defect the XPU path also tripped over: _reduce_tensor_modified rewrote
argument index 6 into a device uuid on every reduction, but only the CUDA-IPC
payload carries a device there. A CPU tensor reduces to 3 arguments and died with
IndexError; a meta tensor reduces to 7, where index 6 is requires_grad and got the
uuid written over it. Key the rewrite off the rebuild function.

Verified on 8x Arc Pro B60, torch 2.14.0.dev+xpu: weight-update sessions succeed
at TP=1, TP=2 and PP=2, and with PP=2 on Llama-3.1-8B, updating a layer that lives
on stage 1 changes greedy output, so stage 1 loaded it; no segments remain.

Per-tensor waits measured, not grouped: a copy into unpinned host memory has
already blocked in copy_() (512 MiB D2H is 282 ms, the synchronize() after it
0.026 ms), so the waits are 10-25us each -- ~9 ms of a 291-tensor 8B payload.
Grouping them at the payload boundary measured no faster (stage 13.2-13.5 s vs
13.1-13.5 s, read 6.9-7.0 s vs 6.5-6.9 s) and would hold every segment mapped
until the boundary, so both sites keep the per-tensor wait with the measurement
recorded beside it.

Whole-model latency on one Arc Pro B60: Llama-3.2-1B (146 tensors, 2.3 GiB)
syncs in 3.7 s, Llama-3.1-8B (291 tensors, 15 GiB) in 21.9 s over 1.5 GiB
buckets. A staged payload is copied rather than mapped, so every tensor in one
call is live on the device before any is loaded; the XPU doc now says to bucket,
since a single call carrying a whole 8B model needs the model's size free on top
of the weights it replaces and OOMs a 24 GiB card.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants