Repository navigation
[Fix] [XPU] weight_sync: stage serialized XPU tensors through host shared memory - #40720
siju-samuel wants to merge 1 commit into
Conversation
0753a34 to
d6367c3
Compare
|
/tag-run-ci-label |
15ba90c to
20766ef
Compare
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
|
@siju-samuel The host shared-memory approach looks reasonable, and the new XPU tests did run and pass. Can you resolve the merge conflicts and get the required CI running on the updated head? One performance question too: staging and reconstruction synchronize once per tensor (producer, consumer). Do you have full-model weight-update latency numbers? It would be good to check whether those waits can be grouped at the payload boundary without changing the buffer-lifetime guarantees. |
838c9ea to
4b06ae0
Compare
|
Thanks for looking. Rebased onto The conflicts were in On the two per-tensor waitsThey are not where the time goes, and grouping them at the payload boundary measured no faster. One Arc Pro B60, torch 2.14.0.dev+xpu. The copy itself blocks, so the
A copy into unpinned host memory — and a tmpfs mapping is unpinned — is synchronous, so the stream has already drained by the time we wait on it. On an idle stream the call is 0.7 µs. For a 291-tensor 8B payload that is 582 waits × ~15 µs ≈ 9 ms, about 0.03% of the transfer. I also implemented the grouped variant to be sure — hold every mapping, issue all copies, one wait at the boundary, then close — on an 8B-shaped payload (291 tensors, 15 GiB):
No gain, because there is nothing in flight to overlap; grouped is marginally slower on the read side across repeats. It would also hold all 291 segments mapped until the boundary. So I kept the per-tensor wait and recorded the measurement in a comment at both sites, so the next reader doesn't have to re-derive it. Full-model latency
Effective 0.6–0.7 GiB/s end to end, which is about what the two copies cost plus the load itself. One thing worth flaggingThe 8B row is bucketed because a single call carrying the whole model OOMs a 24 GiB card, and that is inherent to the fallback rather than a tuning problem: CUDA IPC maps the trainer's memory, but a staged payload is copied, so every tensor in one call is live on the device before any of it is loaded. A whole-model call therefore needs free device memory equal to the model, on top of the weights it is replacing. Trainers that bucket (veRL does) are unaffected. I've documented it in the XPU page next to the If you would still rather the per-tensor waits go away, the honest route is pinned staging buffers, not grouping — I measured those earlier at not consistently faster, and they double host memory, so I left them out. |
engine.update_weights_from_tensor() crashed on Intel XPU, so online RL training (GRPO/RLHF) could not push weights into a colocated engine after optimizer.step(). Inference was unaffected. PyTorch has no XPU tensor IPC: UntypedStorage exposes _share_cuda_ but no _share_xpu_, and torch.multiprocessing.reductions.reduce_tensor has only cuda and meta branches (intel/torch-xpu-ops#1678). An XPU tensor fell through to the CPU storage path and died with "RuntimeError: _share_fd_: only available on CPU"; under Ray the fd fallback fails instead with "AuthenticationError: digest received was wrong", because a trainer actor and an mp.spawn-ed scheduler share no authkey. Stage the bytes through a POSIX shared-memory segment. The producer copies device to host, the payload carries only the segment name, and each scheduler copies host to its own device. No fd passing, so no authkey dependency; posix_fallocate turns an exhausted /dev/shm into ENOSPC instead of SIGBUS. Segment lifetime follows who can observe the readers. Each scheduler reads payload[tp_rank], so a payload has one reader per pipeline stage (DP replicas are rejected for this API). Stage 0's TP0 replies only after its TP group has loaded (the handler ends on a TP barrier), but other stages may still be loading then, and they read in no fixed order: a stage holding a request behind pause_generation relays it onward before dispatching it itself. * With one pipeline stage, the engine APIs and HttpServerEngineAdapter serialize inside staged_xpu_segments(pp_size=1): the producer owns the segments and unlinks them on return, a tensor serialized once per TP rank is staged once, and a request rejected before deserializing (no weight-update session) leaves nothing behind. * Otherwise -- pp_size > 1, or a producer that hands a payload off and returns, like weight_sync.utils.update_weights on its non-root ranks -- the readers unlink: each segment counts finished reads in an 8-byte trailer under flock, and the pp_size-th reader unlinks it, whichever stage that is. Readers never touch multiprocessing's resource_tracker: they attach untracked and unlink untracked. A spawned scheduler shares its Engine's tracker, so a reader that registered and unregistered raced the other readers for one entry (KeyError tracebacks from the tracker) and could remove the producer's own entry, losing the reclaim of a scope whose producer is killed. Before this, every reader unlinked what it read, so with --pp-size 2 stage 1 found stage 0 had removed the segment and the error escaped its scheduler and killed the server; rejected requests leaked their staging; and the tp_size payloads of one call staged tp_size host copies. Allowlist the rebuild callable in SafeUnpickler and accept only the segment-name shape make_shm_name("xputensor") produces. The name arrives in the pickle that POST /update_weights_from_tensor takes, which is ADMIN_OPTIONAL and so unauthenticated by default, and an unchecked name would read and unlink any /dev/shm segment the scheduler's uid owns -- the channel SafeUnpickler was narrowed for in CVE-2025-10164. Install the XPU reduction at engine startup and on scope entry only: the call sites install monkey_patch_xpu_tensor_reductions(), not monkey_patch_torch_reductions(), so CUDA producers keep their payloads unchanged. Fix a defect the XPU path also tripped over: _reduce_tensor_modified rewrote argument index 6 into a device uuid on every reduction, but only the CUDA-IPC payload carries a device there. A CPU tensor reduces to 3 arguments and died with IndexError; a meta tensor reduces to 7, where index 6 is requires_grad and got the uuid written over it. Key the rewrite off the rebuild function. Verified on 8x Arc Pro B60, torch 2.14.0.dev+xpu: weight-update sessions succeed at TP=1, TP=2 and PP=2, and with PP=2 on Llama-3.1-8B, updating a layer that lives on stage 1 changes greedy output, so stage 1 loaded it; no segments remain. Per-tensor waits measured, not grouped: a copy into unpinned host memory has already blocked in copy_() (512 MiB D2H is 282 ms, the synchronize() after it 0.026 ms), so the waits are 10-25us each -- ~9 ms of a 291-tensor 8B payload. Grouping them at the payload boundary measured no faster (stage 13.2-13.5 s vs 13.1-13.5 s, read 6.9-7.0 s vs 6.5-6.9 s) and would hold every segment mapped until the boundary, so both sites keep the per-tensor wait with the measurement recorded beside it. Whole-model latency on one Arc Pro B60: Llama-3.2-1B (146 tensors, 2.3 GiB) syncs in 3.7 s, Llama-3.1-8B (291 tensors, 15 GiB) in 21.9 s over 1.5 GiB buckets. A staged payload is copied rather than mapped, so every tensor in one call is live on the device before any is loaded; the XPU doc now says to bucket, since a single call carrying a whole 8B model needs the model's size free on top of the weights it replaces and OOMs a 24 GiB card.
4b06ae0 to
ba3873b
Compare
Motivation
Engine.update_weights_from_tensor()crashes on Intel XPU, so online RL training (GRPO/RLHF through verl and similar trainers) cannot push weights into a colocated engine after eachoptimizer.step(). Inference is unaffected.PyTorch has no XPU tensor IPC:
UntypedStoragehas_share_cuda_but no_share_xpu_, andtorch.multiprocessing.reductions.reduce_tensorhas only CUDA and meta branches (intel/torch-xpu-ops#1678). An XPU tensor falls through to the CPU storage path:Under Ray it fails instead with
AuthenticationError: digest received was wrong, because fd passing needs an authkey that a trainer actor and anmp.spawn-ed scheduler never share.Modifications
Transport (
python/sglang/srt/utils/xpu_tensor_shm.py). Each serialized XPU tensor is staged through a POSIX shared-memory segment. The producer copies device → host, the payload carries only the segment name (174 B regardless of tensor size), and each scheduler copies host → its own device. There is no fd passing, so no authkey dependency.posix_fallocateturns an exhausted/dev/shmintoENOSPCinstead ofSIGBUS.Segment lifetime follows who can observe the readers. Each scheduler reads
payload[tp_rank], so a payload has one reader per pipeline stage (DP replicas are already rejected for this API,tokenizer_control_mixin.py:565). Stage 0's TP0 replies only after its TP group has loaded:SchedulerWeightUpdaterManager.update_weights_from_tensorends on a TP barrier. Other stages may still be loading at that point, and they read in no fixed order: a stage holding a request behindpause_generationrelays it onward before dispatching it itself (ingest_requestsreturns the held requests for relay, since #48992). So neither the producer nor any single stage can tell it is the last reader.Engine.update_weights_from_tensor,load_lora_adapter_from_tensors,HttpServerEngineAdapter, PP = 1staged_xpu_segments(pp_size=1))pp_size-th reader, whichever stage that istp_sizeweight_sync.utils.update_weightson its non-root ranks)pp_size-th readerReader-owned segments count finished reads in an 8-byte trailer, incremented under
flock; the reader that brings the count topp_sizeunlinks. Readers never touchmultiprocessing's resource tracker — a spawned scheduler shares its Engine's tracker, so a reader that registered and unregistered would race the other readers for one entry and could remove the producer's own, losing the reclaim of a scope whose producer is killed.Security.
_rebuild_xpu_tensor_from_shmis allowlisted inSafeUnpickler.ALLOWED_GLOBALS, and it accepts only the segment-name shapemake_shm_name("xputensor")produces. The name arrives in the pickle thatPOST /update_weights_from_tensortakes. That route isADMIN_OPTIONAL, so it's unauthenticated by default, and an unchecked name would read and unlink any/dev/shmsegment the scheduler's uid owns. That is the channelSafeUnpicklerwas narrowed for in CVE-2025-10164.Patch installation.
monkey_patch_xpu_tensor_reductions()is installed at engine startup (_set_envs_and_config) and on scope entry. The new call sites install only that patch, notmonkey_patch_torch_reductions(), so CUDA producers' payloads are unchanged._reduce_tensor_modifiedfix (patch_torch.py). It rewrote argument index 6 into a device UUID on every reduction, but only the CUDA-IPC payload carries a device there. A CPU tensor reduces to 3 arguments and died withIndexError. A meta tensor reduces to 7, where index 6 isrequires_grad, so the UUID was written over it. The rewrite is now keyed off the rebuild function.Docs.
docs/docs/hardware-platforms/xpu.mdxgains an RL weight-sync section covering/dev/shmsizing, cost, what callers that serialize their own payloads must do, and how to clear a leak.Notes for review
--pp-size 2works. That alone is not safe: the call returns when stage 0 replies, and with the producer unlinking on return, stage 1 found its segment gone and crashed the server. It reproduced deterministically on hardware. A barrier inside the handler can't fix it either, because stage 0 relays the request only after its handler returns. An earlier revision used a "last pipeline stage unlinks" rule instead, which relied on stages reading in order; [Fix] scheduler: Honor pause_generation under PP and release aborted PD rooms and rejected sessions #48992 made a stage relay requests held behindpause_generationbefore dispatching them, so stage 1 can read first. Counting reads is independent of order.staged_xpu_segmentsare no-ops off XPU. The only shared-path change is the_reduce_tensor_modifiedshort-circuit for non-CUDA-IPC payloads. Those payloads previously crashed (CPU) or were corrupted (meta), and the CUDA-IPC rewrite itself is untouched; the existing 2-GPU CUDA case intest_patch_torch.pystill covers it.Known limitations
rm /dev/shm/sgl_shm_xputensor_*as the remedy.Accuracy Tests
Not applicable: this is a transport-only change. Weights arrive bit-exact; the unit tests compare raw bytes across bf16, fp8, int64, 0-d, empty and transposed tensors.
Speed Tests and Profiling
Measured on Arc Pro B60, torch 2.14.0.dev+xpu:
update_weights_from_tensorfor one 32 MiBup_projon Llama-3.2-1B takes 0.041–0.060 s at TP=1, TP=2 and PP=2.tp_size.Tests
test/registered/utils/test_xpu_tensor_shm.py(registered forstage-b-test-1-gpu-xpu) guards one failure mode per case:Parameter);test/registered/rl/test_patch_torch.pyadds CPU and meta tensors through the patched reducer, includingrequires_grad=Trueleaves.End-to-end on 8× Arc Pro B60, inside a
begin_weight_update()/end_weight_update()session:(True, 'Success'), readback 1.5, generation OK, 0 segments left(True, 'Success'). Zeroinglayers.31.down_proj, which lives on stage 1, changes greedy output, so stage 1 loaded it; 0 segments leftpause_generationwith a 1500-token generation in flightWeights were updated, greedy output changes, the paused generation completes aftercontinue_generation, 0 segments left. The call itself returnssuccess=Falseonly becausemainrefuses the default cache flush while the paused request still holds KV (Weights were updated, but the cache was not flushed because requests are still running); passflush_cache=Falsewhen updating under an in-place pauseRe-run on
ba3873b(rebased onto3981de0): TP=1, TP=2 and PP=2 all(True, 'Success')with the last layer's update visible in greedy output, 0 segments left, and noKeyErrorfrommultiprocessing's resource tracker in any log — where two concurrent readers on the previous head produced one. Whether a stage actually held the request in the paused run is timing-dependent and not visible in the logs; the order independence itself is pinned deterministically by the unit test.The previous head of this PR, on the PP=2 case, returned
(True, 'Success')and then crashed stage 1 withShared-memory segment ... is gone.PP=2 on Llama-3.2-1B crashes during generation with
'PPMissingLayer' object has no attribute 'quant_method'. That also happens on unmodifiedmainwith no weight update: tied embeddings putlm_headon stage 0. It is unrelated to this PR, which is why the PP=2 row uses the 8B model.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #43805005497
Latest PR Test (Extra): ❌ Run #43805005431
Latest PR Test (AMD ROCm 10): ❌ Run #43805005391
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: