Repository navigation
Conversation
…latform Shared srt code called torch.get_device_module(...) (or the srt get_device_module() helper, or a stored device_module) for set_device, synchronize, empty_cache, current_stream and current_device, so a platform could not supply them. These now go through current_platform: - DeviceMixin gains current_device() and current_stream(device=None), and get_device / set_device / synchronize / empty_cache move from [Planned] to [Active]. get_device, set_device, current_device, current_stream and synchronize default to torch's device module for device_type, so a platform that does not override them keeps today's behaviour. - The six set_device sites (bootstrap, ModelRunner, the encoder server, the NIXL connector, the Ray scheduler actor, AsyncD2HCopyWorker) call current_platform.set_device(current_platform.get_device(gpu_id)). The Ray actor previously called torch.cuda.set_device directly. - 72 synchronize / empty_cache / current_stream / current_device calls across 41 files are rewritten mechanically; device_module variables and the _prewarm_hccl_group parameter that become unused are removed. Device-specific code under hardware_backend/ and the multimodal_gen platform layer are left as they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/rerun-failed-ci |
alexnails
left a comment
There was a problem hiding this comment.
look over torch.accelerator usage as well
| def current_stream(self, device=None): | ||
| """[Active] The current stream on ``device`` (default: the current device).""" | ||
| return torch.get_device_module(self.device_type).current_stream(device) |
There was a problem hiding this comment.
lets evaluate per hardware if this is right call...
There was a problem hiding this comment.
Covered by the per-platform table on the get_device thread (e32c6d1): every in-tree platform now overrides this method explicitly. MPS was the one where the default was wrong, because torch.mps has no current_device or stream API.
| def current_device(self) -> int: | ||
| """[Active] Index of the current device.""" | ||
| return torch.get_device_module(self.device_type).current_device() |
There was a problem hiding this comment.
lets evaluate per hardware if this is right call...
There was a problem hiding this comment.
Covered by the per-platform table on the get_device thread (e32c6d1): every in-tree platform now overrides this method explicitly. MPS was the one where the default was wrong, because torch.mps has no current_device or stream API.
|
|
||
| # ---- Device and stream state ---- | ||
|
|
||
| def get_device(self, device_id: int = 0) -> "torch.device": |
There was a problem hiding this comment.
lets evaluate per hardware if these are right call...
There was a problem hiding this comment.
Evaluated each backend; fixed in e32c6d1. Every in-tree platform now defines these methods explicitly instead of inheriting the device_type default.
| Platform (torch pin) | get_device / set_device | current_device / current_stream | Finding |
|---|---|---|---|
| CUDA 2.14, ROCm 2.11/2.12 | torch.cuda (already explicit) |
torch.cuda (now explicit) |
The default was already correct. |
| NPU 2.10 + torch_npu 2.10.0.post6 | torch.npu |
torch.npu |
The default was already correct. |
| XPU 2.14+xpu | torch.xpu |
torch.xpu |
The default was already correct. |
| CPU 2.14 | torch.device("cpu"), torch.cpu |
torch.cpu; current_device() stays "cpu" |
qwen3_next passes the value as a tensor device, so returning int 0 would break it. |
| MPS 2.13 | existing MPS overrides | 0 / torch.accelerator.current_stream |
The default was wrong. torch.mps has neither method, so it raised AttributeError. I checked that torch.accelerator returns the MPS stream on 2.13. |
| MUSA 2.11.0.post1 + torch_musa | torch.musa, musa device |
torch.musa |
This was wrong. MUSA inherited the CUDA mixin, so it reached the device only through torchada's torch.cuda redirect, and get_device returned a cuda device. |
HPU (srt_hpu) |
defaults over torch.hpu |
defaults over torch.hpu |
This was wrong. HPU had no platform class, so it resolved to the base platform with device_type="cpu", and set_device/synchronize became CPU no-ops. I added a minimal HpuSRTPlatform. Its defaults reach torch.hpu, which is what the bare torch.get_device_module() calls did before. |
I chose not to make torch.accelerator the general mechanism. torchada documents its dispatch as broken below torch_musa 2.11.0.post2, which is newer than our pin, and it patches torch.accelerator.synchronize because the MUSA implementation syncs a different scope. I use it only for MPS, which has no other stream API.
The DeviceMixin default (torch.get_device_module(device_type)) now serves only out-of-tree platforms. For them it does exactly what these call sites did before (torch.get_device_module(self.device)).
Review follow-up. Instead of relying on the DeviceMixin default that goes through torch.get_device_module(device_type), each in-tree platform now states its own current_device / current_stream: - CUDA (and ROCm through the CUDA mixin), NPU, XPU: their torch module. - CPU: torch.cpu; current_device() stays "cpu", as torch.cpu returns. - MPS: torch.mps has no current_device or stream API, so the default raised AttributeError. current_device() is 0 and current_stream() uses torch.accelerator, which exposes the MPS stream. - MUSA: get_device / set_device / current_device / current_stream / synchronize / empty_cache use torch.musa instead of inheriting the CUDA mixin's torch.cuda calls, which only reach MUSA through torchada's redirect. get_device returns a musa device. - HPU had no platform class, so it resolved to the base platform with device_type "cpu" and its device ops became CPU no-ops. Add a minimal HpuSRTPlatform (resolved when habana_frameworks reports an HPU); its device ops are the defaults over torch.hpu. The srt get_device_module() helper, the last torch.accelerator user in srt, now returns the module for current_platform.device_type. The plugin docs list the full fallback chain. Tests that faked a device module for code that now calls current_platform patch current_platform instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| return torch.get_device_module(accelerator) | ||
| # Named lookup: torch.get_device_module() with no argument initializes the | ||
| # CUDA runtime, which poisons fork() startup. | ||
| return torch.get_device_module(current_platform.device_type) |
There was a problem hiding this comment.
should we be calling get device module here? I do not think so....
Motivation
DeviceMixinalready declaresset_device,synchronizeandempty_cache, but they were marked[Planned]: shared srt code called the torch device module directly instead, throughtorch.get_device_module(...), the srtget_device_module()helper, or a storeddevice_module. The same was true forcurrent_streamandcurrent_device, which had no platform method at all. As a result, a platform can't own these operations: core always reaches into the device's torch module itself. One site, the Ray scheduler actor, calledtorch.cuda.set_devicedirectly.This continues the series that routes core through the platform layer (#44958, #44960, #44961).
Modifications
DeviceMixinNew:
current_device()andcurrent_stream(device=None).get_device,set_device,synchronizeandempty_cachemove from[Planned]to[Active].Defaults:
get_devicereturnstorch.device(device_type, id).set_device,current_device,current_streamandsynchronizecall torch's device module fordevice_type.empty_cachestays a no-op.These defaults serve out-of-tree platforms, where they do what the call sites did before (
torch.get_device_module(self.device)), rather than hitting the oldNotImplementedError(get_device/set_device) or no-op (synchronize).Per-platform implementations. Every in-tree platform defines these methods itself:
torch.cpu;current_device()stays"cpu", astorch.cpureturnsqwen3_nextuses the value as a tensor device)current_device()is 0;current_stream()usestorch.acceleratortorch.mpshas nocurrent_deviceor stream API, so the default raisedAttributeErrortorch.musaforget_device(amusadevice),set_device,current_device,current_stream,synchronize,empty_cachetorch.cudaredirectHpuSRTPlatform, resolved whenhabana_frameworksreports an HPU; its defaults go totorch.hpudevice_type="cpu"and these ops became CPU no-opstorch.accelerator: the only use in srt was the fallback in theget_device_module()helper, which now returns the module forcurrent_platform.device_type. I didn't make it the general mechanism. torchada documentstorch.acceleratordispatch as broken below torch_musa 2.11.0.post2, which is newer than the pinned version. It also patchestorch.accelerator.synchronize, because the MUSA implementation syncs a different scope. The two diffusion-kerneltorch.accelerator.current_device_index()calls inkernels/are untouched.set_devicesites now callcurrent_platform.set_device(current_platform.get_device(gpu_id)):distributed/bootstrap.pyModelRunnerdisaggregation/encoder/server.pydisaggregation/nixl/conn.pyray/scheduler_actor.py(wastorch.cuda.set_device)AsyncD2HCopyWorkersynchronize/empty_cache/current_stream/current_device: 72 calls across 41 files, rewritten mechanically. They are in the scheduler and its PP mixin, overlap utils, the HiSparse coordinator, the cache controller, the mem-cache pools and linkers, the speculative workers, the graph runners and backends, EPLB, the checkpoint engine,utils/common.py, a few models, andbenchmark/one_batch.py.mem_cache/common.pypassed the KV device tosynchronize(device); it now synchronizes the current device, which is the same device.Cleanup:
device_modulelocals and attributes that became unused are removed, along with the_prewarm_hccl_groupdevice_moduleparameter.current_platformpatchcurrent_platforminstead: dynamic chunk sizer, LoRA overlap loader, DSA layer split, NIXL backend init, and the parallel-runtime bootstrap.docs/docs/hardware-platforms/plugin.mdxlists the full fallback chain.Not in this PR:
hardware_backend/npuandhardware_backend/xpu: device-owned code that keeps using its own torch module.DeviceMixin, its call sites follow in a separate PR.Stream,Eventand thestream(...)context.torch.cuda.*calls.Accuracy Tests
N/A: on every in-tree platform, each call resolves to the same torch API as before.
Checklist
device_typemodule);torch.accelerator;torch.musaand nevertorch.cuda;torch.hpu, and HPU platform resolution.test/registered/unittree, run locally (macOS, CPU/MPS) on this branch and onmain, fails on the same set of tests apart from environment-dependent cases. The one exception,test_subprocess_watchdog, is flaky and passes on rerun.ray,sgl_kernel,flashinfer,checkpoint-engine); those fail the same way onmain.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #39108194818
Latest PR Test (Extra): ❌ Run #39108194954
Latest PR Test (AMD ROCm 10): ❌ Run #39108194896