Skip to content

[Refactor] platforms: route device and stream state through current_platform - #44966

Open
alexnails wants to merge 2 commits into
mainfrom
alexnails/platform-device-ops
Open

alexnails wants to merge 2 commits into
mainfrom
alexnails/platform-device-ops

Conversation

@alexnails

@alexnails alexnails commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

DeviceMixin already declares set_device, synchronize and empty_cache, but they were marked [Planned]: shared srt code called the torch device module directly instead, through torch.get_device_module(...), the srt get_device_module() helper, or a stored device_module. The same was true for current_stream and current_device, which had no platform method at all. As a result, a platform can't own these operations: core always reaches into the device's torch module itself. One site, the Ray scheduler actor, called torch.cuda.set_device directly.

This continues the series that routes core through the platform layer (#44958, #44960, #44961).

Modifications

DeviceMixin

  • New: current_device() and current_stream(device=None).

  • get_device, set_device, synchronize and empty_cache move from [Planned] to [Active].

  • Defaults:

    • get_device returns torch.device(device_type, id).
    • set_device, current_device, current_stream and synchronize call torch's device module for device_type.
    • empty_cache stays a no-op.

    These defaults serve out-of-tree platforms, where they do what the call sites did before (torch.get_device_module(self.device)), rather than hitting the old NotImplementedError (get_device/set_device) or no-op (synchronize).

Per-platform implementations. Every in-tree platform defines these methods itself:

Platform Implementation What was wrong with the default
CUDA, ROCm (CUDA mixin), NPU, XPU their own torch module nothing; now explicit
CPU torch.cpu; current_device() stays "cpu", as torch.cpu returns nothing (qwen3_next uses the value as a tensor device)
MPS current_device() is 0; current_stream() uses torch.accelerator torch.mps has no current_device or stream API, so the default raised AttributeError
MUSA torch.musa for get_device (a musa device), set_device, current_device, current_stream, synchronize, empty_cache it inherited the CUDA mixin and reached MUSA only through torchada's torch.cuda redirect
HPU new minimal HpuSRTPlatform, resolved when habana_frameworks reports an HPU; its defaults go to torch.hpu HPU had no platform class, so it resolved to the base platform with device_type="cpu" and these ops became CPU no-ops

torch.accelerator: the only use in srt was the fallback in the get_device_module() helper, which now returns the module for current_platform.device_type. I didn't make it the general mechanism. torchada documents torch.accelerator dispatch as broken below torch_musa 2.11.0.post2, which is newer than the pinned version. It also patches torch.accelerator.synchronize, because the MUSA implementation syncs a different scope. The two diffusion-kernel torch.accelerator.current_device_index() calls in kernels/ are untouched.

set_device sites now call current_platform.set_device(current_platform.get_device(gpu_id)):

  • distributed/bootstrap.py
  • ModelRunner
  • disaggregation/encoder/server.py
  • disaggregation/nixl/conn.py
  • ray/scheduler_actor.py (was torch.cuda.set_device)
  • AsyncD2HCopyWorker

synchronize / empty_cache / current_stream / current_device: 72 calls across 41 files, rewritten mechanically. They are in the scheduler and its PP mixin, overlap utils, the HiSparse coordinator, the cache controller, the mem-cache pools and linkers, the speculative workers, the graph runners and backends, EPLB, the checkpoint engine, utils/common.py, a few models, and benchmark/one_batch.py.

  • Every call site was already resolving the module of the process's device and calling it without (or with the same) device argument.
  • mem_cache/common.py passed the KV device to synchronize(device); it now synchronizes the current device, which is the same device.

Cleanup:

  • device_module locals and attributes that became unused are removed, along with the _prewarm_hccl_group device_module parameter.
  • Unit tests that faked a device module for code that now calls current_platform patch current_platform instead: dynamic chunk sizer, LoRA overlap loader, DSA layer split, NIXL backend init, and the parallel-runtime bootstrap.
  • docs/docs/hardware-platforms/plugin.mdx lists the full fallback chain.

Not in this PR:

  • hardware_backend/npu and hardware_backend/xpu: device-owned code that keeps using its own torch module.
  • The multimodal_gen platform layer: once [Platform] Unify multimodal_gen platforms onto DeviceMixin #43443 moves it onto DeviceMixin, its call sites follow in a separate PR.
  • Stream, Event and the stream(...) context.
  • Direct torch.cuda.* calls.

Accuracy Tests

N/A: on every in-tree platform, each call resolves to the same torch API as before.

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests.
    • New tests cover:
      • the out-of-tree defaults (device_type module);
      • MPS going through torch.accelerator;
      • MUSA using torch.musa and never torch.cuda;
      • HPU device ops reaching torch.hpu, and HPU platform resolution.
    • The full test/registered/unit tree, run locally (macOS, CPU/MPS) on this branch and on main, fails on the same set of tests apart from environment-dependent cases. The one exception, test_subprocess_watchdog, is flaky and passes on rerun.
    • Every touched module imports, except four that need optional packages absent locally (ray, sgl_kernel, flashinfer, checkpoint-engine); those fail the same way on main.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #39108194818
Latest PR Test (Extra): ❌ Run #39108194954
Latest PR Test (AMD ROCm 10): ❌ Run #39108194896

…latform

Shared srt code called torch.get_device_module(...) (or the srt
get_device_module() helper, or a stored device_module) for set_device,
synchronize, empty_cache, current_stream and current_device, so a platform
could not supply them. These now go through current_platform:

- DeviceMixin gains current_device() and current_stream(device=None), and
  get_device / set_device / synchronize / empty_cache move from [Planned] to
  [Active]. get_device, set_device, current_device, current_stream and
  synchronize default to torch's device module for device_type, so a platform
  that does not override them keeps today's behaviour.
- The six set_device sites (bootstrap, ModelRunner, the encoder server, the
  NIXL connector, the Ray scheduler actor, AsyncD2HCopyWorker) call
  current_platform.set_device(current_platform.get_device(gpu_id)). The Ray
  actor previously called torch.cuda.set_device directly.
- 72 synchronize / empty_cache / current_stream / current_device calls across
  41 files are rewritten mechanically; device_module variables and the
  _prewarm_hccl_group parameter that become unused are removed.

Device-specific code under hardware_backend/ and the multimodal_gen platform
layer are left as they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexnails alexnails added apple-silicon bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci CI: run the baseline test suite on this PR labels Oct 18, 2026
@alexnails

Copy link
Copy Markdown
Collaborator Author

/rerun-failed-ci

@alexnails alexnails added the npu label Oct 18, 2026

@alexnails alexnails left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look over torch.accelerator usage as well

Comment on lines +188 to +190
def current_stream(self, device=None):
"""[Active] The current stream on ``device`` (default: the current device)."""
return torch.get_device_module(self.device_type).current_stream(device)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets evaluate per hardware if this is right call...

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by the per-platform table on the get_device thread (e32c6d1): every in-tree platform now overrides this method explicitly. MPS was the one where the default was wrong, because torch.mps has no current_device or stream API.

Comment on lines +184 to +186
def current_device(self) -> int:
"""[Active] Index of the current device."""
return torch.get_device_module(self.device_type).current_device()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets evaluate per hardware if this is right call...

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by the per-platform table on the get_device thread (e32c6d1): every in-tree platform now overrides this method explicitly. MPS was the one where the default was wrong, because torch.mps has no current_device or stream API.


# ---- Device and stream state ----

def get_device(self, device_id: int = 0) -> "torch.device":

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets evaluate per hardware if these are right call...

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evaluated each backend; fixed in e32c6d1. Every in-tree platform now defines these methods explicitly instead of inheriting the device_type default.

Platform (torch pin) get_device / set_device current_device / current_stream Finding
CUDA 2.14, ROCm 2.11/2.12 torch.cuda (already explicit) torch.cuda (now explicit) The default was already correct.
NPU 2.10 + torch_npu 2.10.0.post6 torch.npu torch.npu The default was already correct.
XPU 2.14+xpu torch.xpu torch.xpu The default was already correct.
CPU 2.14 torch.device("cpu"), torch.cpu torch.cpu; current_device() stays "cpu" qwen3_next passes the value as a tensor device, so returning int 0 would break it.
MPS 2.13 existing MPS overrides 0 / torch.accelerator.current_stream The default was wrong. torch.mps has neither method, so it raised AttributeError. I checked that torch.accelerator returns the MPS stream on 2.13.
MUSA 2.11.0.post1 + torch_musa torch.musa, musa device torch.musa This was wrong. MUSA inherited the CUDA mixin, so it reached the device only through torchada's torch.cuda redirect, and get_device returned a cuda device.
HPU (srt_hpu) defaults over torch.hpu defaults over torch.hpu This was wrong. HPU had no platform class, so it resolved to the base platform with device_type="cpu", and set_device/synchronize became CPU no-ops. I added a minimal HpuSRTPlatform. Its defaults reach torch.hpu, which is what the bare torch.get_device_module() calls did before.

I chose not to make torch.accelerator the general mechanism. torchada documents its dispatch as broken below torch_musa 2.11.0.post2, which is newer than our pin, and it patches torch.accelerator.synchronize because the MUSA implementation syncs a different scope. I use it only for MPS, which has no other stream API.

The DeviceMixin default (torch.get_device_module(device_type)) now serves only out-of-tree platforms. For them it does exactly what these call sites did before (torch.get_device_module(self.device)).

Review follow-up. Instead of relying on the DeviceMixin default that goes
through torch.get_device_module(device_type), each in-tree platform now
states its own current_device / current_stream:

- CUDA (and ROCm through the CUDA mixin), NPU, XPU: their torch module.
- CPU: torch.cpu; current_device() stays "cpu", as torch.cpu returns.
- MPS: torch.mps has no current_device or stream API, so the default
  raised AttributeError. current_device() is 0 and current_stream() uses
  torch.accelerator, which exposes the MPS stream.
- MUSA: get_device / set_device / current_device / current_stream /
  synchronize / empty_cache use torch.musa instead of inheriting the CUDA
  mixin's torch.cuda calls, which only reach MUSA through torchada's
  redirect. get_device returns a musa device.
- HPU had no platform class, so it resolved to the base platform with
  device_type "cpu" and its device ops became CPU no-ops. Add a minimal
  HpuSRTPlatform (resolved when habana_frameworks reports an HPU); its
  device ops are the defaults over torch.hpu.

The srt get_device_module() helper, the last torch.accelerator user in srt,
now returns the module for current_platform.device_type. The plugin docs list
the full fallback chain. Tests that faked a device module for code that now
calls current_platform patch current_platform instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation mthreads labels Oct 18, 2026
return torch.get_device_module(accelerator)
# Named lookup: torch.get_device_module() with no argument initializes the
# CUDA runtime, which poisons fork() startup.
return torch.get_device_module(current_platform.device_type)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we be calling get device module here? I do not think so....

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine for now

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apple-silicon bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) deepseek documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang lora memory-pool mthreads npu run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant