Skip to content

[Fix] disaggregation: Gate the decode retraction host-pool default on host-memory support - #46362

Draft
alexnails wants to merge 1 commit into
sgl-project:mainfrom
alexnails:fix/disagg-retraction-backup-platform-default
Draft

alexnails wants to merge 1 commit into
sgl-project:mainfrom
alexnails:fix/disagg-retraction-backup-platform-default

Conversation

@alexnails

@alexnails alexnails commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

When --disaggregation-decode-retraction-backup is unset, a PD decode server resolves it (kv_cache_builder.resolve_decode_retraction_backup) to host_pool whenever the KV pool supports it. That builds a HiCache host pool. The host pool picks its allocator from pool_host.common.ALLOC_MEMORY_FUNCS by device type. Only npu, musa and xpu have entries, and every other device falls back to alloc_with_host_register, i.e. cudaHostRegister. On a device that is neither CUDA nor ROCm the server therefore crashes at startup, just because the user didn't pass a flag. That includes CPU and out-of-tree platforms such as Neuron/Trainium.

Modifications

  • New kv_cache_builder.device_can_back_host_pool(device). It is True when the device type has an explicit host allocator in ALLOC_MEMORY_FUNCS, or when it is a CUDA/ROCm device. It checks keys with in, so it doesn't insert defaultdict entries.
  • resolve_decode_retraction_backup only defaults to host_pool when that is true. Otherwise it defaults to cpu_tensor, the existing fallback used for HIP, DCP, recurrent-state models, etc.
  • An explicit --disaggregation-decode-retraction-backup host_pool is unchanged and still honored.

How it was found

Found bringing up PD on an out-of-tree Neuron/Trainium platform plugin. The decode server crashed in cudaHostRegister while building the backup-only host pool. The plugin currently works around it by setting disaggregation_decode_retraction_backup = "cpu_tensor" in its apply_server_args_defaults.

Accuracy Tests

N/A. There is no change on CUDA/NPU/MUSA/XPU.

Speed Tests and Profiling

N/A.

Tests

  • Run (macOS, CPU only): test/registered/unit/mem_cache/test_retraction_backup_platform_default.py (new). It checks that devices with a host allocator return True, that cuda returns True only on CUDA/ROCm builds, that CPU and unknown out-of-tree devices return False, and that the check doesn't mutate ALLOC_MEMORY_FUNCS.
  • Not run: test_decode_retraction_backup.py is a CUDA test and cannot run here. No GPU PD e2e. The full resolve_decode_retraction_backup was not exercised end to end on CPU.

Open questions

  • Should this be a platform hook instead, e.g. SRTPlatform.supports_host_memory_registration() or a platform-provided host allocator? Then an out-of-tree platform could opt in to host_pool with its own pinned allocator, instead of inserting into ALLOC_MEMORY_FUNCS. I kept the device-type check because it matches how the allocator is chosen today.
  • HiCache itself (--enable-hierarchical-cache) on such devices has the same cudaHostRegister fallback. Not addressed here.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #40158956661
Latest PR Test (Extra): ❌ Run #40158956654
Latest PR Test (AMD ROCm 10): ❌ Run #40158956600

… host-memory support

When --disaggregation-decode-retraction-backup is unset, a PD decode server
resolves it to host_pool, which builds a HiCache host pool. The pool's
allocator comes from ALLOC_MEMORY_FUNCS keyed by device type, and every
device without an entry falls back to cudaHostRegister, which crashes at
startup on CPU and on out-of-tree devices.

Only default to host_pool when the device has a host allocator (npu, musa,
xpu) or is a CUDA/ROCm device; otherwise default to cpu_tensor. An explicit
flag is still honored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant