Repository navigation
Conversation
… host-memory support When --disaggregation-decode-retraction-backup is unset, a PD decode server resolves it to host_pool, which builds a HiCache host pool. The pool's allocator comes from ALLOC_MEMORY_FUNCS keyed by device type, and every device without an entry falls back to cudaHostRegister, which crashes at startup on CPU and on out-of-tree devices. Only default to host_pool when the device has a host allocator (npu, musa, xpu) or is a CUDA/ROCm device; otherwise default to cpu_tensor. An explicit flag is still honored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 28, 2026
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When
--disaggregation-decode-retraction-backupis unset, a PD decode server resolves it (kv_cache_builder.resolve_decode_retraction_backup) tohost_poolwhenever the KV pool supports it. That builds a HiCache host pool. The host pool picks its allocator frompool_host.common.ALLOC_MEMORY_FUNCSby device type. Onlynpu,musaandxpuhave entries, and every other device falls back toalloc_with_host_register, i.e.cudaHostRegister. On a device that is neither CUDA nor ROCm the server therefore crashes at startup, just because the user didn't pass a flag. That includes CPU and out-of-tree platforms such as Neuron/Trainium.Modifications
kv_cache_builder.device_can_back_host_pool(device). It is True when the device type has an explicit host allocator inALLOC_MEMORY_FUNCS, or when it is a CUDA/ROCm device. It checks keys within, so it doesn't insert defaultdict entries.resolve_decode_retraction_backuponly defaults tohost_poolwhen that is true. Otherwise it defaults tocpu_tensor, the existing fallback used for HIP, DCP, recurrent-state models, etc.--disaggregation-decode-retraction-backup host_poolis unchanged and still honored.How it was found
Found bringing up PD on an out-of-tree Neuron/Trainium platform plugin. The decode server crashed in
cudaHostRegisterwhile building the backup-only host pool. The plugin currently works around it by settingdisaggregation_decode_retraction_backup = "cpu_tensor"in itsapply_server_args_defaults.Accuracy Tests
N/A. There is no change on CUDA/NPU/MUSA/XPU.
Speed Tests and Profiling
N/A.
Tests
test/registered/unit/mem_cache/test_retraction_backup_platform_default.py(new). It checks that devices with a host allocator return True, thatcudareturns True only on CUDA/ROCm builds, that CPU and unknown out-of-tree devices return False, and that the check doesn't mutateALLOC_MEMORY_FUNCS.test_decode_retraction_backup.pyis a CUDA test and cannot run here. No GPU PD e2e. The fullresolve_decode_retraction_backupwas not exercised end to end on CPU.Open questions
SRTPlatform.supports_host_memory_registration()or a platform-provided host allocator? Then an out-of-tree platform could opt in tohost_poolwith its own pinned allocator, instead of inserting intoALLOC_MEMORY_FUNCS. I kept the device-type check because it matches how the allocator is chosen today.--enable-hierarchical-cache) on such devices has the samecudaHostRegisterfallback. Not addressed here.Checklist
Review and Merge Process
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #40158956661
Latest PR Test (Extra): ❌ Run #40158956654
Latest PR Test (AMD ROCm 10): ❌ Run #40158956600