Repository navigation
Typed framework memory regions, same-process device copies, and batched peer writes for the SGLang direct linker - #39
ishandhanani wants to merge 15 commits into
Conversation
|
Consumer PR: sgl-project/sglang#39946 (KVCR direct external linker). Dynamo companion: ai-dynamo/dynamo#14992. |
Add FrameworkMemoryRegion and KVCRBackendConfigs.framework_regions so a framework can register its actual allocations (DRAM or VRAM, per device) as NIXL transfer endpoints. Registration runs on the progress thread, one NIXL call per (mem_type, device_id) group, and agent metadata is captured only after every group registered. Exact duplicate views collapse to one registration, partially overlapping regions of one endpoint are refused, and configuring both framework_dram and framework_regions is refused. The core retains region owners until close() releases the registration. Local DRAM deposit and deliver batch copies per (source, destination) endpoint so a NIXL descriptor list never mixes memory types, and a block whose spans disagree on memory type or device is rejected up front. Tests cover grouping, validation, owner lifetime, startup registration failure cleanup, mixed-endpoint batching, close with a pending copy, and a real NIXL UCX GPU <-> DRAM deposit/deliver round trip.
NIXL routes a transfer between two endpoints of the same agent through its network transport, and UCX has no device-memory lane for that loopback: puts between VRAM and host fall back to software emulation over TCP at 0.3-0.7 GB/s on an H100 host (PCIe memcpy: 38 GB/s). Local DRAM copies with a VRAM endpoint now go through cudaMemcpyBatchAsync on KVCR-owned streams, split across up to four streams, and complete through recorded events. The engine binds libcudart with ctypes, validates VRAM spans against registered framework regions with numpy, and falls back to NIXL when the runtime is unavailable or LocalDramOptions(device_copy=False) is set. Per-span Python work is removed from the hot path: MemDescriptor is a gc-free msgspec Struct (6x cheaper to build), descriptor normalisation uses C-level maps with cached layout sizes, and device copies express KVCR slots as address arrays instead of materialised descriptors. Measured deposit/deliver of 46 pages x 56 spans x 128 KiB (338 MB): A400 11.6 GB/s (memcpy 13.4), H100 17-20 GB/s (memcpy 39), versus 0.4-0.9 GB/s through NIXL loopback on both.
The progress thread slept 1 ms between iterations whenever an iteration observed no work, so a device copy that lands in a few milliseconds was detected up to 1 ms late. Sleep 100 us while any op is in flight and keep the 1 ms idle wait otherwise.
Each cudart call takes microseconds; binding libcudart through PyDLL keeps the GIL across them instead of paying a switch interval to reacquire it whenever another Python thread is busy, which is the normal state of a serving process. Slot allocation under eviction recomputed the shortfall over every required pool twice per candidate. The shortfall is now tracked only for the pools still short, the per-layout slot counts are cached, and block byte counts come from C-level maps, so a page-sized eviction is one pass.
Peer write notifications can only arrive through peers reached over the framework control channel. Without one, the per-iteration get_new_notifs call held the GIL for about a tenth of the progress thread's time for nothing.
The batch copy's driver-side validation grows with the span count, so that one call goes through a releasing CDLL handle; every other runtime call stays on the PyDLL handle and keeps the GIL for its few microseconds.
…ives With nothing in flight the progress thread slept 1 ms between iterations, so it woke a thousand times a second in the framework process and each wake was a GIL hand-off on the scheduler thread; on a four-rank DP-attention deployment that class of wake-up cost every request tens of milliseconds. Submissions now wake the loop through a pipe, the peer control channel offers a wait that polls its socket and that pipe together, and the loop parks for at most 20 ms otherwise. The 0.1 ms cadence while operations are in flight is unchanged.
A framework with several physical pools per page (DeepSeek V4 has six) had to issue one fetch per pool, so one restored page range cost one control message, one peer write, and one notification per pool. fetch now takes expected_layouts mapping keys to their own layouts; keys without an entry keep the operation's expected_layout. Slot reservation, layout checks, and capacity waiters use the key's layout.
Descriptor lists are built per span so layout checks stay exact, but NIXL request creation and posting cost per descriptor: a DeepSeek V4 page has 147 spans, so one 11-page fetch posted 1617 descriptors and the source spent 18 ms per operation in transfer_submit. Aligned source and destination spans that are contiguous on both sides (a page's spans in a KVCR slot are) are now merged up to 512 KiB, which keeps enough pieces for UCX to spread the copy over its workers.
cudaMemcpyBatchAsync costs host time per span (about 10 us each, so a 590-span DeepSeek V4 restore batch spent 7.5 ms in submission for 1 ms of DMA). Pages allocated in sequence have adjacent device rows and adjacent KVCR slots, so consecutive operands contiguous in both address spaces are merged with one vectorised pass before the request is built; a merged run copies the same bytes with fewer, larger spans.
Merged descriptors above 128 KiB are copied inline by the posting thread instead of asynchronously across UCX workers, which made a dense model's 338 MB fetch slower (89-101 ms against 78-95 ms unmerged). Spans at or above the threshold are left as they are; only smaller spans are combined.
… spans async Spans above UCX's 128 KiB asynchronous put size are copied inline by the posting thread whatever their size, so runs of them (DeepSeek V4's 146 KB sliding-window spans, 43 per page) now merge up to 512 KiB, which measured 24-32 ms per 100 MB fetch against 40-46 ms unmerged; runs of spans at or below 128 KiB still never grow past it, so a dense model's 128 KiB spans keep their asynchronous multi-worker path.
Each stream costs a batch call and an event, about 0.35 ms of host time together, so a 19 MB restore split over four streams spent more in submission than in DMA. Streams are now bounded by bytes as well as by span count (one per 16 MiB), which keeps large copies parallel.
c705a8a to
450885b
Compare
NIXL registration of the framework's pools scales with their size; on a B200 node with 46 GB of KV per rank plus a 32 GB DRAM tier the 30 s default is marginal when the pools go through an RDMA NIC. Keep the default, allow the deployment to raise it.
|
I maintain kvcached's SGLang integration, including the pool and allocator compat layer for sglang 0.5.16 through 0.5.20 (ovg-project/kvcached#514, #516, #525) and the zero-attention-layer pool handling that hybrid models need (ovg-project/kvcached#524). The direct linker calls into those same pool classes. Rather than start anything parallel, I would like to help this lane land. Concretely, I can review and validate this PR and #44 on a GPU host, with an eye on compatibility across the recent sglang pool-class revisions. A validation plan for the resiliency side would be the other piece I can take, exercising the linker path across Guard promotion and worker replacement. If a short design or test-plan doc would help, I will write one. |
|
Hi Rishabh @rishabhsinha17, sorry just saw this. This was an initial PR to try things out, now the main covers what's needed and I wasn't paying attention. Glad to hear your interest and it would be great to collaborate. As of now this branch has the current ongoing linker integration, but needs some final clean ups: https://github.com/linhu-nv/sglang/tree/linhu/kvcr-tp-dedup-v4 @linhu-nv and @ziqifan617 are the main contacts for SGLang integration, so that would be great if you get connected, and then plan the collaboration. (I'm going to close this PR with the next comment). |
|
Closing this initial integration PR. Typed and strided framework registration, pool/index references, and labelled partial delivery are now covered by #59, #60, and #61. Configurable startup timeout landed in #82, with subsequent PRs addressing metadata caching, msgspec records, and NIXL transfer preparation. The separate CUDA copy engine has not been required for the current SGLang integration. If profiling demonstrates a remaining local-copy bottleneck, we can address it in NIXL or a focused KVCR change. The additional per-key fetch layouts, span coalescing, and progress-loop tuning remain deferred and can be revisited independently when a concrete workload justifies them. |
|
Thanks. I read linhu/kvcr-tp-dedup-v4 at 561eccf and #59 to #61 and #82. The slice I would own is elastic pools, which reserve virtual memory and back it later. KVCRLayout.init in storage/kvcr/layout.py registers every K and V buffer with |
Why
SGLang's direct external linker (ishandhanani/sglang#10) moves KV pages straight between GPU KV pools and KVCR-owned DRAM and restores prefixes from a peer worker's tier over NIXL. Two things stood in its way in KVCR:
framework_dram), so GPU pools could not be transfer endpoints at all.Everything here is additive.
framework_dramkeeps working, the public API only gains optional arguments, and no wire format changes.What changes
Registration (
770bef8)FrameworkMemoryRegion(address, length, mem_type, device_id, owner=None)andKVCRBackendConfigs.framework_regions.mem_typeis the NIXL segment type (DRAMorVRAM),device_idthe device index. One NIXLregister_memoryper(mem_type, device_id)group on the progress thread; agent metadata is captured after every group registered; a failing group raises and earlier groups are deregistered.framework_dramandframework_regionsis refused.ownerreferences are held untilclose().Same-process device copies (
fd37a91,4a84d1f,97f4b35,0c8d2bc,f19582c,450885b)device_copy.py: a local DRAM fill or delivery with a VRAM endpoint is issued ascudaMemcpyBatchAsyncon KVCR-owned streams and completes through recorded events. The CUDA runtime is bound through ctypes, so KVCR gains no framework dependency;LocalDramOptions.device_copy=Falsekeeps the NIXL path for diagnostics.Progress thread (
d507a4a,9e91430,266bd91)submit) plus the peer control socket (ZmqPeerControlChannel.wait), bounded at 20 ms, instead of waking every millisecond; in a framework process every wake is a GIL hand-off on the scheduler thread. NIXL notification polling is skipped when no control channel exists.Peer writes (
58421c6,7911692,1355b6b,e304c9b)fetch(..., expected_layouts={key: layout}): a layout per key, so one operation, one control message, one NIXL WRITE and one notification cover a page range across several physical pools (DeepSeek V4 has six per page). Keys without an entry keepexpected_layout.initialize_xfer, aligned source and destination spans that are contiguous on both sides are coalesced. Runs at or below UCX's 128 KiB asynchronous put size never grow past it (dense 128 KiB spans stay on the multi-worker path); runs already above it merge up to 512 KiB. Transfer-identical; measured on both cases.Types:
MemDescriptoris amsgspec.Struct(frozen=True, gc=False); local DRAM slot allocation is incremental.Measured
Local copy path, 46 pages x 56 spans x 128 KiB = 338 MB:
Peer path, DeepSeek V4 Flash, two replicas on one 8x H100 node, 3000-token prompts, steady state after the first contact:
The remaining per-request floor on that deployment (about 0.25 s for any prompt) is SGLang's launch-bound DeepSeek V4 prefill forward, not KVCR. First contact with a new peer agent still costs 0.4-1.7 s of
add_remote_agentinside NIXL plus a cold first write; it is paid once per peer pair.Validation
uv run pytest tests/uniton this branch rebased ontomain(317e62f): 386 passed, 3 failed. The failures fail identically onmainin this environment (os.pidfd_openmissing for twotest_kvcr_servicecases, one real-NIXLtest_guard_integrationcase).ruff checkandruff format --checkclean.test_framework_regions.py(grouping by type and device, legacy compatibility and refusal of both spellings, overlap refusal, owner retention, startup failure cleanup, mixed-endpoint batching with byte verification),test_device_copy.py(batch landing on events, stream splitting and single-stream small copies, unregistered span rejection, size mismatch, per-span fallback, hard error handling, operand coalescing, close with a pending DMA), plus per-key layout, idle wake-up, control-channel wait and span coalescing cases in the DRAM, progress, control-channel and remote-source suites.test_gpu_transfer.py, skipped without CUDA) on an RTX A400 with NIXL 1.3.2 and torch 2.14.0+cu130: GPU to KVCR DRAM and back through the copy engine and through NIXL, mixed DRAM and GPU sources in one deposit, out-of-region descriptor failing its entry, bogus VRAM region failing startup cleanly.Consumers