Skip to content

Typed framework memory regions, same-process device copies, and batched peer writes for the SGLang direct linker - #39

Closed
ishandhanani wants to merge 15 commits into
mainfrom
idhanani/framework-gpu-regions
Closed

ishandhanani wants to merge 15 commits into
mainfrom
idhanani/framework-gpu-regions

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Sep 17, 2026 •

Copy link
Copy Markdown

Why

SGLang's direct external linker (ishandhanani/sglang#10) moves KV pages straight between GPU KV pools and KVCR-owned DRAM and restores prefixes from a peer worker's tier over NIXL. Two things stood in its way in KVCR:

  • Frameworks could only hand KVCR one contiguous DRAM region (framework_dram), so GPU pools could not be transfer endpoints at all.
  • A copy whose two endpoints belong to the same NIXL agent has no native UCX lane for device memory and falls back to software emulation over loopback TCP: 0.3-0.7 GB/s on an H100 host.

Everything here is additive. framework_dram keeps working, the public API only gains optional arguments, and no wire format changes.

What changes

Registration (770bef8)

  • FrameworkMemoryRegion(address, length, mem_type, device_id, owner=None) and KVCRBackendConfigs.framework_regions. mem_type is the NIXL segment type (DRAM or VRAM), device_id the device index. One NIXL register_memory per (mem_type, device_id) group on the progress thread; agent metadata is captured after every group registered; a failing group raises and earlier groups are deregistered.
  • Exact duplicate views collapse to one registration; partially overlapping regions of one endpoint are refused; configuring both framework_dram and framework_regions is refused. owner references are held until close().
  • GPU regions are transfer endpoints (deposit sources, deliver destinations), not an eviction tier. KVCR-owned storage stays DRAM.

Same-process device copies (fd37a91, 4a84d1f, 97f4b35, 0c8d2bc, f19582c, 450885b)

  • device_copy.py: a local DRAM fill or delivery with a VRAM endpoint is issued as cudaMemcpyBatchAsync on KVCR-owned streams and completes through recorded events. The CUDA runtime is bound through ctypes, so KVCR gains no framework dependency; LocalDramOptions.device_copy=False keeps the NIXL path for diagnostics.
  • Validation and argument marshalling are vectorised with numpy (a page contributes one span per layer buffer). Microsecond runtime calls keep the GIL; only the batch call releases it. Operands contiguous on both sides are merged before submission, and the stream count is bounded by span count and by bytes (one stream per 16 MiB) so a small copy is one call and one event.
  • Completion semantics are unchanged: an entry completes when the copy or transfer is done, and a consumer reading a GPU destination orders its stream after that.

Progress thread (d507a4a, 9e91430, 266bd91)

  • 0.1 ms cadence while operations are in flight. With nothing in flight the loop parks on a wake-up pipe (written by submit) plus the peer control socket (ZmqPeerControlChannel.wait), bounded at 20 ms, instead of waking every millisecond; in a framework process every wake is a GIL hand-off on the scheduler thread. NIXL notification polling is skipped when no control channel exists.

Peer writes (58421c6, 7911692, 1355b6b, e304c9b)

  • fetch(..., expected_layouts={key: layout}): a layout per key, so one operation, one control message, one NIXL WRITE and one notification cover a page range across several physical pools (DeepSeek V4 has six per page). Keys without an entry keep expected_layout.
  • Before initialize_xfer, aligned source and destination spans that are contiguous on both sides are coalesced. Runs at or below UCX's 128 KiB asynchronous put size never grow past it (dense 128 KiB spans stay on the multi-worker path); runs already above it merge up to 512 KiB. Transfer-identical; measured on both cases.

Types: MemDescriptor is a msgspec.Struct(frozen=True, gc=False); local DRAM slot allocation is incremental.

Measured

Local copy path, 46 pages x 56 spans x 128 KiB = 338 MB:

Host NIXL loopback (before) device copy engine memcpy ceiling
H100 NVL 0.4-0.5 GB/s 20.7-21.0 GB/s 39 GB/s
RTX A400 (PCIe Gen4 x8) 0.6-0.8 GB/s 11.6-12.9 GB/s 13.4 GB/s

Peer path, DeepSeek V4 Flash, two replicas on one 8x H100 node, 3000-token prompts, steady state after the first contact:

Topology metric before after
TP4 + DP attention 4 hinted restore TTFT 0.40-0.56 s 0.31-0.36 s
TP4 + DP attention 4 recompute TTFT of the same prompt 0.88-0.98 s 0.75-0.82 s
TP4 + DP attention 4 preparation wait per request 0.17-0.34 s 37-70 ms
TP4 + DP attention 4 restore into GPU 21 ms 6-8 ms
TP4, four ranks fetching at once preparation wait per rank 1.06-1.09 s 40-58 ms
TP4, four ranks fetching at once hinted TTFT vs recompute 2.13 s vs 1.41 s (e2e) 0.48-0.55 s vs 1.11-1.18 s

The remaining per-request floor on that deployment (about 0.25 s for any prompt) is SGLang's launch-bound DeepSeek V4 prefill forward, not KVCR. First contact with a new peer agent still costs 0.4-1.7 s of add_remote_agent inside NIXL plus a cold first write; it is paid once per peer pair.

Validation

  • uv run pytest tests/unit on this branch rebased onto main (317e62f): 386 passed, 3 failed. The failures fail identically on main in this environment (os.pidfd_open missing for two test_kvcr_service cases, one real-NIXL test_guard_integration case). ruff check and ruff format --check clean.
  • Fake-agent tests: test_framework_regions.py (grouping by type and device, legacy compatibility and refusal of both spellings, overlap refusal, owner retention, startup failure cleanup, mixed-endpoint batching with byte verification), test_device_copy.py (batch landing on events, stream splitting and single-stream small copies, unregistered span rejection, size mismatch, per-span fallback, hard error handling, operand coalescing, close with a pending DMA), plus per-key layout, idle wake-up, control-channel wait and span coalescing cases in the DRAM, progress, control-channel and remote-source suites.
  • Real NIXL UCX GPU tests (test_gpu_transfer.py, skipped without CUDA) on an RTX A400 with NIXL 1.3.2 and torch 2.14.0+cu130: GPU to KVCR DRAM and back through the copy engine and through NIXL, mixed DRAM and GPU sources in one deposit, out-of-region descriptor failing its entry, bogus VRAM region failing startup cleanly.
  • Through the SGLang consumer on real GPUs, judged by served tokens and greedy output equality: Qwen3-0.6B TP1 and TP2, DP-attention owners, PD prefill, peer reuse with no-hint, stale-hint and dead-peer controls; DeepSeek V4 Flash single-worker round trip at TP4 and two-replica reuse at TP4 and at TP4 with DP attention 4.

Consumers

@ishandhanani

Copy link
Copy Markdown
Author

Consumer PR: sgl-project/sglang#39946 (KVCR direct external linker). Dynamo companion: ai-dynamo/dynamo#14992.

@ishandhanani ishandhanani changed the title Register typed framework memory regions, including GPU endpoints [fable] Register typed framework memory regions, including GPU endpoints Sep 18, 2026
ishandhanani and others added 14 commits September 21, 2026 18:47
Add FrameworkMemoryRegion and KVCRBackendConfigs.framework_regions so a
framework can register its actual allocations (DRAM or VRAM, per device)
as NIXL transfer endpoints. Registration runs on the progress thread, one
NIXL call per (mem_type, device_id) group, and agent metadata is captured
only after every group registered. Exact duplicate views collapse to one
registration, partially overlapping regions of one endpoint are refused,
and configuring both framework_dram and framework_regions is refused.
The core retains region owners until close() releases the registration.

Local DRAM deposit and deliver batch copies per (source, destination)
endpoint so a NIXL descriptor list never mixes memory types, and a block
whose spans disagree on memory type or device is rejected up front.

Tests cover grouping, validation, owner lifetime, startup registration
failure cleanup, mixed-endpoint batching, close with a pending copy, and
a real NIXL UCX GPU <-> DRAM deposit/deliver round trip.
NIXL routes a transfer between two endpoints of the same agent through its
network transport, and UCX has no device-memory lane for that loopback: puts
between VRAM and host fall back to software emulation over TCP at 0.3-0.7 GB/s
on an H100 host (PCIe memcpy: 38 GB/s). Local DRAM copies with a VRAM endpoint
now go through cudaMemcpyBatchAsync on KVCR-owned streams, split across up to
four streams, and complete through recorded events. The engine binds libcudart
with ctypes, validates VRAM spans against registered framework regions with
numpy, and falls back to NIXL when the runtime is unavailable or
LocalDramOptions(device_copy=False) is set.

Per-span Python work is removed from the hot path: MemDescriptor is a
gc-free msgspec Struct (6x cheaper to build), descriptor normalisation uses
C-level maps with cached layout sizes, and device copies express KVCR slots as
address arrays instead of materialised descriptors.

Measured deposit/deliver of 46 pages x 56 spans x 128 KiB (338 MB): A400
11.6 GB/s (memcpy 13.4), H100 17-20 GB/s (memcpy 39), versus 0.4-0.9 GB/s
through NIXL loopback on both.
The progress thread slept 1 ms between iterations whenever an iteration
observed no work, so a device copy that lands in a few milliseconds was
detected up to 1 ms late. Sleep 100 us while any op is in flight and keep the
1 ms idle wait otherwise.
Each cudart call takes microseconds; binding libcudart through PyDLL keeps the
GIL across them instead of paying a switch interval to reacquire it whenever
another Python thread is busy, which is the normal state of a serving process.

Slot allocation under eviction recomputed the shortfall over every required
pool twice per candidate. The shortfall is now tracked only for the pools
still short, the per-layout slot counts are cached, and block byte counts
come from C-level maps, so a page-sized eviction is one pass.
Peer write notifications can only arrive through peers reached over the
framework control channel. Without one, the per-iteration get_new_notifs call
held the GIL for about a tenth of the progress thread's time for nothing.
The batch copy's driver-side validation grows with the span count, so that
one call goes through a releasing CDLL handle; every other runtime call stays
on the PyDLL handle and keeps the GIL for its few microseconds.
…ives

With nothing in flight the progress thread slept 1 ms between iterations,
so it woke a thousand times a second in the framework process and each
wake was a GIL hand-off on the scheduler thread; on a four-rank DP-attention
deployment that class of wake-up cost every request tens of milliseconds.
Submissions now wake the loop through a pipe, the peer control channel
offers a wait that polls its socket and that pipe together, and the loop
parks for at most 20 ms otherwise. The 0.1 ms cadence while operations are
in flight is unchanged.
A framework with several physical pools per page (DeepSeek V4 has six) had
to issue one fetch per pool, so one restored page range cost one control
message, one peer write, and one notification per pool. fetch now takes
expected_layouts mapping keys to their own layouts; keys without an entry
keep the operation's expected_layout. Slot reservation, layout checks, and
capacity waiters use the key's layout.
Descriptor lists are built per span so layout checks stay exact, but NIXL
request creation and posting cost per descriptor: a DeepSeek V4 page has
147 spans, so one 11-page fetch posted 1617 descriptors and the source
spent 18 ms per operation in transfer_submit. Aligned source and
destination spans that are contiguous on both sides (a page's spans in a
KVCR slot are) are now merged up to 512 KiB, which keeps enough pieces for
UCX to spread the copy over its workers.
cudaMemcpyBatchAsync costs host time per span (about 10 us each, so a
590-span DeepSeek V4 restore batch spent 7.5 ms in submission for 1 ms of
DMA). Pages allocated in sequence have adjacent device rows and adjacent
KVCR slots, so consecutive operands contiguous in both address spaces are
merged with one vectorised pass before the request is built; a merged run
copies the same bytes with fewer, larger spans.
Merged descriptors above 128 KiB are copied inline by the posting thread
instead of asynchronously across UCX workers, which made a dense model's
338 MB fetch slower (89-101 ms against 78-95 ms unmerged). Spans at or
above the threshold are left as they are; only smaller spans are combined.
… spans async

Spans above UCX's 128 KiB asynchronous put size are copied inline by the
posting thread whatever their size, so runs of them (DeepSeek V4's 146 KB
sliding-window spans, 43 per page) now merge up to 512 KiB, which measured
24-32 ms per 100 MB fetch against 40-46 ms unmerged; runs of spans at or
below 128 KiB still never grow past it, so a dense model's 128 KiB spans
keep their asynchronous multi-worker path.
Each stream costs a batch call and an event, about 0.35 ms of host time
together, so a 19 MB restore split over four streams spent more in
submission than in DMA. Streams are now bounded by bytes as well as by span
count (one per 16 MiB), which keeps large copies parallel.
@ishandhanani
ishandhanani force-pushed the idhanani/framework-gpu-regions branch from c705a8a to 450885b Compare September 22, 2026 01:49
@ishandhanani ishandhanani changed the title [fable] Register typed framework memory regions, including GPU endpoints Typed framework memory regions, same-process device copies, and batched peer writes for the SGLang direct linker Sep 22, 2026
NIXL registration of the framework's pools scales with their size; on a B200
node with 46 GB of KV per rank plus a 32 GB DRAM tier the 30 s default is
marginal when the pools go through an RDMA NIC. Keep the default, allow the
deployment to raise it.
@rishabhsinha17

Copy link
Copy Markdown

I maintain kvcached's SGLang integration, including the pool and allocator compat layer for sglang 0.5.16 through 0.5.20 (ovg-project/kvcached#514, #516, #525) and the zero-attention-layer pool handling that hybrid models need (ovg-project/kvcached#524). The direct linker calls into those same pool classes. Rather than start anything parallel, I would like to help this lane land.

Concretely, I can review and validate this PR and #44 on a GPU host, with an eye on compatibility across the recent sglang pool-class revisions. A validation plan for the resiliency side would be the other piece I can take, exercising the linker path across Guard promotion and worker replacement. If a short design or test-plan doc would help, I will write one.

@mkhazraee

Copy link
Copy Markdown
Collaborator

Hi Rishabh @rishabhsinha17, sorry just saw this. This was an initial PR to try things out, now the main covers what's needed and I wasn't paying attention. Glad to hear your interest and it would be great to collaborate. As of now this branch has the current ongoing linker integration, but needs some final clean ups: https://github.com/linhu-nv/sglang/tree/linhu/kvcr-tp-dedup-v4

@linhu-nv and @ziqifan617 are the main contacts for SGLang integration, so that would be great if you get connected, and then plan the collaboration.

(I'm going to close this PR with the next comment).

@mkhazraee

Copy link
Copy Markdown
Collaborator

Closing this initial integration PR. Typed and strided framework registration, pool/index references, and labelled partial delivery are now covered by #59, #60, and #61. Configurable startup timeout landed in #82, with subsequent PRs addressing metadata caching, msgspec records, and NIXL transfer preparation.

The separate CUDA copy engine has not been required for the current SGLang integration. If profiling demonstrates a remaining local-copy bottleneck, we can address it in NIXL or a focused KVCR change. The additional per-key fetch layouts, span coalescing, and progress-loop tuning remain deferred and can be revisited independently when a concrete workload justifies them.

@mkhazraee mkhazraee closed this Oct 16, 2026
@rishabhsinha17

Copy link
Copy Markdown

Thanks. I read linhu/kvcr-tp-dedup-v4 at 561eccf and #59 to #61 and #82. The slice I would own is elastic pools, which reserve virtual memory and back it later. KVCRLayout.init in storage/kvcr/layout.py registers every K and V buffer with count = buffer.shape[0] // row_span, and kvcr's _register_memory_regions registers that whole extent, which for an elastic pool is its reservation rather than its backed span, the span get_contiguous_buf_infos sizes for prefill-decode transfers. kvcached hits this on every launch, and its default interleaved layout fails the contiguous page piece check above page_size 1. I would add unit coverage and the kvcached-side guard, run the manual harness, and coordinate with @linhu-nv and @ziqifan617. Where should this live, a kvcr issue or the sglang fork?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants