Skip to content

[Refactor] mem_cache: Gate unified page reuse and moves on one per-band page ledger - #48008

Merged
ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-window-rails
Nov 8, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-window-rails

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

Under --enable-unified-memory, the bands of the shared KV arena reuse freed pages and move live pages (compaction of an end, a float making room for its neighbour) while forwards and host copies may still read or write them. Whether a page may be written, and by whom, was decided separately on each take, free and move path, and several paths checked nothing:

  • a schedule-stream write into a page (a compaction copy, page zeroing, a state copy into a state slot) had no record of a forward that might still read bytes a move vacated, or write a page it was handed, nor of host copies and transfers on the page;
  • eager compaction (SGLANG_DISABLE_LAZY_COMPACTION=1) never read the pages reserved for a host load, and could move one under its queued copy;
  • HiCache kept every page of every band in place while any host copy was queued or unacknowledged, so under write-through compaction rarely ran.

This PR gives each band one record of who may still touch its pages, and routes every hand-out and every return of a page through one path that knows who will write it.

Modifications

  • One static storage per pool (first commit, static_kv_storage.py, allocation.py, disaggregation/decode.py): kv_storage_of keeps one StaticKVStorage per pool instead of building one per call, so the static pools' startup bindings (window retention, shape, the PD draft pool) live on it; take_extend / take_decode take the step's tree_cache and batch per call. The allocator's window_retention attribute is removed. Each caller of bind_shape knows part of the shape (the prefix tree its window and checkpoint interval, the scheduler the capacities), so a binding merges into the one held: a field it leaves None keeps its earlier value, and the tree's slots do not depend on the order of the bindings.
  • Page ledger (kv/alloc/page_usage.py, kv/alloc/signals.py, kv/alloc/band.py):
    • PageUsage lists, per band, unfinished forwards with the ids they write, host copies with the rows they touch, pages a move vacated while a reader may still read them (vacated pages that return to the free list once their readers retire, and fences over bytes that left the band), and fixed pages.
    • A forward's hazard on a page is Hazard.FORWARD: it may still write the page, or read a vacated page's old bytes. A page its owner frees while a forward runs waits in its band's pending batches under that forward ([Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free #48852) and reaches the free list quiet.
    • An owner free never overlaps a host load's reservation; at invariant level WARN and up the free checks this on the device (FREE_OUTSIDE_RESERVATION).
    • Decisions read retirement (a forward whose result the scheduler processed), never whether an event fired, so every rank makes the same ones; waits are stream waits on the GPU.
    • Two questions, two methods: PageUsage.forward_retired(event) answers for a forward's event only, from its logical retirement, and refuses any other key; transfer_ended(key) answers for a transfer, and a host copy holds no decision, since whoever writes its pages waits its events once the choice is made. A host copy's key is a StreamTransfer around its events (signals.py), so it is never looked up as a forward; a release key holding forwards and copies is asked with ended, member by member.
    • The ledger keeps its own copy of every id tensor it records (page_usage._owned: forwards, transfers, vacated batches), so a request-table row reused after a free cannot rewrite the ids of a recorded write. Records are made on the stream page moves run on, where that copy is made (stated on Band.record_transfer and Band.set_inflight_forward).
    • A move waits only the forwards and copies that write a page it copies; what it vacates is fenced for its next writer. Fixed pages never move: compaction packs around them, and no end frontier or float span shrinks past one, free or not.
  • One acquire and one release path (band.py, page_usage.CLEARANCE):
    • Band.acquire(n, consumer) is the only path that hands out writable pages; Band.release the only one that returns them. The consumer is a forward, a schedule-stream copy, page zeroing, a host load or a network receive; one table says what each must wait for each hazard (a forward's late write, a forward's read of vacated bytes, a host copy's read or write, a network transfer, a fixed page). Zeroing waits a forward's late write as a schedule-stream copy does: freed bytes can pass to another band, and a finite FP32 state word read as BF16 attention padding can be a NaN. A wait that two consumers of one take name is issued once.
    • acquire keeps [Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free #48852's take order: holes; when they are short, the oldest whole pending batches that cover the rest, settled (Band.settle_pending); then the frontier. Every check runs before a settle, so a refused take changes nothing. A planned take of a float extends only into its planned block.
    • Every allocator operation runs in one scope (allocator_operation): a band take, a chain allocation, a planned or scoped take, a host load's reservation, a flush and the urgent recovery. Before the operation's first stream wait, the bands it touches read ahead once on the host what it will read (_before_wait): an operation that moves pages reads everything (Band._read_ahead: free lists, pending pages, the pages unfinished forwards write, transfer pages); a take in a chain where a fence may meet a span reads only what host hole selection needs; an ordinary take with no fence, a pending settle included, reads nothing. After that wait, a host read of the device is an invariant violation (HOST_READ_BEFORE_WAIT, through utils/invariants.py::expect_host: logged at invariant level WARN, raised at STRICT). A take keeps the free list's host copy only inside an operation that read it ahead, or while a fence may make it choose holes on the host.
    • Clearance walks the ledger once per take (PageUsage.take_clearance_runs, clearance): every use is bisected into the take's sorted runs, and each signal is asked once per call. Fences whose readers retired and whose signals completed are dropped at retirement.
    • Host hole selection (Band._take_holes_on_host) marks hazarded holes in one sweep of the sorted holes against the sorted fence byte spans, instead of asking the ledger per hole.
    • Which holes a take hands out, and in what order, is one rule over retirement and releases (Band._take_holes): holes no unretired forward makes the consumer wait for come first, then the rest, each class in order from the take's end. The device path and the host path choose the same pages in the same order, so every rank takes the same pages whatever its host copies have done; a fired event only skips a wait on the pages chosen. With no hazarded hole the take reads nothing on the host.
    • The urgent-recovery entry of [Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free #48852 (settle_pending_for_recovery) also reads ahead, after its one read of the holes and pending pages and before any wait, what each band's moves read on the host: the pages unfinished forwards write and live transfers touch. Inside it, a host read of the free list after a wait raises. An end a pin source holds reclaims nothing on an urgent flush, so it is left out of the read and the settle. The operation carries one read plan (_AllocatorOperation.reads_all) that the recovery and every later wait of the operation follow, so a band the recovery leaves out is never read in full; it reads its geometry only where a fence needs it.
    • Each band keeps its free list's host copy with the list on every rebind, or drops it; an empty list's copy is always known. The copy is a kv/alloc/host_free_list.py::HostFreeList: one ordered backing list consumed from both ends by offsets, so a take of a few pages touches only those pages and a free appends; once the consumed ends together pass the pages held, the backing list compacts, so it stays within about twice the pages held. A band keeps it only while a fence it may meet is outstanding (_fence_may_order_takes: an unfinished fence over bytes the band may span, PageUsage.fence_outstanding); a completed fence, or one over bytes the band cannot reach, keeps none. A hole take under such a fence walks from its end until it has enough quiet pages, checking each against the sorted fence spans, and walks the whole list only when too few holes are quiet.
    • A forward's write set and every free path name pages in one coordinate: Band.virtual_pages_of(token_ids) turns the ids the allocator hands out (widened by dcp_size under decode context parallelism) into virtual pages, for _forward_written_pages, _free_lazy, free_segment, free_group_end, FloatBand.free and the chain's window and full-segment frees. _physical_pages_of keeps translating through the v2p table, whose unit is the physical page.
    • New state slots are acquired as schedule-stream copies (ChainAllocator.alloc_state_slots).
    • A float's on-demand moves (make_room, compact_holes) clear every destination, wait only writers of their sources, and fence what they vacate; an interior shift, whose destinations include sources vacated earlier in the same pass, waits every running forward.
    • A host load's reservation (Band.alloc_physical) reads the free list ahead. When holes do not cover it, the band settles its pending pages first, so the reservation takes holes and growth only. If the band's holes and pending pages together still fall short, a relief follows, so the operation counts as moving and every chain band reads ahead before that settle waits; the reserved pages are named exactly, as the span's growth plus the holes that left the free list, from host copies the take kept.
  • Forward recording and retirement (managers/scheduler.py, scheduler_pp_mixin.py, managers/utils.py, kv/storage.py, kv/alloc/chain.py): the scheduler records each launched forward through the storage (record_forward, with token and state write ids; unknown for overlapped speculative and pipeline verify writes) and retires it at the processed hook (Scheduler._on_step_processed → ForwardSteps.forward_processed), after the result processor's synchronize, together with the pages pending under it. _record_unified_forward records the forward's event with the chain's forward steps too; a pipeline stage carries it in its microbatch metadata (unified_forward_done), and its skipped output records its completion behind it. A speculative forward's draft and verify slots are reserved before the step is launched: no managed page is allocated inside a forward. A static pool's storage answers these as no-ops. Without overlap scheduling, a forward is recorded only under pipeline parallelism: a single stage runs it on the schedule stream and processes its result before the next allocation.
  • HiCache without a band-wide pin (cache_controller.py, l2_transfer.py, hybrid_cache/*, unified_radix_cache.py, unified_cache/components/swa.py, rust_tree_core/adapter.py, buffer_mode/pipeline.py, disaggregation/decode_kvcache_offload_manager.py):
    • window copies name their rows by the owner's token ids and translate them at submission (KVStorage.host_ids), so a move before submission is seen; after it, a row a load-back writes is copied only after that copy, and a row a backup only reads moves at once, its old bytes reused only after the read;
    • submitted copies are recorded on the rows they touch (KVStorage.record_host_transfer), and a load's window reservation is fixed only until its rows are named by owner ids;
    • a host copy is recorded on the stream page moves run on, with ids built there or ordered before it; the copy itself may run on any stream (stated on KVStorage.record_host_transfer). UnifiedRadixCache.start_restore_kv_cache(..., copy_stream=None) builds the ids and records the copy on the current (scheduler) stream; with copy_stream, only the submission runs there, after copy_stream.wait_stream(current). PD decode's _start_host_restores (host receive and retraction restore) passes the forward stream for the kernel I/O backend instead of entering that stream around the whole restore (mem_cache/common.py, disaggregation/decode.py);
    • the scheduler no longer registers the HiCache pin source, and set_hicache_transfer_done_event / has_inflight_device_transfers are removed.
    • PD transfers keep their band-wide pin sources.
  • Lazy compaction only (arg_groups/kv_cache_hook.py, band.py, chain.py, kv/build.py): every unified launch asserts lazy compaction; the eager end-band compaction and every eager branch of the bands are removed, and ends compact lazily by default.
  • Tests:
    • new test/registered/unit/mem_cache/test_unified_page_writability.py (CPU): the clearance table, each consumer × hazard pair through the ledger, a transfer's key never answered as a forward (test_a_transfer_key_is_never_a_forward), hole takes under an overlapped forward, a compaction into a slot a running forward writes, float moves and hand-back, a state slot reused under a running forward, and the startup refusal;
    • test_unified_page_writability.py::TestTakesReadNoEventCompletion (whether a host copy has finished changes no take: the pages, their order and the owner's available_size agree), TestFreedPagesAreQuiet (a free during a recorded forward stays out of the holes; a free of a reserved page trips the check) and TestHostLoadReservations (a reservation from pending pages stays fixed through relief; drained holes are reserved exactly, every host read before any wait; a load cancelled after submission keeps its copy hazard);
    • new test_unified_transfer_ordering.py (CUDA): float and end moves against delayed host copies, each transfer recorded on the stream the moves run on;
    • test_unified_handout_zeroing.py::TestZeroingWaitsALateStateWrite (CUDA): a forward on its own stream writes FP32 words, whose low halves read as BF16 NaNs, into a state slot after the slot is freed; the attention band grows into the slot's bytes and zeroes its pages; after both streams finish, the pages read zero;
    • test_unified_kv_storage.py::test_a_later_shape_keeps_the_tree_fields_it_leaves_out: a capacity binding after the tree's binding leaves tree_slots() unchanged;
    • test_unified_page_writability.py::TestWidenedPagesUnderDcp (CPU, DCP 2): an owner band growing down and one growing up free a page while a forward writes it, and compaction into it must wait that forward;
    • new test_unified_fixed_free_pages.py, test_unified_host_transfer_tracking.py, test_unified_scheduling_no_host_sync.py (CUDA, sync-debug mode; each case registers its forward and asserts that the wait it guards ran); managers/test_pp_skip_output_retirement.py carries the storage's forward event;
    • test_unified_scheduler_hooks.py: the processed step retires the forward; test_unified_handout_zeroing.py: a hand-out takes a pending page, its zeroing queued behind the forward;
    • new test_unified_take_clearance_runs.py: one pass over the ledger gives the same waits and remaining fences as run-by-run on randomized takes;
    • test_unified_release_after_forward.py: a two-band planned take reads every band before its first wait; a take's settle reads ahead and waits each forward once with no host read after; whether the forward fired decides only whether a wait is queued; a drained move source keeps the free list's host copy through an urgent flush, at invariant levels WARN and STRICT; an owner shortfall leaves a pinned window's pending pages alone and reads nothing of it; TestAHostLoadThatNeedsRelief: a window-only host load on a full, window and state chain, with pending pages, too few local pages and room a relief recovers from a neighbour, reserves its pages with no host read after the first wait and keeps byte accounting, at WARN and STRICT;
    • test_unified_host_transfer_tracking.py::TestRestoreRecordedWhereMovesReadIt (CPU, stand-in streams): through the real _start_host_restores, the ids and the record are on the scheduler stream and the copy on the forward stream after a forward-waits-scheduler wait (kernel backend), or all on the scheduler stream (direct backend);
    • test_unified_mamba_compaction.py (CUDA) covers moves under delayed forwards and host copies through the ledger, and that the ledger holds a copy of the write ids it is given; HiCache tests cover queued and unacknowledged copies with moving rows;
    • unit fixtures build lazy ends; tests that iterated over an eager mode keep the lazy one; tests that read removed internals read the ledger;
    • test_unified_release_after_forward.py::test_pinned_ends_are_not_read_when_the_float_settles: with every band pinned, a float's settle under owner pressure reads neither end; TestHostFreeListStorage: prefix and suffix takes against a list oracle, and the storage bound;
    • test_unified_take_host_work.py::TestSmallTakesUnderAFenceScaleWithTheTakes: 256 one-page takes under an outstanding copy fence or an unretired forward touch at most a few pages each, the same at 4096 and 16384 holes, with no device read of the free list; a completed fence and an unreachable one keep no host copy;
    • the allocator trace tool (python/sglang/test/unified_allocator_trace.py, test/manual/mem_cache/unified_trace_compare.py) reads pending pages from the ledger and records lazy compaction only (--lazy is removed); new TestBuilderAdapterRoundTrip (CPU) records and replays seeded traces through the real adapter on the window, Mamba and three-band chains.

Behaviour changes

  • Startup refusal: --enable-unified-memory with SGLANG_DISABLE_LAZY_COMPACTION=1 now fails at startup with "--enable-unified-memory requires lazy compaction; unset SGLANG_DISABLE_LAZY_COMPACTION." Before, only PD disaggregation refused it.
  • With HiCache, unified pages move while host copies are queued or unacknowledged (each move waits only the copies that touch its pages); before, the whole pool stayed in place.
  • A schedule-stream write into a page an overlapped forward may still write, and the zeroing of a page handed out by a band that zeroes (MHA and MLA pools), now wait that forward on the GPU (no host block); a forward's own take from a band that does not zero still waits nothing. For a page its owner freed, this happens through the pending batches of [Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free #48852.
  • Hole takes, and the capacity they leave, are the same on every rank whatever host copies have finished there.
  • Layout-changing geometry with pending pages, or inside an urgent recovery without a current host copy of the free list, is an invariant violation (GEOMETRY_READY): logged at invariant level WARN, raised at STRICT.
  • No change without --enable-unified-memory beyond the static storage being kept per pool.

Accuracy Tests

This stack is validated at its top, on main 3e38c5b77a: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed against main, all reported in #48014.

Checklist

  • Format with pre-commit on the changed files.
  • Unit tests added and updated.

CI States

Latest PR Test (Base): 🚫 Run #42476407492
Latest PR Test (Extra): 🚫 Run #42476407402
Latest PR Test (AMD ROCm 10): 🚫 Run #42476407337

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 2d10448 to 75012d2 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 0e9aae3 to 8df6f60 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 75012d2 to 6388235 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 8df6f60 to 4af2ce8 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 6388235 to 860b862 Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch 2 times, most recently from 7e1cf2d to 11c2779 Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 860b862 to e5b52e0 Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 11c2779 to 8262d14 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from e5b52e0 to d171f97 Compare November 4, 2026 09:31
This was referenced Nov 4, 2026
@ch-wan
ch-wan requested a review from yuan-luo as a code owner November 5, 2026 11:24
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch 2 times, most recently from e6dadd0 to 0b14d62 Compare November 5, 2026 22:15
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from ebe3e51 to a54b0e9 Compare November 5, 2026 23:59
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch 3 times, most recently from 9277df7 to 72f4463 Compare November 6, 2026 03:37
@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from a54b0e9 to ce3ae69 Compare November 6, 2026 06:33
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 72f4463 to dd8bc46 Compare November 6, 2026 06:33
@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from ce3ae69 to 551c591 Compare November 6, 2026 10:43
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from dd8bc46 to 8e4c606 Compare November 6, 2026 10:43
@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 551c591 to 0b37510 Compare November 6, 2026 12:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 8e4c606 to 6b06dfb Compare November 6, 2026 12:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 0b37510 to 6c2996f Compare November 6, 2026 13:40
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 6b06dfb to 5b1b6fa Compare November 6, 2026 13:40
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 6c2996f to cc7cf65 Compare November 6, 2026 21:21
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-rails branch from 5b1b6fa to 22dc693 Compare November 6, 2026 21:21
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from cc7cf65 to 3812bbc Compare November 7, 2026 09:02
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

…nd page ledger

Under `--enable-unified-memory`, when a physical page may be written, and by
whom, was decided separately by each take, free and move path. Several paths
checked nothing: a hole freed while an overlapped forward still wrote it could
be handed to a schedule-stream copy or a compaction, a float handed recent
holes back to the gap unguarded, and eager compaction ignored pages reserved
for a host load. HiCache kept every band's pages in place while any host copy
was queued or unacknowledged.

Ledger:
- Each band keeps one `PageUsage` record (`kv/alloc/page_usage.py`) of who may
  still touch its pages: unfinished forwards with the ids they write (unknown
  for overlapped speculative and pipeline verify writes), host copies, pages a
  move vacated while a reader could still read them (pending pages and
  fences), and fixed pages.
- The scheduler records every launched forward through the storage and
  retires it when its result is processed; decisions read retirement, never
  whether an event fired, so every rank makes the same ones.
- A move waits only the forwards and copies that write its sources; vacated
  bytes are fenced for their next writer. Pages that must stay (a host load's
  reservation) are fixed per page, and compaction moves around them instead
  of stopping the band; no frontier or float span shrinks past a fixed page.
- HiCache window copies name their rows by the owner's token ids and
  translate them at submission, and submitted copies are recorded on the rows
  they touch, so the band-wide HiCache pin is removed. PD transfers keep their
  band-wide pin sources.

One acquire and one release path:
- `Band.acquire(n, consumer)` is the only path that hands out writable pages
  and `Band.release` the only one that returns them. What a consumer (a
  forward, a schedule-stream copy, page zeroing, a host load, a network
  receive) must wait for each hazard is one table, `page_usage.CLEARANCE`.
  Holes no hazard makes the consumer wait for are taken first, without a host
  read when none is hazarded.
- New state slots are acquired as schedule-stream copies: a state copy may
  write a slot an overlapped forward still writes in place.
- A float's on-demand moves (`make_room`, `compact_holes`) clear their
  destinations, wait only writers of their sources, and fence what they
  vacate.
- A band's or a chain's take, a flush and a recovery each run as one
  allocator operation (the outermost call). Before its first stream wait it
  reads ahead on the host what it reads after: everything, for an operation
  that moves pages; for a take, only the free and pending pages of bands a
  fence may make choose holes on the host, so an ordinary reuse of pending
  pages reads nothing. A device read on the host after that wait fails at
  invariant level WARN and up. An unfenced take outside such a read keeps
  no host copy of the free list.
- A take chooses holes from the byte ranges unretired forwards hold, found in
  one walk of each ledger and swept once over the holes; copy fences order
  nothing, so they leave the device's choice and only its clearance is read.
  The clearance walks each ledger once for all the chosen runs, asking each
  fence's signals once; fences whose readers retired and whose signals
  completed are dropped at retirement.

Lifecycle:
- Unified memory requires lazy compaction: startup refuses
  `SGLANG_DISABLE_LAZY_COMPACTION=1` with `--enable-unified-memory` (before,
  only PD refused it), and eager end-band compaction is removed.

Static pools:
- `kv_storage_of` keeps one `StaticKVStorage` per pool, created on first use
  and rebuilt (keeping its startup bindings) when the request table changes.
  `shape` and the PD draft pool are bound to it (`bind_shape`,
  `bind_transfer_draft`) rather than passed to a new instance, and a static
  window's retention is held by the storage instead of the allocator.
- `take_extend` / `take_decode` take `tree_cache` and `batch` per call (a
  unified pool ignores them); the decode preallocation passes its tree cache
  through. The static storage answers the ledger's recording calls as no-ops.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 hicache Hierarchical Caching for SGLang highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed memory-pool mthreads npu run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant