Skip to content

[mem_cache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes - #41520

Merged
hnyls2002 merged 12 commits into
mainfrom
lsyin/adopt-cut3
Oct 2, 2026
Merged

hnyls2002 merged 12 commits into
mainfrom
lsyin/adopt-cut3

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator
  • cache_unfinished_req(req, chunked) -> checkpoint_req(req, *, up_to); maybe_cache_unfinished_req -> checkpoint_kv_cache; callers no longer pass chunked
  • InsertParams.chunked -> record_end: only the insert in insert_req bumps hit_count (and checks the write-through threshold); checkpoints, prompt-split re-inserts and storage/linker fills do not
  • write_through_threshold defaults to 1 for both write-through policies, keeping the first-finished-request backup timing of write_through_selective

CI States

Latest PR Test (Base): ✅ Run #36938189711
Latest PR Test (Extra): 🚫 Run #36938189462
Latest PR Test (AMD ROCm 10): 🚫 Run #36938189741

# Conflicts:
#	rust/sglang-radix-tree/src/python_bindings.rs
#	rust/sglang-radix-tree/src/tests/unified_tree_core.rs
#	test/registered/chunked_prefill/test_scripted_core_1gpu.py
@mintlify

mintlify Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
sglang-doc 🟢 Ready View Preview Oct 1, 2026, 10:20 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@hnyls2002 hnyls2002 added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 1, 2026
@hnyls2002 hnyls2002 added the highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed label Oct 1, 2026
@hnyls2002
hnyls2002 merged commit 50be533 into main Oct 2, 2026
232 of 256 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/adopt-cut3 branch October 2, 2026 04:41
nvpohanh added a commit to ekrim/sglang that referenced this pull request Oct 2, 2026
sgl-project#41520 renamed `cache_unfinished_disagg_prefill` to
`checkpoint_disagg_prefill` and `maybe_cache_unfinished_req` to
`checkpoint_kv_cache`. Update the new prefill-complete unit tests to patch
the new names. Also patch `checkpoint_kv_cache` in
`test_completed_prefill_parks_until_normal_pointer_handshake`: the new
helper reads `req.finished()` and `req.extend_range`, which the test's
lightweight request stub does not provide, and the test only checks the
bootstrap handshake.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sdulloor added a commit to Mad-Labs-HQ/sglang that referenced this pull request Oct 2, 2026
Upstream sgl-project#41520 (50be533) meant hits to count only when a request
finishes ("checkpoints ... do not"), but its insert_req counts every node
past req.kv.cache_inserted_len and advances it on every insert, and
checkpoint_kv_cache reaches that same insert_req. So each chunk of a chunked
prefill counts a hit, and with write_through (threshold 1) each chunk's KV
and Mamba state is backed up the moment it is stored; before sgl-project#41520 only the
whole prompt was.

On Qwen3.8-Flash-Next (--hicache-size 50, 4096-token chunks) that is about
three times as many host Mamba writes per long prompt. The host Mamba pool
(~224 slots) overflows, paused conversations lose their Mamba state on the
host, a returning prefix has KV on host but no state on its path, and it is
recomputed from token 0. Measured on the scratch box: 0/20 host-tier hits
and 0% returning-turn reuse under a 25-minute agent load (147 turns), against
20/20 and 95.7% (1,065 turns) for the same series on the 2026-09-25 base.

checkpoint_kv_cache passes its chunked flag on to insert_req; a chunked
insert counts no hit (inserted_len = up_to) and leaves cache_inserted_len
alone, so the first whole-prompt insert counts those nodes once. The other
insert_req implementations take the keyword and ignore it.
ormandj added a commit to ormandj/sglang that referenced this pull request Oct 7, 2026
Main removed InsertParams.chunked when checkpointing moved to
checkpoint_req (sgl-project#41520); the unfinished-request path is selected by
is_finished=False alone.
caihuali95 pushed a commit to caihuali95/sglang that referenced this pull request Oct 9, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len
and floors the frontier to a page. When the window is not a whole number of
pages, that can leave less than a window between the frontier and
page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1);
at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64
tokens there, and with the default 128-token eviction interval and one
token per step, every decode eviction lands on that residue.

page_floor(pre_len) is where an EAGLE request's insert ends when the next
verify finishes it with one token, since a bigram key spans one token less
than the request owns. The new leaf then holds 64 live SWA tokens against a
127-token window, the match after the insert rejects it, and
cache_protected_len stays at the previous prefix. The release then frees
[cache_protected_len, owned), including the pages the insert just gave the
tree, which the tree frees again when it evicts them. Since sgl-project#41520 folded
the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64
leaked and double-freed KV on triton and trtllm_mha alike, and some runs
hit an illegal memory access.

The retained span is now max(window, page) rounded up to whole pages, so
the frontier stays a full window below page_floor(pre_len). Page-aligned
windows and page size 1 keep the old frontier; an unaligned window keeps at
most one more SWA page per request.

Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey
runs a 7-token window at page size 4, the same misalignment. In
test_finish_after_decode_eviction_frees_only_the_requests_pages, a request
decodes, evicts at pre_len 23 and finishes at 24; before this change the
full pool counts 8 tokens twice ("264 != 256"), after it every page is
either free or held by the tree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ormandj added a commit to ormandj/sglang that referenced this pull request Oct 10, 2026
Main removed InsertParams.chunked when checkpointing moved to
checkpoint_req (sgl-project#41520); the unfinished-request path is selected by
is_finished=False alone.
caihuali95 pushed a commit to caihuali95/sglang that referenced this pull request Oct 13, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len
and floors the frontier to a page. When the window is not a whole number of
pages, that can leave less than a window between the frontier and
page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1);
at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64
tokens there, and with the default 128-token eviction interval and one
token per step, every decode eviction lands on that residue.

page_floor(pre_len) is where an EAGLE request's insert ends when the next
verify finishes it with one token, since a bigram key spans one token less
than the request owns. The new leaf then holds 64 live SWA tokens against a
127-token window, the match after the insert rejects it, and
cache_protected_len stays at the previous prefix. The release then frees
[cache_protected_len, owned), including the pages the insert just gave the
tree, which the tree frees again when it evicts them. Since sgl-project#41520 folded
the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64
leaked and double-freed KV on triton and trtllm_mha alike, and some runs
hit an illegal memory access.

The retained span is now max(window, page) rounded up to whole pages, so
the frontier stays a full window below page_floor(pre_len). Page-aligned
windows and page size 1 keep the old frontier; an unaligned window keeps at
most one more SWA page per request.

Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey
runs a 7-token window at page size 4, the same misalignment. In
test_finish_after_decode_eviction_frees_only_the_requests_pages, a request
decodes, evicts at pre_len 23 and finishes at 24; before this change the
full pool counts 8 tokens twice ("264 != 256"), after it every page is
either free or held by the tree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jiminator pushed a commit that referenced this pull request Oct 16, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len
and floors the frontier to a page. When the window is not a whole number of
pages, that can leave less than a window between the frontier and
page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1);
at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64
tokens there, and with the default 128-token eviction interval and one
token per step, every decode eviction lands on that residue.

page_floor(pre_len) is where an EAGLE request's insert ends when the next
verify finishes it with one token, since a bigram key spans one token less
than the request owns. The new leaf then holds 64 live SWA tokens against a
127-token window, the match after the insert rejects it, and
cache_protected_len stays at the previous prefix. The release then frees
[cache_protected_len, owned), including the pages the insert just gave the
tree, which the tree frees again when it evicts them. Since #41520 folded
the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64
leaked and double-freed KV on triton and trtllm_mha alike, and some runs
hit an illegal memory access.

The retained span is now max(window, page) rounded up to whole pages, so
the frontier stays a full window below page_floor(pre_len). Page-aligned
windows and page size 1 keep the old frontier; an unaligned window keeps at
most one more SWA page per request.

Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey
runs a 7-token window at page size 4, the same misalignment. In
test_finish_after_decode_eviction_frees_only_the_requests_pages, a request
decodes, evicts at pre_len 23 and finishes at 24; before this change the
full pool counts 8 tokens twice ("264 != 256"), after it every page is
either free or held by the tree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit b15bf09)
ch-wan pushed a commit to caihuali95/sglang that referenced this pull request Oct 18, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len
and floors the frontier to a page. When the window is not a whole number of
pages, that can leave less than a window between the frontier and
page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1);
at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64
tokens there, and with the default 128-token eviction interval and one
token per step, every decode eviction lands on that residue.

page_floor(pre_len) is where an EAGLE request's insert ends when the next
verify finishes it with one token, since a bigram key spans one token less
than the request owns. The new leaf then holds 64 live SWA tokens against a
127-token window, the match after the insert rejects it, and
cache_protected_len stays at the previous prefix. The release then frees
[cache_protected_len, owned), including the pages the insert just gave the
tree, which the tree frees again when it evicts them. Since sgl-project#41520 folded
the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64
leaked and double-freed KV on triton and trtllm_mha alike, and some runs
hit an illegal memory access.

The retained span is now max(window, page) rounded up to whole pages, so
the frontier stays a full window below page_floor(pre_len). Page-aligned
windows and page size 1 keep the old frontier; an unaligned window keeps at
most one more SWA page per request.

Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey
runs a 7-token window at page size 4, the same misalignment. In
test_finish_after_decode_eviction_frees_only_the_requests_pages, a request
decodes, evicts at pre_len 23 and finishes at 24; before this change the
full pool counts 8 tokens twice ("264 != 256"), after it every page is
either free or held by the tree.
wl55721 added a commit to wl55721/sglang that referenced this pull request Oct 27, 2026
…mantics

sgl-project#41520 counts a cache hit only when a request finishes, so hit_count advances by one per finished request instead of two. Request the victim prefix twice so it lands at hit_count == 2 (exactly the SLRU default threshold, below the configured 4), and update the phase-2 comment that described the pre-sgl-project#41520 double-count behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed memory-pool parallel-stages CI: stages dispatch together instead of waiting on each other run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant