Repository navigation
[mem_cache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes - #41520
Merged
Merged
Conversation
hnyls2002
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
Jialin,
ShangmingCai,
Ying1123,
alphabetc1,
hanming-lu,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sogalin,
xiezhq-hermann and
yizhang2077
as code owners
September 28, 2026 05:06
# Conflicts: # rust/sglang-radix-tree/src/python_bindings.rs # rust/sglang-radix-tree/src/tests/unified_tree_core.rs # test/registered/chunked_prefill/test_scripted_core_1gpu.py
hnyls2002
requested review from
JustinTong0323,
wisclmy0611 and
zijiexia
as code owners
September 30, 2026 23:56
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
4 of 5 tasks
nvpohanh
added a commit
to ekrim/sglang
that referenced
this pull request
Oct 2, 2026
sgl-project#41520 renamed `cache_unfinished_disagg_prefill` to `checkpoint_disagg_prefill` and `maybe_cache_unfinished_req` to `checkpoint_kv_cache`. Update the new prefill-complete unit tests to patch the new names. Also patch `checkpoint_kv_cache` in `test_completed_prefill_parks_until_normal_pointer_handshake`: the new helper reads `req.finished()` and `req.extend_range`, which the test's lightweight request stub does not provide, and the test only checks the bootstrap handshake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sdulloor
added a commit
to Mad-Labs-HQ/sglang
that referenced
this pull request
Oct 2, 2026
Upstream sgl-project#41520 (50be533) meant hits to count only when a request finishes ("checkpoints ... do not"), but its insert_req counts every node past req.kv.cache_inserted_len and advances it on every insert, and checkpoint_kv_cache reaches that same insert_req. So each chunk of a chunked prefill counts a hit, and with write_through (threshold 1) each chunk's KV and Mamba state is backed up the moment it is stored; before sgl-project#41520 only the whole prompt was. On Qwen3.8-Flash-Next (--hicache-size 50, 4096-token chunks) that is about three times as many host Mamba writes per long prompt. The host Mamba pool (~224 slots) overflows, paused conversations lose their Mamba state on the host, a returning prefix has KV on host but no state on its path, and it is recomputed from token 0. Measured on the scratch box: 0/20 host-tier hits and 0% returning-turn reuse under a 25-minute agent load (147 turns), against 20/20 and 95.7% (1,065 turns) for the same series on the 2026-09-25 base. checkpoint_kv_cache passes its chunked flag on to insert_req; a chunked insert counts no hit (inserted_len = up_to) and leaves cache_inserted_len alone, so the first whole-prompt insert counts those nodes once. The other insert_req implementations take the keyword and ignore it.
Merged
5 tasks done
4 of 5 tasks
ormandj
added a commit
to ormandj/sglang
that referenced
this pull request
Oct 7, 2026
Main removed InsertParams.chunked when checkpointing moved to checkpoint_req (sgl-project#41520); the unfinished-request path is selected by is_finished=False alone.
4 of 5 tasks
caihuali95
pushed a commit
to caihuali95/sglang
that referenced
this pull request
Oct 9, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len and floors the frontier to a page. When the window is not a whole number of pages, that can leave less than a window between the frontier and page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1); at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64 tokens there, and with the default 128-token eviction interval and one token per step, every decode eviction lands on that residue. page_floor(pre_len) is where an EAGLE request's insert ends when the next verify finishes it with one token, since a bigram key spans one token less than the request owns. The new leaf then holds 64 live SWA tokens against a 127-token window, the match after the insert rejects it, and cache_protected_len stays at the previous prefix. The release then frees [cache_protected_len, owned), including the pages the insert just gave the tree, which the tree frees again when it evicts them. Since sgl-project#41520 folded the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64 leaked and double-freed KV on triton and trtllm_mha alike, and some runs hit an illegal memory access. The retained span is now max(window, page) rounded up to whole pages, so the frontier stays a full window below page_floor(pre_len). Page-aligned windows and page size 1 keep the old frontier; an unaligned window keeps at most one more SWA page per request. Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey runs a 7-token window at page size 4, the same misalignment. In test_finish_after_decode_eviction_frees_only_the_requests_pages, a request decodes, evicts at pre_len 23 and finishes at 24; before this change the full pool counts 8 tokens twice ("264 != 256"), after it every page is either free or held by the tree. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ormandj
added a commit
to ormandj/sglang
that referenced
this pull request
Oct 10, 2026
Main removed InsertParams.chunked when checkpointing moved to checkpoint_req (sgl-project#41520); the unfinished-request path is selected by is_finished=False alone.
4 of 5 tasks
2 tasks done
caihuali95
pushed a commit
to caihuali95/sglang
that referenced
this pull request
Oct 13, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len and floors the frontier to a page. When the window is not a whole number of pages, that can leave less than a window between the frontier and page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1); at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64 tokens there, and with the default 128-token eviction interval and one token per step, every decode eviction lands on that residue. page_floor(pre_len) is where an EAGLE request's insert ends when the next verify finishes it with one token, since a bigram key spans one token less than the request owns. The new leaf then holds 64 live SWA tokens against a 127-token window, the match after the insert rejects it, and cache_protected_len stays at the previous prefix. The release then frees [cache_protected_len, owned), including the pages the insert just gave the tree, which the tree frees again when it evicts them. Since sgl-project#41520 folded the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64 leaked and double-freed KV on triton and trtllm_mha alike, and some runs hit an illegal memory access. The retained span is now max(window, page) rounded up to whole pages, so the frontier stays a full window below page_floor(pre_len). Page-aligned windows and page size 1 keep the old frontier; an unaligned window keeps at most one more SWA page per request. Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey runs a 7-token window at page size 4, the same misalignment. In test_finish_after_decode_eviction_frees_only_the_requests_pages, a request decodes, evicts at pre_len 23 and finishes at 24; before this change the full pool counts 8 tokens twice ("264 != 256"), after it every page is either free or held by the tree. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jiminator
pushed a commit
that referenced
this pull request
Oct 16, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len and floors the frontier to a page. When the window is not a whole number of pages, that can leave less than a window between the frontier and page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1); at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64 tokens there, and with the default 128-token eviction interval and one token per step, every decode eviction lands on that residue. page_floor(pre_len) is where an EAGLE request's insert ends when the next verify finishes it with one token, since a bigram key spans one token less than the request owns. The new leaf then holds 64 live SWA tokens against a 127-token window, the match after the insert rejects it, and cache_protected_len stays at the previous prefix. The release then frees [cache_protected_len, owned), including the pages the insert just gave the tree, which the tree frees again when it evicts them. Since #41520 folded the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64 leaked and double-freed KV on triton and trtllm_mha alike, and some runs hit an illegal memory access. The retained span is now max(window, page) rounded up to whole pages, so the frontier stays a full window below page_floor(pre_len). Page-aligned windows and page size 1 keep the old frontier; an unaligned window keeps at most one more SWA page per request. Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey runs a 7-token window at page size 4, the same misalignment. In test_finish_after_decode_eviction_frees_only_the_requests_pages, a request decodes, evicts at pre_len 23 and finishes at 24; before this change the full pool counts 8 tokens twice ("264 != 256"), after it every page is either free or held by the tree. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit b15bf09)
5 tasks
ch-wan
pushed a commit
to caihuali95/sglang
that referenced
this pull request
Oct 18, 2026
free_swa_out_of_window_slots keeps max(window, page) tokens behind pre_len and floors the frontier to a page. When the window is not a whole number of pages, that can leave less than a window between the frontier and page_floor(pre_len). gpt-oss runs a 127-token window (sliding_window - 1); at page size 64, a decode eviction at pre_len = 63 mod 64 leaves only 64 tokens there, and with the default 128-token eviction interval and one token per step, every decode eviction lands on that residue. page_floor(pre_len) is where an EAGLE request's insert ends when the next verify finishes it with one token, since a bigram key spans one token less than the request owns. The new leaf then holds 64 live SWA tokens against a 127-token window, the match after the insert rejects it, and cache_protected_len stays at the previous prefix. The release then frees [cache_protected_len, owned), including the pages the insert just gave the tree, which the tree frees again when it evicts them. Since sgl-project#41520 folded the finished insert into checkpoint(), gpt-oss EAGLE3 at page size 64 leaked and double-freed KV on triton and trtllm_mha alike, and some runs hit an illegal memory access. The retained span is now max(window, page) rounded up to whole pages, so the frontier stays a full window below page_floor(pre_len). Page-aligned windows and page size 1 keep the old frontier; an unaligned window keeps at most one more SWA page per request. Test (test_unified_radix_cache_unittest.py, CPU): TestSWAWindowUnderBigramKey runs a 7-token window at page size 4, the same misalignment. In test_finish_after_decode_eviction_frees_only_the_requests_pages, a request decodes, evicts at pre_len 23 and finishes at 24; before this change the full pool counts 8 tokens twice ("264 != 256"), after it every page is either free or held by the tree.
2 tasks done
5 tasks
wl55721
added a commit
to wl55721/sglang
that referenced
this pull request
Oct 27, 2026
…mantics sgl-project#41520 counts a cache hit only when a request finishes, so hit_count advances by one per finished request instead of two. Request the victim prefix twice so it lands at hit_count == 2 (exactly the SLRU default threshold, below the configured 4), and update the phase-2 comment that described the pre-sgl-project#41520 double-count behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cache_unfinished_req(req, chunked)->checkpoint_req(req, *, up_to);maybe_cache_unfinished_req->checkpoint_kv_cache; callers no longer passchunkedInsertParams.chunked->record_end: only the insert ininsert_reqbumpshit_count(and checks the write-through threshold); checkpoints, prompt-split re-inserts and storage/linker fills do notwrite_through_thresholddefaults to 1 for both write-through policies, keeping the first-finished-request backup timing ofwrite_through_selectiveCI States
Latest PR Test (Base): ✅ Run #36938189711
Latest PR Test (Extra): 🚫 Run #36938189462
Latest PR Test (AMD ROCm 10): 🚫 Run #36938189741