Skip to content

[Scheduler] Track prefill progress as prefix_len - #42824

Merged
hnyls2002 merged 1 commit into
mainfrom
lsyin/prefix-len
Oct 7, 2026
Merged

hnyls2002 merged 1 commit into
mainfrom
lsyin/prefix-len

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator
  • Add Req.prefix_len as the prefill progress cursor; replace len(req.prefix_indices) readers
  • Cache checkpoint implementations no longer write the request's prefix; checkpoint_kv_cache advances prefix_len to the chunk end, published or not
  • alloc_for_extend reads the prefix from the row for requests that already hold one, and consumes prefix_indices for new ones

CI States

Latest PR Test (Base): 🚫 Run #37698422874
Latest PR Test (Extra): ❌ Run #37698422660
Latest PR Test (AMD ROCm 10): ⏳ Run #37698422788

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

Follow-up note (not in this PR): for requests that already hold their row (later chunks, borrowed session records), alloc_for_extend reads row[:prefix_len] and write_cache_indices writes it straight back to the same row. This round trip predates the PR (the copy used to happen at checkpoint) and is now just moved to allocation; letting the writer skip the prefix span for row holders would remove it, but that changes the Triton kernel inputs, so it is left for a separate change.

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 6, 2026
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test registered/chunked_prefill/test_scripted_core_1gpu.py registered/chunked_prefill/test_scripted_swa_1gpu.py registered/scheduler/test_mixed_chunked_prefill.py registered/scheduler/test_retract_decode.py registered/radix_cache/test_radix_cache_hit.py registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py registered/hicache/test_hicache_variants.py registered/hicache/test_hicache_storage_file_backend.py registered/sessions/test_streaming_session.py registered/dllm/test_dllm_batching_fdfo.py registered/disaggregation/test_disaggregation_basic.py registered/disaggregation/test_disaggregation_chunked_prefill_abort.py

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (4 tests): ✅ View workflow run

cd test/ && python3 registered/chunked_prefill/test_scripted_core_1gpu.py
cd test/ && python3 registered/scheduler/test_mixed_chunked_prefill.py
cd test/ && python3 registered/scheduler/test_retract_decode.py
cd test/ && python3 registered/radix_cache/test_radix_cache_hit.py

🚀 1-gpu-h100 (4 tests): ❌ View workflow run

cd test/ && python3 registered/chunked_prefill/test_scripted_swa_1gpu.py
cd test/ && python3 registered/hicache/test_hicache_variants.py
cd test/ && python3 registered/sessions/test_streaming_session.py
cd test/ && python3 registered/dllm/test_dllm_batching_fdfo.py

🚀 2-gpu-h100 (5 tests): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_full.py
cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_swa.py
cd test/ && python3 registered/hicache/test_hicache_storage_file_backend.py
cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py
cd test/ && python3 registered/disaggregation/test_disaggregation_chunked_prefill_abort.py

🚀 4-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mamba.py

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test registered/dllm/test_dllm_batching_fdfo.py registered/unit/mem_cache/test_hisparse_allocator.py

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/dllm/test_dllm_batching_fdfo.py

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/mem_cache/test_hisparse_allocator.py

@hnyls2002
hnyls2002 force-pushed the lsyin/match-writeback branch from 8ae6f07 to c134d31 Compare October 7, 2026 22:40
Base automatically changed from lsyin/match-writeback to main October 7, 2026 22:43
@hnyls2002
hnyls2002 merged commit e9240b3 into main Oct 7, 2026
85 of 112 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/prefix-len branch October 7, 2026 22:59
vincentzed pushed a commit to vincentzed/sglang that referenced this pull request Oct 8, 2026
main dropped Req.prefix_indices and Req.extend_range (sgl-project#42822, sgl-project#42824, sgl-project#42825)
and moved the cache match into match_kv_cache. The replay bookkeeping now
lives in match_kv_cache and uses req.prefix_len. The request stubs of two
scheduler unit tests get the new replay fields.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apple-silicon documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant