Repository navigation
Conversation
8900980 to
7a9c9a3
Compare
|
Hi @imReese, I noticed #35645 after commenting on #35642. I have prepared an updated version against current main with scheduler-level regression tests covering exhausted and remaining chunk budgets. Are you still planning to continue this PR? I would be happy to contribute the tests, or open an updated replacement PR with credit to #35645 if you no longer plan to continue. |
c2b3f6b to
f5a7393
Compare
|
Rebased this PR onto current Validation:
@Kangwenqiao thanks for the tests and for agreeing to consolidate here. #35645 is now updated; please close #37686 as discussed. @merrymercy @hnyls2002 @Ying1123 @xiezhq-hermann could you please review and trigger |
e15acea to
c05ee4e
Compare
|
Following up on the overlap with #38164: I removed its chunk-budget helper and loop guard, leaving admission handling here. Thanks to @imReese and @Kangwenqiao for the implementation and tests. I reproduced a positive sub-page remainder on Follow-up commit is directly on your current head and can be cherry-picked. It checks the initial admission budget even without a resumed chunk, treats less than one page as exhausted in the autoregressive budget state, and preserves your NO_TOKEN/cache/Mamba handling. The tests retain your scaffold and use real Req and PrefillAdder admission methods, with cache access and batch preparation mocked. Existing author credits are preserved. Author Linux CPU run at CUDA_VISIBLE_DEVICES=9 PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_scheduler_chunk_budget.py test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_scheduler_init_req_max_new_tokens.pyThe combined patchset with the narrowed #38164 on main |
c05ee4e to
42868a3
Compare
|
Updated this PR with @ormandj's follow-up and rebased it onto current The new patch covers the positive sub-page remainder cases as well as admission without a resumed chunk, while preserving the existing Local validation on the new head
The PR is mergeable again. |
42868a3 to
852116f
Compare
|
Thanks @imReese, @Kangwenqiao, and @ormandj for chasing this. The cost #35642 reported, host load-back started for a candidate that a spent chunk budget then rejects, is gone on main: since #39283 the admission shape is selected before any load-back, and a spent budget rejects the candidate there. What remained was harmless bookkeeping, and #44420 now stops the admission loop at a spent pass budget before the first candidate, without a new place that sets |
Motivation
Fixes #35642.
A resumed chunked request can consume the remaining chunk budget for the current scheduling pass. The scheduler nevertheless continues scanning the waiting queue, so requests can perform prefix-cache matching and host-cache load-back before
add_one_req()rejects them because no chunk tokens remain.Modifications
NO_TOKENtakes precedence over chunk exhaustion inbudget_state(), preserving the existingbatch_is_fullbehavior for hierarchical cache and the unified-cache external linker.Accuracy Tests
This change does not modify model computation or output values.
test/registered/unit/managers/test_scheduler_chunk_budget.py: 3 tests passed in a macOS CPU logic run with import-only platform stubs.python -m py_compile python/sglang/srt/managers/scheduler.py test/registered/unit/managers/test_scheduler_chunk_budget.pypre-commit run --from-ref upstream/main --to-ref HEADNative local collection is unavailable because the macOS environment does not provide Triton. Repository CI is still required for the pinned Linux environment and broader coverage.
Speed Tests and Profiling
No benchmark was run. The change removes unnecessary waiting-queue cache matching and possible host-cache load-back work from scheduling passes with an exhausted chunk budget.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #35705394490
Latest PR Test (Extra): ❌ Run #35705393503
Latest PR Test (AMD ROCm 10): ❌ Run #35705394072