Skip to content

[Scheduler] Stop waiting-queue scan after chunk budget exhaustion - #35645

Closed
imReese wants to merge 3 commits into
sgl-project:mainfrom
imReese:fix/chunk-budget-waiting-scan
Closed

imReese wants to merge 3 commits into
sgl-project:mainfrom
imReese:fix/chunk-budget-waiting-scan

Conversation

@imReese

@imReese imReese commented Aug 20, 2026 •

Copy link
Copy Markdown

Motivation

Fixes #35642.

A resumed chunked request can consume the remaining chunk budget for the current scheduling pass. The scheduler nevertheless continues scanning the waiting queue, so requests can perform prefix-cache matching and host-cache load-back before add_one_req() rejects them because no chunk tokens remain.

Modifications

  • Recheck the prefill budget after adding a resumed chunked request.
  • Skip waiting-request admission when the chunk budget is exhausted while preserving the resumed chunk in the runnable batch.
  • Handle the case where NO_TOKEN takes precedence over chunk exhaustion in budget_state(), preserving the existing batch_is_full behavior for hierarchical cache and the unified-cache external linker.
  • Avoid Mamba waiting-request preallocation when admission is skipped.
  • Add scheduler-level CPU regression coverage for exhausted, partially available, and simultaneously exhausted chunk/total-token budgets. The tests were contributed by @Kangwenqiao in [Scheduler] Stop waiting-queue scan after exhausted chunk budget #37686 and adapted here for the current scheduler APIs.

Accuracy Tests

This change does not modify model computation or output values.

  • test/registered/unit/managers/test_scheduler_chunk_budget.py: 3 tests passed in a macOS CPU logic run with import-only platform stubs.
  • Negative control against the pre-fix scheduler: the two exhausted-budget cases failed and the remaining-budget case passed, confirming that the tests detect the regression.
  • python -m py_compile python/sglang/srt/managers/scheduler.py test/registered/unit/managers/test_scheduler_chunk_budget.py
  • pre-commit run --from-ref upstream/main --to-ref HEAD

Native local collection is unavailable because the macOS environment does not provide Triton. Repository CI is still required for the pinned Linux environment and broader coverage.

Speed Tests and Profiling

No benchmark was run. The change removes unnecessary waiting-queue cache matching and possible host-cache load-back work from scheduling passes with an exhausted chunk budget.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #35705394490
Latest PR Test (Extra): ❌ Run #35705393503
Latest PR Test (AMD ROCm 10): ❌ Run #35705394072

@Dami-wq-k03

Copy link
Copy Markdown

Hi @imReese, I noticed #35645 after commenting on #35642. I have prepared an updated version against current main with scheduler-level regression tests covering exhausted and remaining chunk budgets. Are you still planning to continue this PR? I would be happy to contribute the tests, or open an updated replacement PR with credit to #35645 if you no longer plan to continue.

@Dami-wq-k03

Copy link
Copy Markdown

Follow-up: I opened #37686 as the updated replacement, rebased onto current main with focused scheduler regression tests. The implementation credits #35645 and includes @imReese as co-author.

@imReese
imReese force-pushed the fix/chunk-budget-waiting-scan branch from c2b3f6b to f5a7393 Compare September 4, 2026 06:05
@imReese

imReese commented Sep 4, 2026

Copy link
Copy Markdown
Author

Rebased this PR onto current main (7825e5ffca) and added the scheduler-level regression tests from #37686 as a separate commit authored by @Kangwenqiao. I also adapted the fix/tests for the latest dynamic_chunk_sizer and unified-cache external-linker changes.

Validation:

  • focused scheduler tests: 3 passed in the macOS CPU/import-stub logic environment
  • negative control on the pre-fix scheduler: 2 exhausted-budget cases failed, 1 remaining-budget case passed
  • complete changed-range pre-commit: passed

@Kangwenqiao thanks for the tests and for agreeing to consolidate here. #35645 is now updated; please close #37686 as discussed.

@merrymercy @hnyls2002 @Ying1123 @xiezhq-hermann could you please review and trigger /tag-and-rerun-ci? Native Linux/pinned-dependency CI is still needed.

@ormandj

ormandj commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Following up on the overlap with #38164: I removed its chunk-budget helper and loop guard, leaving admission handling here. Thanks to @imReese and @Kangwenqiao for the implementation and tests.

I reproduced a positive sub-page remainder on c05ee4e37b with mixed prefill/decode. With page_size=64 and chunked_prefill_size=128, one decode token leaves 127 tokens; a resumed 64-token prefill leaves 63. The waiting request still reaches init_next_round_input before add_one_req rejects it. The same unnecessary scan occurs without a resumed chunk and after the first waiting prefill consumes a page.

Follow-up commit is directly on your current head and can be cherry-picked. It checks the initial admission budget even without a resumed chunk, treats less than one page as exhausted in the autoregressive budget state, and preserves your NO_TOKEN/cache/Mamba handling. The tests retain your scaffold and use real Req and PrefillAdder admission methods, with cache access and batch preparation mocked. Existing author credits are preserved.

Author Linux CPU run at 3878952b1b: 48 tests and 585 subtests passed across the three files below. With the same tests but unchanged runtime source from c05ee4e37b, three test methods and six subtests fail at the unwanted waiting-request scan. Coverage includes page boundaries, resumed-chunk preservation, radix/hierarchical/external-linker batch-full behavior, Mamba preallocation, total-token exhaustion, the first-request input-budget exception and the separate DLLM budget. Environment: Python 3.12.3, PyTorch 2.13.0+cu130, Transformers 5.16.0.dev0, Tokenizers 0.23.1; CUDA hidden.

CUDA_VISIBLE_DEVICES=9 PYTHONPATH=python python -m pytest -q test/registered/unit/managers/test_scheduler_chunk_budget.py test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_scheduler_init_req_max_new_tokens.py

The combined patchset with the narrowed #38164 on main 554f817948 passed 114 CPU tests and 587 subtests. Lint passed. No GPU, full-model or speed result is claimed. The latest Base CI run on your head stopped at the missing run-ci label, so upstream CI is still needed. Please feel free to incorporate or adapt the follow-up here.

imReese commented Sep 22, 2026 •

Copy link
Copy Markdown
Author

Updated this PR with @ormandj's follow-up and rebased it onto current main.

The new patch covers the positive sub-page remainder cases as well as admission without a resumed chunk, while preserving the existing NO_TOKEN / cache-mode / Mamba handling. I also adapted the regression fixture to the current scheduler/prefill-budget APIs after the rebase.

Local validation on the new head 42868a325c:

  • test_scheduler_chunk_budget.py, test_prefill_adder.py, and test_scheduler_init_req_max_new_tokens.py: 60 passed, 601 subtests passed
  • py_compile on the touched scheduler/policy/test files: passed
  • pre-commit run --from-ref upstream/main --to-ref HEAD: passed

The PR is mergeable again.

@hnyls2002

Copy link
Copy Markdown
Collaborator

Thanks @imReese, @Kangwenqiao, and @ormandj for chasing this. The cost #35642 reported, host load-back started for a candidate that a spent chunk budget then rejects, is gone on main: since #39283 the admission shape is selected before any load-back, and a spent budget rejects the candidate there. What remained was harmless bookkeeping, and #44420 now stops the admission loop at a spent pass budget before the first candidate, without a new place that sets batch_is_full or a page-size threshold in budget_state. Closing as superseded; please reopen if I've missed a case.

@hnyls2002 hnyls2002 closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Scheduler scans waiting queue after resumed chunk exhausts chunk budget

4 participants