Repository navigation
fix(planner): honor vLLM effective scheduling token limit - #17512
JasperChan929 wants to merge 3 commits into
Conversation
Publish a smaller resolved vLLM scheduling limit through existing native runtime metadata and carry it into Planner capabilities and query limits. Keep raw batched capacity for the AIS forward model, preserve legacy defaults, and include the optional ceiling in query-model reuse identity. Cover registration, metadata consumption, capacity and attention-DP boundaries without changing the separate mixed-decode budget accounting. Closes ai-dynamo#17509 Signed-off-by: wenzhi chen <wc3042@nyu.edu>
|
👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Publish vLLM scheduling metadata components/src/dynamo/vllm/capacity.py, components/src/dynamo/vllm/main.py, components/src/dynamo/vllm/tests/* |
Model registration publishes a validated scheduled-token limit in runtime metadata when it is below the batched-token limit. Tests cover missing, invalid, and larger or equal limits. |
Carry the limit into Planner capabilities components/src/dynamo/planner/monitoring/worker_info.py, components/src/dynamo/planner/connectors/mdc.py, components/src/dynamo/planner/core/types.py, components/src/dynamo/planner/core/base.py, components/src/dynamo/planner/core/util.py, components/src/dynamo/planner/environment/base.py, components/src/dynamo/planner/tests/unit/test_mdc.py, components/src/dynamo/planner/tests/unit/test_environment.py |
Worker metadata and engine capabilities gain the optional scheduled-token field. Worker-info change detection and refresh now include it. |
Apply the scheduling budget in Planner components/src/dynamo/planner/core/perf_model/*, components/src/dynamo/planner/tests/unit/test_ais_perf_adapter.py, components/src/dynamo/planner/tests/unit/test_engine_query.py, components/src/dynamo/planner/tests/integration/test_scheduling_limit_metadata.py, docs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md |
Performance limits validate the optional value and use the lesser of scheduled and batched-token limits as the scheduling budget. Prefill chunking and capacity calculations use this budget. Decode capacity also applies a budget-based cap when the scheduled-token limit is set. |
Priority:
Estimated code review effort: 3 (Moderate) | ~25 minutes
Severity of issue fixed: Medium
Merge Risk: 
Minimal · up to ca15a
-
No actionable merge-blocking issue is established. The scheduling-limit change is mergeable after normal checks in a matching native-binding environment.
Pre-merge checks: 
4 · 
1
-
Failed checks: 1 warning Passed checks: 4 passed-
Full details: Linked Issues check
-
Explanation
The PR satisfies the coding requirements in [
#17509]. vLLM publishes a smaller validmax_num_scheduled_tokensvalue through existing runtime metadata. MDC,WorkerInfo, capabilities, and Planner change detection carry the value independently frommax_num_batched_tokens. Planner usesmin(batched, scheduled)for scheduling geometry and per-rank decode limits. AIS options retain raw batched capacity. Query model keys include the scheduling limit, and retained observations replay after a change. Missing, equal, larger, legacy, and invalid-value paths have coverage in the reported unit and integration tests.
Full details: Out of Scope Changes check
-
Explanation
The changes remain within [
#17509]. The modified production files implement metadata publication, transport, validation, query limits, and model invalidation. The added tests verify those paths. The guide documents the static representative-worker behavior and the stated limits. No change implements the separate work in [#12021] or [#17417], and no unrelated serving hot-path, Kubernetes polling, speculative-width, or broader DP-capacity work is present.
Full details: Description check
-
Explanation
The description has a clear Overview, Key Changes, Validation, and Issues and Ongoing Work section, and it links the work to issue
#17509with a closing reference. It covers the change, the test evidence, and the scope limits. The main gap is that the Overview and Key Changes mix in dense technical detail, and the Validation table lists several claims that are hard to verify from the diff alone. The template's checklist item is marked as done. Overall the description is mostly complete and matches the template.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 21.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 23 functions across 17 files. (1 skipped: 1 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches: 💡 1
-
🛠️ Fix failing CI checks: 💡
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
| return min( | ||
| self._limits.max_num_seqs, | ||
| self._limits.max_num_batched_tokens // max(1, isl), | ||
| self._limits.scheduling_token_budget // max(1, isl), |
There was a problem hiding this comment.
This treats the per-rank scheduling ceiling as an engine-wide prefill ceiling. The capacity query passes an aggregate isl * batch_size to _synthetic_prefill_by_rank, which splits it across attention-DP ranks, so with DP=2, limit=64, and ISL=32 vLLM can schedule four requests (64 tokens per rank) while Planner searches only two. This underestimates prefill capacity whenever attention DP is greater than one.
🤖 AI Fix
Use a saturating aggregate prefill scheduling budget of scheduling_token_budget * attention_dp_size for synthetic prefill capacity and timing calculations, while retaining the per-rank budget for real per-rank queue-drain metrics.
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """An independent scheduling ceiling survives native registration into Planner.""" |
There was a problem hiding this comment.
This module docstring only restates the file and sole test’s purpose; it records no additional constraint or rationale.
🤖 AI Fix
Remove the redundant module docstring.
| assert info.max_num_batched_tokens == 256 | ||
| assert info.max_num_scheduled_tokens == 64 | ||
|
|
||
| @pytest.mark.parametrize("backend", ["vllm", "sglang", "trtllm"]) |
There was a problem hiding this comment.
The sglang and trtllm cases cannot distinguish this regression from the vllm case: backend only selects defaults, while _card() supplies both asserted capacity fields and the new scheduling-metadata extraction is backend-independent. Retaining the vllm case still protects the supported legacy-card behavior that missing runtime metadata yields no scheduling override.
🤖 AI Fix
Replace this backend parametrization with a single vllm case, removing the sglang and trtllm variants.
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """Scheduling metadata retains model capacity across the native JSON boundary.""" |
There was a problem hiding this comment.
This module docstring merely summarizes behavior already expressed by the test names and assertions, without preserving a non-obvious reason or contract.
🤖 AI Fix
Remove the redundant module docstring.
Signed-off-by: wenzhi chen <wc3042@nyu.edu>
Scale the independent per-rank scheduling ceiling for synthetic pure prefill timing and capacity, retaining the legacy total chunk ceiling. Keep observed rank queues and mixed TTFT chunk defaults unchanged. Add regressions for DP1/2/4 capacity, aggregate timing, the legacy total ceiling, real queues and mixed timing. Remove redundant test docstrings and backend variants identified in review. Refs ai-dynamo#17509 Signed-off-by: wenzhi chen <wc3042@nyu.edu>
Overview
Planner receives vLLM's model-input ceiling but loses a separately configured, smaller scheduling ceiling. With batched256/scheduled64, a 192-token prefill is modeled as one chunk although vLLM schedules three. Publish the independent ceiling and use it for Planner scheduling geometry while retaining raw model-input capacity for AISimulate.
Key Changes
Core logic adds91 nonblank, noncomment lines across the existing metadata/query path; no Rust schema or binding changes.
capacity.py/main.py: publish a smaller resolved scheduler limit through existing runtime metadata; absent, equal and larger limits retain the existing wire representation.mdc.py: carry the optional value through WorkerInfo, capability mapping and change detection without replacing the batched value.engine_query.py/ais_adapter.py: usemin(batched, scheduled)for scheduling limits, bound decode batches per attention-DP rank, pool pure synthetic prefill budgets without widening the legacy total ceiling, and invalidate/replay the query model when the independent limit changes. AIS model-input options remain raw batched capacity.Validation
235 scoped tests passed, including43 new cases. Key regressions are
test_prefill_chunks_use_scheduling_limit,test_decode_scheduling_limit_applies_per_attention_dp_rank,test_scheduling_change_rebuilds_queries_without_shrinking_ais_capacity, andtest_native_scheduling_limit_reaches_planner. Additional guards cover synthetic prefill DP1/2/4, the legacy total ceiling, observed rank queues and unchanged mixed TTFT chunking. The legacy MDC fixture preserves the prior fallback.The baseline replay uses the same native-emitted card and recorded GPU observations through the actual baseline/fixed consumers and exact AISimulate pin
0.13.0.dev202610240000000082; it fails the64-token query assertion before the fix and passes afterward. Earlier actual vLLM0.32/Qwen3-0.6B observations independently confirm[64,64,64]for this configuration. These are query-correctness checks, not serving-speed or autoscaler A/B results.Local native checks use the available runtime1.6dev binding and compatible Python facade with current Planner sources. They exercise real file discovery, Rust metadata serialization and the AIS native estimator, but not a rebuilt runtime1.7 or complete
register_vllm_modelserving path. The entrypoint call is statically reviewed; producer/helper and transport are exercised separately and together in the native regression.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit