Skip to content

fix(planner): honor vLLM effective scheduling token limit - #17512

Open
JasperChan929 wants to merge 3 commits into
ai-dynamo:mainfrom
JasperChan929:fix/planner-effective-scheduling-limit
Open

JasperChan929 wants to merge 3 commits into
ai-dynamo:mainfrom
JasperChan929:fix/planner-effective-scheduling-limit

Conversation

@JasperChan929

@JasperChan929 JasperChan929 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Planner receives vLLM's model-input ceiling but loses a separately configured, smaller scheduling ceiling. With batched256/scheduled64, a 192-token prefill is modeled as one chunk although vLLM schedules three. Publish the independent ceiling and use it for Planner scheduling geometry while retaining raw model-input capacity for AISimulate.

Key Changes

Core logic adds91 nonblank, noncomment lines across the existing metadata/query path; no Rust schema or binding changes.

  1. capacity.py / main.py: publish a smaller resolved scheduler limit through existing runtime metadata; absent, equal and larger limits retain the existing wire representation.
  2. mdc.py: carry the optional value through WorkerInfo, capability mapping and change detection without replacing the batched value.
  3. engine_query.py / ais_adapter.py: use min(batched, scheduled) for scheduling limits, bound decode batches per attention-DP rank, pool pure synthetic prefill budgets without widening the legacy total ceiling, and invalidate/replay the query model when the independent limit changes. AIS model-input options remain raw batched capacity.

Validation

Behavior Before After
Same actual native-emitted MDC, batched256/scheduled64, queued192 Consumer drops64; native AIS is queried with192 Query64, priced as three equal rounds; raw AIS capacity stays256
Existing file-discovery registration through the actual publisher, native MDC and Planner Smaller limit has no consumer path Integer64 survives transport and controls the query
Default, equal or larger configured limit Existing behavior 270 baseline/fixed reference controls remain identical
Decode budget64 on DP1/2/4 No independent scheduling bound Total batches64/128/256 remain available, each rank at most64
Pure synthetic prefill, DP2, budget64/rank, ISL32 No independent scheduling bound Four requests fit the128-token aggregate step, with64 tokens per rank; observed rank queues keep the64-token ceiling
Only the scheduling limit changes No field in query-model identity Query limits rebuild and retained observations replay
Explicit invalid limit Not represented Estimator stays unavailable; it does not silently use a larger fallback

235 scoped tests passed, including43 new cases. Key regressions are test_prefill_chunks_use_scheduling_limit, test_decode_scheduling_limit_applies_per_attention_dp_rank, test_scheduling_change_rebuilds_queries_without_shrinking_ais_capacity, and test_native_scheduling_limit_reaches_planner. Additional guards cover synthetic prefill DP1/2/4, the legacy total ceiling, observed rank queues and unchanged mixed TTFT chunking. The legacy MDC fixture preserves the prior fallback.

The baseline replay uses the same native-emitted card and recorded GPU observations through the actual baseline/fixed consumers and exact AISimulate pin 0.13.0.dev202610240000000082; it fails the64-token query assertion before the fix and passes afterward. Earlier actual vLLM0.32/Qwen3-0.6B observations independently confirm [64,64,64] for this configuration. These are query-correctness checks, not serving-speed or autoscaler A/B results.

Local native checks use the available runtime1.6dev binding and compatible Python facade with current Planner sources. They exercise real file discovery, Rust metadata serialization and the AIS native estimator, but not a rebuilt runtime1.7 or complete register_vllm_model serving path. The entrypoint call is statically reviewed; producer/helper and transport are exercised separately and together in the native regression.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • New Features
    • Planner capacity estimates and prefill chunking now account for vLLM’s scheduled-token limit when it is available, separately from the batched-token limit.
    • Scheduling-limit updates are reflected in planner estimates; invalid limit values are rejected.
  • Documentation
    • Updated the Planner guide with scheduling-limit behavior, worker-limit considerations, and guidance for applying limit changes.

Publish a smaller resolved vLLM scheduling limit through existing native
runtime metadata and carry it into Planner capabilities and query limits.
Keep raw batched capacity for the AIS forward model, preserve legacy
defaults, and include the optional ceiling in query-model reuse identity.

Cover registration, metadata consumption, capacity and attention-DP
boundaries without changing the separate mixed-decode budget accounting.

Closes ai-dynamo#17509

Signed-off-by: wenzhi chen <wc3042@nyu.edu>
@JasperChan929
JasperChan929 requested a review from a team as a code owner October 28, 2026 08:52
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@JasperChan929
JasperChan929 requested review from a team as code owners October 28, 2026 08:52
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator October 28, 2026 08:52 — with GitHub Actions Inactive
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator October 28, 2026 08:52 — with GitHub Actions Inactive
@JasperChan929
JasperChan929 requested review from a team as code owners October 28, 2026 08:52
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend planner labels Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between f38fb53 and ca15adb.

Files selected for processing: 18
  • components/src/dynamo/planner/connectors/mdc.py
  • components/src/dynamo/planner/core/base.py
  • components/src/dynamo/planner/core/perf_model/ais_adapter.py
  • components/src/dynamo/planner/core/perf_model/engine_query.py
  • components/src/dynamo/planner/core/types.py
  • components/src/dynamo/planner/core/util.py
  • components/src/dynamo/planner/environment/base.py
  • components/src/dynamo/planner/monitoring/worker_info.py
  • components/src/dynamo/planner/tests/integration/test_scheduling_limit_metadata.py
  • components/src/dynamo/planner/tests/unit/test_ais_perf_adapter.py
  • components/src/dynamo/planner/tests/unit/test_engine_query.py
  • components/src/dynamo/planner/tests/unit/test_environment.py
  • components/src/dynamo/planner/tests/unit/test_mdc.py
  • components/src/dynamo/vllm/capacity.py
  • components/src/dynamo/vllm/main.py
  • components/src/dynamo/vllm/tests/test_scheduling_budget.py
  • components/src/dynamo/vllm/tests/test_vllm_unit.py
  • docs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

vLLM now publishes its scheduled-token limit separately when that limit is below its batched-token limit. Planner carries the optional value through worker metadata and capabilities, then uses it in prefill chunking and capacity estimates.

Changes

Scheduling Limit Propagation and Planning

Layer / File(s) Summary
Publish vLLM scheduling metadata
components/src/dynamo/vllm/capacity.py, components/src/dynamo/vllm/main.py, components/src/dynamo/vllm/tests/*
Model registration publishes a validated scheduled-token limit in runtime metadata when it is below the batched-token limit. Tests cover missing, invalid, and larger or equal limits.
Carry the limit into Planner capabilities
components/src/dynamo/planner/monitoring/worker_info.py, components/src/dynamo/planner/connectors/mdc.py, components/src/dynamo/planner/core/types.py, components/src/dynamo/planner/core/base.py, components/src/dynamo/planner/core/util.py, components/src/dynamo/planner/environment/base.py, components/src/dynamo/planner/tests/unit/test_mdc.py, components/src/dynamo/planner/tests/unit/test_environment.py
Worker metadata and engine capabilities gain the optional scheduled-token field. Worker-info change detection and refresh now include it.
Apply the scheduling budget in Planner
components/src/dynamo/planner/core/perf_model/*, components/src/dynamo/planner/tests/unit/test_ais_perf_adapter.py, components/src/dynamo/planner/tests/unit/test_engine_query.py, components/src/dynamo/planner/tests/integration/test_scheduling_limit_metadata.py, docs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md
Performance limits validate the optional value and use the lesser of scheduled and batched-token limits as the scheduling budget. Prefill chunking and capacity calculations use this budget. Decode capacity also applies a budget-based cap when the scheduled-token limit is set.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to ca15a

No actionable merge-blocking issue is established. The scheduling-limit change is mergeable after normal checks in a matching native-binding environment.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 21.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: making Planner honor vLLM's effective scheduling token limit.
Linked Issues check Passed The PR satisfies the coding requirements in [#17509]. vLLM publishes a smaller valid max_num_scheduled_tokens value through existing runtime metadata. MDC, WorkerInfo, capabilities, and Planner ch…
Out of Scope Changes check Passed The changes remain within [#17509]. The modified production files implement metadata publication, transport, validation, query limits, and model invalidation. The added tests verify those paths. The g…
Description check Passed The description has a clear Overview, Key Changes, Validation, and Issues and Ongoing Work section, and it links the work to issue #17509 with a closing reference. It covers the change, the test evide…
Full details: Linked Issues check

Explanation

The PR satisfies the coding requirements in [#17509]. vLLM publishes a smaller valid max_num_scheduled_tokens value through existing runtime metadata. MDC, WorkerInfo, capabilities, and Planner change detection carry the value independently from max_num_batched_tokens. Planner uses min(batched, scheduled) for scheduling geometry and per-rank decode limits. AIS options retain raw batched capacity. Query model keys include the scheduling limit, and retained observations replay after a change. Missing, equal, larger, legacy, and invalid-value paths have coverage in the reported unit and integration tests.

Full details: Out of Scope Changes check

Explanation

The changes remain within [#17509]. The modified production files implement metadata publication, transport, validation, query limits, and model invalidation. The added tests verify those paths. The guide documents the static representative-worker behavior and the stated limits. No change implements the separate work in [#12021] or [#17417], and no unrelated serving hot-path, Kubernetes polling, speculative-width, or broader DP-capacity work is present.

Full details: Description check

Explanation

The description has a clear Overview, Key Changes, Validation, and Issues and Ongoing Work section, and it links the work to issue #17509 with a closing reference. It covers the change, the test evidence, and the scope limits. The main gap is that the Overview and Key Changes mix in dense technical detail, and the Validation table lists several claims that are hard to verify from the diff alone. The template's checklist item is marked as done. Overall the description is mostly complete and matches the template.

Full details: Docstring Coverage

Explanation

Docstring coverage is 21.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 23 functions across 17 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

return min(
self._limits.max_num_seqs,
self._limits.max_num_batched_tokens // max(1, isl),
self._limits.scheduling_token_budget // max(1, isl),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This treats the per-rank scheduling ceiling as an engine-wide prefill ceiling. The capacity query passes an aggregate isl * batch_size to _synthetic_prefill_by_rank, which splits it across attention-DP ranks, so with DP=2, limit=64, and ISL=32 vLLM can schedule four requests (64 tokens per rank) while Planner searches only two. This underestimates prefill capacity whenever attention DP is greater than one.

🤖 AI Fix

Use a saturating aggregate prefill scheduling budget of scheduling_token_budget * attention_dp_size for synthetic prefill capacity and timing calculations, while retaining the per-rank budget for real per-rank queue-drain metrics.

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""An independent scheduling ceiling survives native registration into Planner."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This module docstring only restates the file and sole test’s purpose; it records no additional constraint or rationale.

🤖 AI Fix

Remove the redundant module docstring.

assert info.max_num_batched_tokens == 256
assert info.max_num_scheduled_tokens == 64

@pytest.mark.parametrize("backend", ["vllm", "sglang", "trtllm"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sglang and trtllm cases cannot distinguish this regression from the vllm case: backend only selects defaults, while _card() supplies both asserted capacity fields and the new scheduling-metadata extraction is backend-independent. Retaining the vllm case still protects the supported legacy-card behavior that missing runtime metadata yields no scheduling override.

🤖 AI Fix

Replace this backend parametrization with a single vllm case, removing the sglang and trtllm variants.

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Scheduling metadata retains model capacity across the native JSON boundary."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This module docstring merely summarizes behavior already expressed by the test names and assertions, without preserving a non-obvious reason or contract.

🤖 AI Fix

Remove the redundant module docstring.

Signed-off-by: wenzhi chen <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator October 28, 2026 09:12 — with GitHub Actions Inactive
Scale the independent per-rank scheduling ceiling for synthetic pure
prefill timing and capacity, retaining the legacy total chunk ceiling.
Keep observed rank queues and mixed TTFT chunk defaults unchanged.

Add regressions for DP1/2/4 capacity, aggregate timing, the legacy total
ceiling, real queues and mixed timing. Remove redundant test docstrings
and backend variants identified in review.

Refs ai-dynamo#17509

Signed-off-by: wenzhi chen <wc3042@nyu.edu>

This branch was previously deployed

1 inactive deployment
external_collaborator — e65edeb6 Deployed Oct 28, 2026 by JasperChan929 via ok-to-test #40615
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix planner size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Planner misses vLLM's effective scheduling token limit

1 participant