Skip to content

fix(planner): reserve decode tokens in aggregated prefill estimates - #17417

Open
JasperChan929 wants to merge 1 commit into
ai-dynamo:mainfrom
JasperChan929:fix/planner-agg-prefill-budget
Open

JasperChan929 wants to merge 1 commit into
ai-dynamo:mainfrom
JasperChan929:fix/planner-agg-prefill-budget

Conversation

@JasperChan929

@JasperChan929 JasperChan929 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Aggregated prefill estimates used the full iteration token budget before adding decode work. With a budget of 8 and six decodes, Planner could price eight more prefill tokens in the same step. Reserve the decode slots first so the queued-prefill and capacity-TTFT queries respect the per-rank ceiling.

  • Use each rank's scheduled ordinary-decode count for queued-prefill chunking.
  • Bound capacity TTFT under the existing balanced synthetic DP distribution, preserving its total chunk ceiling.
  • Keep empty queues at zero, return unavailable when queued work has no headroom, and retain grouped pricing for repeated full chunks.

Fixes #17416.

Validation

  • 17 new regression cases. Before the fix: 11 failures and 23 passing controls/existing cases. Afterward: all 34 engine-query cases pass.
  • 141 passed across engine query, AIS adapter, load/throughput scaling and the actual native AIS estimator integration tests.
  • Pinned formatting/lint/file hooks and git diff --check pass. Changed-file marker collection finds 34 tests with no missing marker sets.
  • Full pre-commit's global marker collector reports the same 155 collection-error files on this branch and untouched upstream in the local environment; that full check is not green.
  • Native AIS replay trained on real vLLM 0.32 GPU observations produces legal queried shapes and matches the legal-budget reference in the controlled Planner-state case. An additional 6,596 local reference comparisons cover uneven queues, exhausted budgets, reversed rank order and non-divisible synthetic DP splits.

To run the scoped suite with the Planner dependencies installed:

python -m pytest -q \
  components/src/dynamo/planner/tests/unit/test_engine_query.py \
  components/src/dynamo/planner/tests/unit/test_ais_perf_adapter.py \
  components/src/dynamo/planner/tests/unit/test_load_based_scaling.py \
  components/src/dynamo/planner/tests/unit/test_throughput_scaling.py \
  components/src/dynamo/planner/tests/integration/test_ais_native_estimator.py
Additional single-GPU prediction calibration

With Qwen3-0.6B/vLLM 0.32 on an RTX 5090 Laptop GPU, 72 independent training probes precede 180 held-out probes across budgets 64/256/1024 and 0/7/31 persistent decode streams. Both query versions share the same frozen native regression model per budget and predict the same recorded executions. Engine-API TTFT prediction MAE changes from 14.16ms to 10.63ms; the execution-interval-sum MAE changes from 6.96ms to 3.95ms. All 60 pure-prefill controls have identical predictions.

Nine short-background guard probes retain a counterexample: when decodes finish during prefill, execution-interval MAE worsens from 4.62ms to 10.63ms. The query still assumes a fixed decode population. These are prediction-error comparisons, not serving speedups, and do not establish online scaling, goodput or GPU-cost improvements.

Scope and related caller behavior

This reserves one query token per ordinary decode request. Speculative verification width, effective-scheduling-limit metadata, sequence/KV capacity modeling and a broader DP capacity redesign are outside this patch. No full Dynamo 1.7 serving deployment or real multi-GPU autoscaler A/B was run locally.

#15758/#15764 addresses a separate caller behavior: unavailable post-consolidation prefill estimates can be interpreted as no prefill signal. A constructed actual-Planner/native-model case with two workers, 32 decodes each and a 64-token ceiling reaches that path after this fix. Binding #15764's exact refusal function on the same state/model holds two workers. That existing author's changes are not included here. This PR can be reviewed independently; the caller interaction should inform merge ordering and any claim about complete scaling safety.

Summary by CodeRabbit

  • Bug Fixes
    • Improved estimates for queued prefill work when decode requests consume part of the available batch capacity.
    • Estimates now handle empty queues and per-rank capacity limits, including cases where queued work cannot fit.
  • Tests
    • Added coverage for prefill draining, aggregated capacity limits, and zero-work scenarios.

Reserve ordinary decode query tokens before planning queued-prefill chunks
and capacity TTFT. Use rank-local allowances for observed queues, retain
the balanced synthetic DP distribution and total chunk ceiling, and return
unavailable when queued work has no token headroom.

Keep grouped evaluation of repeated full chunks and add regressions for
mixed budgets, uneven DP ranks, saturation, and capacity TTFT eligibility.

Fixes ai-dynamo#17416

Signed-off-by: wenzhi chen <wc3042@nyu.edu>
@JasperChan929
JasperChan929 requested a review from a team as a code owner October 27, 2026 18:16
@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor planner labels Oct 27, 2026
@coderabbitai

coderabbitai Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between df083f8 and 8f18784.

Files selected for processing: 2
  • components/src/dynamo/planner/core/perf_model/engine_query.py
  • components/src/dynamo/planner/tests/unit/test_engine_query.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Queued-prefill and aggregate TTFT estimates now reserve token slots for scheduled decode requests when calculating prefill chunks. The changes also cover empty queues, unavailable capacity, and repeated chunk estimates.

Changes

Aggregated prefill capacity

Layer / File(s) Summary
Per-rank queued-prefill budgeting
components/src/dynamo/planner/core/perf_model/engine_query.py, components/src/dynamo/planner/tests/unit/test_engine_query.py
Queued-prefill chunk limits account for scheduled decode requests on each rank. Empty queues return zero, and queued work without remaining capacity returns None. Tests cover chunk shapes, estimates, and grouping identical drain rounds.
Aggregated TTFT capacity budgeting
components/src/dynamo/planner/core/perf_model/engine_query.py, components/src/dynamo/planner/tests/unit/test_engine_query.py
Aggregated TTFT chunks use the smaller of the global token limit and the remaining per-rank capacity after decode distribution. Tests cover rank counts, decode loads, and aggregate engine capacity.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 8f187

No merge-blocking issue was established.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surv… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: reserving decode tokens in aggregated prefill estimates.
Description check Passed The description is detailed and covers the change, validation, scope, reviewer-relevant files, limitations, and linked issue through “Fixes #17416.” It does not use the template headings exactly and d…
Linked Issues check Passed The PR meets the coding requirements in #17416. get_queued_prefill_time returns zero for empty queues, subtracts each rank's scheduled ordinary decode count for aggregated workers, returns unavailab…
Out of Scope Changes check Passed The reviewed changes stay within #17416. The source changes update aggregated queued-prefill and capacity-TTFT query budgeting. The added tests verify those query behaviors. No change implements the s…
Full details: Description check

Explanation

The description is detailed and covers the change, validation, scope, reviewer-relevant files, limitations, and linked issue through “Fixes #17416.” It does not use the template headings exactly and does not include a dedicated “Where should the reviewer start?” section, but the required information is mostly present.

Full details: Linked Issues check

Explanation

The PR meets the coding requirements in #17416. get_queued_prefill_time returns zero for empty queues, subtracts each rank's scheduled ordinary decode count for aggregated workers, returns unavailable when queued work has no usable capacity, and preserves grouped full-chunk pricing. _aggregated_ttft applies the balanced synthetic DP distribution and limits the total prefill chunk. Added tests cover single-rank and multi-rank budgets, unavailable cases, empty queues, grouping, and capacity TTFT. The reported scoped tests pass.

Full details: Out of Scope Changes check

Explanation

The reviewed changes stay within #17416. The source changes update aggregated queued-prefill and capacity-TTFT query budgeting. The added tests verify those query behaviors. No change implements the separately identified caller fix in #15758/#15764 or the excluded speculative-width, scheduling-limit, sequence/KV-capacity, serving-deployment, or autoscaler work.

Full details: Docstring Coverage

Explanation

Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 8 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

This branch was previously deployed

1 inactive deployment
external_collaborator — 8f187848 Deployed Oct 27, 2026 by JasperChan929 via ok-to-test #39686
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix planner size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Aggregated prefill estimates reuse token slots occupied by decode

1 participant