Repository navigation
fix(planner): reserve decode tokens in aggregated prefill estimates - #17417
JasperChan929 wants to merge 1 commit into
Conversation
Reserve ordinary decode query tokens before planning queued-prefill chunks and capacity TTFT. Use rank-local allowances for observed queues, retain the balanced synthetic DP distribution and total chunk ceiling, and return unavailable when queued work has no token headroom. Keep grouped evaluation of repeated full chunks and add regressions for mixed budgets, uneven DP ranks, saturation, and capacity TTFT eligibility. Fixes ai-dynamo#17416 Signed-off-by: wenzhi chen <wc3042@nyu.edu>
|
👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Per-rank queued-prefill budgeting components/src/dynamo/planner/core/perf_model/engine_query.py, components/src/dynamo/planner/tests/unit/test_engine_query.py |
Queued-prefill chunk limits account for scheduled decode requests on each rank. Empty queues return zero, and queued work without remaining capacity returns None. Tests cover chunk shapes, estimates, and grouping identical drain rounds. |
Aggregated TTFT capacity budgeting components/src/dynamo/planner/core/perf_model/engine_query.py, components/src/dynamo/planner/tests/unit/test_engine_query.py |
Aggregated TTFT chunks use the smaller of the global token limit and the remaining per-rank capacity after decode distribution. Tests cover rank counts, decode loads, and aggregate engine capacity. |
Priority:
Estimated code review effort: 3 (Moderate) | ~20 minutes
Severity of issue fixed: Medium
Merge Risk: 
Minimal · up to 8f187
-
No merge-blocking issue was established.
Pre-merge checks: 
4 · 
1
-
Failed checks: 1 warning Passed checks: 4 passed-
Full details: Description check
-
Explanation
The description is detailed and covers the change, validation, scope, reviewer-relevant files, limitations, and linked issue through “Fixes
#17416.” It does not use the template headings exactly and does not include a dedicated “Where should the reviewer start?” section, but the required information is mostly present.
Full details: Linked Issues check
-
Explanation
The PR meets the coding requirements in
#17416.get_queued_prefill_timereturns zero for empty queues, subtracts each rank's scheduled ordinary decode count for aggregated workers, returns unavailable when queued work has no usable capacity, and preserves grouped full-chunk pricing._aggregated_ttftapplies the balanced synthetic DP distribution and limits the total prefill chunk. Added tests cover single-rank and multi-rank budgets, unavailable cases, empty queues, grouping, and capacity TTFT. The reported scoped tests pass.
Full details: Out of Scope Changes check
-
Explanation
The reviewed changes stay within
#17416. The source changes update aggregated queued-prefill and capacity-TTFT query budgeting. The added tests verify those query behaviors. No change implements the separately identified caller fix in#15758/#15764 or the excluded speculative-width, scheduling-limit, sequence/KV-capacity, serving-deployment, or autoscaler work.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 8 functions across 2 files.
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Summary
Aggregated prefill estimates used the full iteration token budget before adding decode work. With a budget of 8 and six decodes, Planner could price eight more prefill tokens in the same step. Reserve the decode slots first so the queued-prefill and capacity-TTFT queries respect the per-rank ceiling.
Fixes #17416.
Validation
git diff --checkpass. Changed-file marker collection finds 34 tests with no missing marker sets.To run the scoped suite with the Planner dependencies installed:
Additional single-GPU prediction calibration
With Qwen3-0.6B/vLLM 0.32 on an RTX 5090 Laptop GPU, 72 independent training probes precede 180 held-out probes across budgets 64/256/1024 and 0/7/31 persistent decode streams. Both query versions share the same frozen native regression model per budget and predict the same recorded executions. Engine-API TTFT prediction MAE changes from 14.16ms to 10.63ms; the execution-interval-sum MAE changes from 6.96ms to 3.95ms. All 60 pure-prefill controls have identical predictions.
Nine short-background guard probes retain a counterexample: when decodes finish during prefill, execution-interval MAE worsens from 4.62ms to 10.63ms. The query still assumes a fixed decode population. These are prediction-error comparisons, not serving speedups, and do not establish online scaling, goodput or GPU-cost improvements.
Scope and related caller behavior
This reserves one query token per ordinary decode request. Speculative verification width, effective-scheduling-limit metadata, sequence/KV capacity modeling and a broader DP capacity redesign are outside this patch. No full Dynamo 1.7 serving deployment or real multi-GPU autoscaler A/B was run locally.
#15758/#15764 addresses a separate caller behavior: unavailable post-consolidation prefill estimates can be interpreted as no prefill signal. A constructed actual-Planner/native-model case with two workers, 32 decodes each and a 64-token ceiling reaches that path after this fix. Binding #15764's exact refusal function on the same state/model holds two workers. That existing author's changes are not included here. This PR can be reviewed independently; the caller interaction should inform merge ordering and any claim about complete scaling safety.
Summary by CodeRabbit