Skip to content

fix(planner): refuse agg scale-down when the prefill consolidation check fails - #15764

Open
aylonsk wants to merge 1 commit into
ai-dynamo:mainfrom
aylonsk:aylonsk/planner-agg-consolidation-unknown
Open

aylonsk wants to merge 1 commit into
ai-dynamo:mainfrom
aylonsk:aylonsk/planner-agg-consolidation-unknown

Conversation

@aylonsk

@aylonsk aylonsk commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

In agg SLA mode, a prefill consolidation check that couldn't be evaluated was read as "no prefill signal", so decode was allowed to scale the deployment down on its own. This PR treats an unevaluable check as a refusal for any worker that has a TTFT estimate, as the decode side already blocks scale-down when its own check can't be evaluated. With the real regression model, I could only make the refusal change the scaling action in attention-DP groups. For a single-rank worker, the decode side's own check fails on the same unfit store at the same tick; see the end of Details.

Summary

  • In _agg_prefill_scaling, a None post-consolidation estimate for a worker with a live TTFT estimate now sets consolidation_unknown. The function then returns num_workers with scale_down_refused_consolidation, which is the existing refused path. The case is logged at INFO with the worker label.
  • A worker without a TTFT estimate carries no prefill signal. It no longer stops the loop, so it neither blocks scale-down nor hides a later worker's refusal.
  • With no estimate at all, the function still returns None and decode decides alone, as before.

Details:

PlannerEnginePerfModel.estimate_queued_prefill_time returns None in two cases. One is when the AIS query raises. The other, with no log, is when the regression store the query maps to has no fit yet; AISimulate's regression models "return None until the selected store has a ready fit". Against the pinned wheel, a model tuned only on mixed prefill+decode iterations reports ready while its pure_prefill store is unfit.

The post-consolidation query can map to a different store than the estimate. Adding a worker's queued decode moves it from pure_prefill to contains_locally_mixed. In an attention-DP group, the rescaled queues can also add an iteration with a prefill-only rank beside a decode-only rank, which maps to cross_rank_aggregated.

That case drives the per-worker rule. An idle worker's next-request query is pure prefill. If an unevaluable check blocked scale-down regardless of whether the worker had an estimate, an idle replica could never be drained.

Decisions from PlannerScalingState.advance_load with a mocked perf model (hold = no decision):

scenario main this PR
3 workers, live estimate, check unevaluable 3 → 2 hold
A healthy; B live estimate, check unevaluable (either order) 2 → 1 hold
no prefill estimate at all 3 → 2 3 → 2
A, B healthy; C idle with no estimate and an unevaluable check (either order) 3 → 2 3 → 2
A healthy; C no estimate; B refused (C first) 3 → 2 hold
3 ready + 1 starting, live estimate, check unevaluable 3 → 2, startup cancelled hold

The last two rows are side effects of the change, and both are deliberate:

  • C first, then a refusal: on main, a worker without an estimate stops the loop early, so a later worker's real refusal is never seen.
  • Pending startup: the hold also keeps a pending replica, because cancel_startup is gated on the refusal, as it is for refused checks today.

An unevaluable check is reported with the same scale_down_refused_consolidation reason as a failed one. no_change would re-enable startup cancellation through any_refused, and a separate Enum state would mean extending the append-only state list. The INFO line tells the two cases apart in logs.

How often this changes a decision: I replayed simulated vLLM-like agg workers through PlannerScalingState with the real regression model and evaluated both versions on the same fitted model each tick. The simulation is local and not part of this PR. It covered 38,844 ticks: 3 workers, 30-minute runs, chat, long-context, prefill-heavy and KVBM-offload workloads, and 3 seeds.

  • The case showed up only with KVBM, whose async onboarding vLLM reports as WAITING_FOR_REMOTE_KVS. _compute_queued counts that as queued decode. In a prefill-only workload with host-tier prefix hits, it covered 2,160 ticks.
  • The scaling action was the same at every tick, with or without a starting replica. For a single-rank worker, the decode side's own check uses the same unfit store at the same tick, since aggregated ITL queries add the learned average prefill. What did change was the reported reason: no_change became scale_down_refused_consolidation at 1,191 ticks.
  • A constructed attention-DP state does change: with attention_dp=2 and only cross_rank_aggregated unfit, main scales 2 → 1 on decode alone and this PR holds.

Validation

  • test_agg_unevaluable_prefill_check_holds_only_with_an_estimate in test_pending_startup_scaling.py, with three rows:
    • every worker has a live estimate and an unevaluable check: hold. This row fails on main.
    • no estimate at all: decode decides.
    • an idle worker without an estimate next to healthy workers: scale down.
  • pytest components/src/dynamo/planner/tests in a Linux container, rebased on main at c872a52: 1675 passed, 2 failed. The same 2 tests (test_wait_for_unacknowledged_decision) fail on main because the container has no etcd binary. My container's runtime and AISimulate wheels predate current main, so the two offline replay test files (test_replay_adapter_fpm.py, test_replay_warmup.py) can't import there; they fail the same way on main.
  • mypy --explicit-package-bases components/src/dynamo/planner (the CI invocation, scoped to the planner): no new errors. The 6 it reports in my container also appear on main (missing stubs, and url_validator.py:293).
  • pre-commit hooks on the commit: passed.

Where should the reviewer start?

components/src/dynamo/planner/core/load_scaling.py: _agg_prefill_scaling.

Related Issues

Summary by CodeRabbit

  • Bug Fixes
    • Improved scaling decisions when prefill-time estimates are unavailable. If a pre-consolidation estimate exists but a post-consolidation estimate cannot be evaluated, scale-down is refused and reported as a consolidation refusal. If no pre-consolidation estimate is available, the missing post-consolidation estimate alone no longer blocks scale-down.
    • Added coverage for aggregate scaling scenarios with missing or partial estimates.

@aylonsk
aylonsk requested a review from a team as a code owner October 6, 2026 21:52
@copy-pr-bot

copy-pr-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@aylonsk
aylonsk temporarily deployed to external_collaborator October 6, 2026 21:52 — with GitHub Actions Inactive
@aylonsk
aylonsk temporarily deployed to external_collaborator October 6, 2026 21:52 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

👋 Hi aylonsk! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor planner labels Oct 6, 2026
@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 1aca6068-aa13-4959-8593-801a750e9b09
📥 Commits

Reviewing files that changed from the base of the PR and between 7d82c10 and de02820.

📒 Files selected for processing (2)
  • components/src/dynamo/planner/core/load_scaling.py
  • components/src/dynamo/planner/tests/unit/test_pending_startup_scaling.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

Aggregate SLA scaling now distinguishes an unavailable post-consolidation estimate from the absence of a prefill estimate. The change also adds parameterized tests for decode scaling decisions across these estimate conditions.

Changes

Aggregate prefill scaling

Layer / File(s) Summary
Consolidation estimate decision
components/src/dynamo/planner/core/load_scaling.py, components/src/dynamo/planner/tests/unit/test_pending_startup_scaling.py
If a pre-consolidation estimate exists but the post-consolidation estimate is unavailable, the decision is marked unknown and scale-down is refused. If no pre-consolidation estimate exists, the unavailable post-estimate does not independently block scale-down. Tests cover unavailable estimates and expected decode decisions.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to de028

The reported scale-down refusal protects the TTFT budget when a post-consolidation estimate is available. No identified issue remains that should prevent merging after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Issue [#15758] requires an unevaluable post-consolidation TTFT check to refuse scale-down when a worker has a TTFT estimate, while allowing decode-only decisions when prefill has no estimate. In `_agg…
Out of Scope Changes check Passed The PR changes _agg_prefill_scaling in load_scaling.py and adds focused aggregate-mode SLA coverage in test_pending_startup_scaling.py. Both changes implement or verify issue [#15758]. The diff …
Title check Passed The title clearly describes the main change: preventing aggregate scale-down when the prefill consolidation check cannot safely pass. The wording uses “fails” broadly, but it remains accurate and spec…
Description check Passed The description includes all required sections, explains the behavior change and edge cases, identifies the review starting point, documents validation results, and links issue #15758.
  • Fix all pre-merge checks with AI
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

…eck fails

Before an agg scale-down, _agg_prefill_scaling re-predicts TTFT with one
worker's load moved onto the survivors. When that post-consolidation
query returned None, it cleared can_scale_down without marking the
decision refused and returned None. _advance_load_agg reads None as
"prefill signal unavailable" and lets decode scale down alone, so a
deployment whose TTFT estimate was within SLA but whose safety check
could not be evaluated went from three replicas to two. The AIS adapter
returns None when the engine query raises, and also, without a log, when
the regression store the query maps to has no fit yet.

The refused path already returns num_workers so that the caller does not
mistake it for a missing signal, and the decode side blocks scale-down
when its post-consolidation ITL cannot be evaluated.

Treat an unevaluable check as a refusal, and log it, for any worker that
has a TTFT estimate. A worker without an estimate, for example an idle
one whose query lands on an unfitted pure-prefill store, carries no
prefill signal: it no longer stops the loop, so it neither blocks
scale-down nor hides a later worker's refusal. With no estimate at all
the function still returns None and decode decides alone, as before.

Signed-off-by: Akira Lonske <akiralonske@gmail.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — eaa0fea7 Deployed Oct 14, 2026 by aylonsk via ok-to-test #32136
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix planner size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Agg load scaling reads an unevaluable prefill consolidation check as "no prefill signal" and scales down on decode alone

1 participant