Repository navigation
Conversation
aylonsk
temporarily deployed
to
external_collaborator
October 6, 2026 21:52 — with
GitHub Actions
Inactive
aylonsk
temporarily deployed
to
external_collaborator
October 6, 2026 21:52 — with
GitHub Actions
Inactive
Contributor
|
👋 Hi aylonsk! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
Contributor
…eck fails Before an agg scale-down, _agg_prefill_scaling re-predicts TTFT with one worker's load moved onto the survivors. When that post-consolidation query returned None, it cleared can_scale_down without marking the decision refused and returned None. _advance_load_agg reads None as "prefill signal unavailable" and lets decode scale down alone, so a deployment whose TTFT estimate was within SLA but whose safety check could not be evaluated went from three replicas to two. The AIS adapter returns None when the engine query raises, and also, without a log, when the regression store the query maps to has no fit yet. The refused path already returns num_workers so that the caller does not mistake it for a missing signal, and the decode side blocks scale-down when its post-consolidation ITL cannot be evaluated. Treat an unevaluable check as a refusal, and log it, for any worker that has a TTFT estimate. A worker without an estimate, for example an idle one whose query lands on an unfitted pure-prefill store, carries no prefill signal: it no longer stops the loop, so it neither blocks scale-down nor hides a later worker's refusal. With no estimate at all the function still returns None and decode decides alone, as before. Signed-off-by: Akira Lonske <akiralonske@gmail.com>
aylonsk
force-pushed
the
aylonsk/planner-agg-consolidation-unknown
branch
from
October 14, 2026 03:25
de02820 to
eaa0fea
Compare
aylonsk
temporarily deployed
to
external_collaborator
October 14, 2026 03:25 — with
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
In agg SLA mode, a prefill consolidation check that couldn't be evaluated was read as "no prefill signal", so decode was allowed to scale the deployment down on its own. This PR treats an unevaluable check as a refusal for any worker that has a TTFT estimate, as the decode side already blocks scale-down when its own check can't be evaluated. With the real regression model, I could only make the refusal change the scaling action in attention-DP groups. For a single-rank worker, the decode side's own check fails on the same unfit store at the same tick; see the end of Details.
Summary
_agg_prefill_scaling, aNonepost-consolidation estimate for a worker with a live TTFT estimate now setsconsolidation_unknown. The function then returnsnum_workerswithscale_down_refused_consolidation, which is the existing refused path. The case is logged at INFO with the worker label.Noneand decode decides alone, as before.Details:
PlannerEnginePerfModel.estimate_queued_prefill_timereturnsNonein two cases. One is when the AIS query raises. The other, with no log, is when the regression store the query maps to has no fit yet; AISimulate's regression models "returnNoneuntil the selected store has a ready fit". Against the pinned wheel, a model tuned only on mixed prefill+decode iterations reportsreadywhile itspure_prefillstore is unfit.The post-consolidation query can map to a different store than the estimate. Adding a worker's queued decode moves it from
pure_prefilltocontains_locally_mixed. In an attention-DP group, the rescaled queues can also add an iteration with a prefill-only rank beside a decode-only rank, which maps tocross_rank_aggregated.That case drives the per-worker rule. An idle worker's next-request query is pure prefill. If an unevaluable check blocked scale-down regardless of whether the worker had an estimate, an idle replica could never be drained.
Decisions from
PlannerScalingState.advance_loadwith a mocked perf model (hold= no decision):mainThe last two rows are side effects of the change, and both are deliberate:
main, a worker without an estimate stops the loop early, so a later worker's real refusal is never seen.cancel_startupis gated on the refusal, as it is for refused checks today.An unevaluable check is reported with the same
scale_down_refused_consolidationreason as a failed one.no_changewould re-enable startup cancellation throughany_refused, and a separate Enum state would mean extending the append-only state list. The INFO line tells the two cases apart in logs.How often this changes a decision: I replayed simulated vLLM-like agg workers through
PlannerScalingStatewith the real regression model and evaluated both versions on the same fitted model each tick. The simulation is local and not part of this PR. It covered 38,844 ticks: 3 workers, 30-minute runs, chat, long-context, prefill-heavy and KVBM-offload workloads, and 3 seeds.WAITING_FOR_REMOTE_KVS._compute_queuedcounts that as queued decode. In a prefill-only workload with host-tier prefix hits, it covered 2,160 ticks.no_changebecamescale_down_refused_consolidationat 1,191 ticks.attention_dp=2and onlycross_rank_aggregatedunfit,mainscales 2 → 1 on decode alone and this PR holds.Validation
test_agg_unevaluable_prefill_check_holds_only_with_an_estimateintest_pending_startup_scaling.py, with three rows:main.pytest components/src/dynamo/planner/testsin a Linux container, rebased onmainat c872a52: 1675 passed, 2 failed. The same 2 tests (test_wait_for_unacknowledged_decision) fail onmainbecause the container has noetcdbinary. My container's runtime and AISimulate wheels predate currentmain, so the two offline replay test files (test_replay_adapter_fpm.py,test_replay_warmup.py) can't import there; they fail the same way onmain.mypy --explicit-package-bases components/src/dynamo/planner(the CI invocation, scoped to the planner): no new errors. The 6 it reports in my container also appear onmain(missing stubs, andurl_validator.py:293).Where should the reviewer start?
components/src/dynamo/planner/core/load_scaling.py:_agg_prefill_scaling.Related Issues
Summary by CodeRabbit