Skip to content

fix(planner): normalize speculative OP query geometry - #17772

Draft
JasperChan929 wants to merge 3 commits into
ai-dynamo:mainfrom
JasperChan929:fix/planner-speculative-query-geometry
Draft

JasperChan929 wants to merge 3 commits into
ai-dynamo:mainfrom
JasperChan929:fix/planner-speculative-query-geometry

Conversation

@JasperChan929

@JasperChan929 JasperChan929 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Planner sends logical decode counts to AIS's packed OP interface. With speculative ngram K3, one request can reach AIS as batch_size=1 with scale_num_tokens=4, making capacity estimates unavailable; divisible batches can silently price the wrong work. Pack queries after DP partitioning and respect per-rank verification budgets while preserving overloaded workers' scaling votes.

Draft: merge requires #15764 / #15758, or an equivalent reviewed consolidation-safety guard. The dependency is still open. Known budget saturation is handled here; genuinely unavailable consolidation estimates still need that separate caller fix.

Key Changes

Core logic: 265 added nonblank, noncomment, nondocstring lines across three performance-query/scaling files; high complexity. No new package, public configuration flag or telemetry field.

  • engine_query.py: use one verify_width and two mode/backend flags. Pack scheduled decode count/KV per rank, preserve logical observations and u32 overflow checks, and reserve full configured vLLM verification work. Public load queries return an explicit saturation status rather than None; infeasible capacity candidates are skipped without discarding earlier feasible results. Unsupported decode/aggregated widths larger than a rank's whole budget are rejected; prefill-only workers keep chunked prefill support.
  • load_scaling.py: separate saturated workers from finite latency values. All-saturated groups can scale up under the existing all-workers policy; a healthy/saturated mixture holds. Current and post-consolidation saturation veto shrink, including the aggregate combiner's missing-signal fallback. Duration diagnostics remain finite or unavailable, without fabricated infinity values. Genuine native/model None remains missing data.
  • ais_adapter.py: retain raw logical observations, withhold speculative decode-containing OP iterations whose actual verification widths are unknown, and invalidate duration feedback on a successful width change. Same-width estimator/limit rebuilds replay raw history; transient initialization failures retain it. The operator guide now contains configuration and limitations, with unsupported snapshot-regeneration instructions removed.

Validation

Current revision: 1,621 tests pass: 1,490 Planner unit tests, 29 real AIS estimator/profile integration tests and 102 orchestrator tests. The native tests use AISimulate 0.13.0.dev202610290000000087, Python 3.12 and installed ai-dynamo-runtime==1.7.0, with offline model metadata and no GPU/model-weight load.

Case Before With this change
One speculative logical decode, or divisible native batch Missing prediction or finite prediction for the wrong work Finite public query/capacity; matches the native static decode oracle
Per-rank decode/queued prefill budgets Verification is undercharged or budget failures erase votes Legal native shapes; explicit saturation retained by load scaling
Both workers: scheduled 3 + queued 2 decodes, B16/W4 Live scheduled work fits, but combined-query None loses scale-up signal Real OrchestratorEngineAdapter.tick: 2→3, scale_up
One healthy worker and one saturated worker The hot vote can disappear into fallback behavior Hold 2, consolidation refused
Current work fits; consolidation exhausts decode and prefill headroom Unavailable post estimate does not reliably carry a veto Hold 2, consolidation refused
All workers cold/genuinely unavailable Insufficient data Still insufficient data, without saturation-driven scale-up
Verification width changes in regression Old-width durations must not replay Same-width limit rebuild retains prediction; changed width becomes cold; new samples match a clean new-width reference

The six new public native/orchestrator rows run against the unchanged prior PR head a2ba064d: five fail and the cold-model control passes; all six pass after this revision. The prefill-no-headroom and decode-only consumer rows are included. Individual new native cases take about 10–20 ms locally. Existing independent rank budget, missing-model, empty-queue, failed-rebuild and same-width OP→regression replay controls remain covered. Tests now explain the oracle's 126 + 2//2 + 1 = 128 context and the per-rank learned prefill mean (8 + 100) / 2 = 54, and use public load queries for the changed prediction assertions.

Width invalidation intentionally covers every mode: FPM duration samples lack verification-width identity. Native regression loses readiness until retrained; native interpolation retains its base model but loses learned correction. This revision adds native regression transition evidence; it does not claim a native speculative interpolation width-transition experiment.

These are local CPU/native checks, not hosted CI or measured multi-GPU serving performance. Earlier controlled replay and query-overhead numbers were collected on a prior head and are not claimed as measurements of this revision. Budget saturation is a conservative scaling-pressure signal for an infeasible configured full round, not a measured SLA violation. Configured full rounds still approximate partial/draftless rounds; actual-width decode calibration needs additional telemetry. DP>1 synthetic capacity's existing independent count/KV partitioning defect is outside this fix.

Remaining Review/API Point

AIS87 provenance exposes the resolved config and selected estimator but not resolved verification width; the latest official AIS source checked (f3713ba3) also lacks that export. The existing canonical SDK build_spec_scheme(None, SpeculationConfig(...)) is retained to resolve geometry; None is supported for this SDK operation. Removing this import/helper and marker-collection stub requires an additive upstream provenance field or getter, followed by a compatible pin update. This reviewer suggestion is therefore not claimed complete. No alternate width formula or nonexistent field is introduced.

Issues and Ongoing Work

AI assistance was used for implementation, tests and review.

Signed-off-by: wenzhi chen <wc3042@nyu.edu>
@copy-pr-bot

copy-pr-bot Bot commented Nov 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 1, 2026 13:24 — with GitHub Actions Inactive
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 1, 2026 13:24 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation planner labels Nov 1, 2026
Signed-off-by: wenzhi chen <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 1, 2026 13:47 — with GitHub Actions Inactive
@JasperChan929
JasperChan929 marked this pull request as ready for review November 1, 2026 13:52
@JasperChan929
JasperChan929 requested review from a team as code owners November 1, 2026 13:52
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The Planner now resolves speculative decode width for AIS performance estimates, scales OP query inputs, and checks verification work against token budgets. It also excludes decode-containing observations from OP tuning and clears timing history when the active width changes.

Changes

Speculative decode performance modeling

Layer / File(s) Summary
Resolve width and normalize AIS inputs
components/src/dynamo/planner/core/perf_model/engine_query.py, components/src/dynamo/planner/tests/unit/test_engine_query.py, components/src/dynamo/planner/tests/integration/test_ais_native_estimator.py, tests/report_pytest_markers.py
The model derives speculative width and estimation mode from resolved estimator provenance. OP-mode AIS inputs scale decode request and KV-token counts by width, with invalid or overflowing values rejected. Decode-containing iterations are excluded from native OP tuning. Added tests cover packed inputs, logical inputs for regression and interpolation, filtering, overflow, and provenance checks.
Apply width-aware token budgets
components/src/dynamo/planner/core/perf_model/engine_query.py, components/src/dynamo/planner/tests/unit/test_engine_query.py, components/src/dynamo/planner/tests/integration/test_ais_native_estimator.py
Queue-drain, mixed-ITL, TTFT, capacity, and forward-pass estimates account for width-scaled decode work. Estimates return None when scheduled work or queued prefill cannot fit the applicable budget. Tests cover rank-local headroom, capacity, prefill cropping, and TTFT verification work.
Handle width changes and calibration history
components/src/dynamo/planner/core/perf_model/ais_adapter.py, components/src/dynamo/planner/tests/unit/test_ais_perf_adapter.py, components/src/dynamo/planner/tests/integration/test_ais_native_estimator.py, docs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md, docs/fern/translations/zh-CN/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md
The adapter clears retained and pending iterations when the engine model’s decode width changes. Tests cover failed initialization, successful rebuilds, and withheld observation history. The guide and Chinese translation document speculative OP estimates, calibration limits, feedback history, and snapshot guidance.

Estimated code review effort: 4 (Complex) | ~45 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to a2ba0

No confirmed issue currently blocks merging. The speculative native fallback remains unverified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: normalizing speculative OP query geometry in the Planner.
Linked Issues check Passed The PR addresses the coding requirements in directly linked issue #17770. AISCoreEnginePerfModel.best_available() derives the configured verification width from resolved AIS provenance. `_fpm_to_ais…
Out of Scope Changes check Passed The changed production files implement #17770. The AIS adapter history handling supports width changes introduced by the normalization. The tests verify the new packing, budget, calibration, and lifec…
Description check Passed The description states the problem (logical decode counts sent to AIS packed OP interface produce unavailable or wrong-work estimates for speculative ngram K3) and the intended fix (pack queries after…
Full details: Linked Issues check

Explanation

The PR addresses the coding requirements in directly linked issue #17770. AISCoreEnginePerfModel.best_available() derives the configured verification width from resolved AIS provenance. _fpm_to_ais_dict() packs decode request and KV counts for the AIS interface. Query paths enforce the configured vLLM verification-token budget per attention-DP rank and reject infeasible or overflowing work. Logical load and physical KV accounting remain separate. Decode-containing observations are excluded from native OP tuning, while load averages and pure-prefill calibration remain available. The PR adds focused unit and integration regressions for packing, budgets, rank handling, overflow, calibration, and history changes. The separate #15758/#15764 consolidation guard is not a requirement of #17770 and is explicitly outside this PR.

Full details: Out of Scope Changes check

Explanation

The changed production files implement #17770. The AIS adapter history handling supports width changes introduced by the normalization. The tests verify the new packing, budget, calibration, and lifecycle behavior. The Planner documentation and marker-only AIS stub support the implementation and its test coverage. No unrelated production behavior or separate #15758/#15764 implementation is included.

Full details: Description check

Explanation

The description states the problem (logical decode counts sent to AIS packed OP interface produce unavailable or wrong-work estimates for speculative ngram K3) and the intended fix (pack queries after DP partitioning, respect per-rank verification budgets, preserve scaling votes). It also includes Key Changes, Validation with concrete test counts and a before/after table, and Issues and Ongoing Work with 'Closes #17770' and merge dependencies. The template's Overview, Key Changes, Validation, and Issues sections are all present and substantively filled. The required 'I used Dynamo's /pr-checklist skill' checkbox is absent from the description, and the description is very long and dense, with some detail (e.g., unsupported snapshot-regeneration notes, reviewer-suggestion status) that goes beyond what the template asks for. These are non-critical gaps that do not make the description largely incomplete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 6bcaabc and a2ba064.

Files selected for processing: 8

  • components/src/dynamo/planner/core/perf_model/ais_adapter.py
  • components/src/dynamo/planner/core/perf_model/engine_query.py
  • components/src/dynamo/planner/tests/integration/test_ais_native_estimator.py
  • components/src/dynamo/planner/tests/unit/test_ais_perf_adapter.py
  • components/src/dynamo/planner/tests/unit/test_engine_query.py
  • docs/fern/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md
  • docs/fern/translations/zh-CN/pages/developer-guide/knowledge-base/modular-components/planner/planner-guide.md
  • tests/report_pytest_markers.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says this is unsafe without #15764, but nothing stops the merge button. Can you mark it draft or stack it on that branch until it lands?

def _estimate_forward_pass_seconds(
self, metrics_by_rank: list[ForwardPassMetrics]
) -> Optional[float]:
if self._budget_decode_width > 1 and any(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When prefill + decode × width exceeds the budget this returns None, and _scale_decision drops None estimates. So the most loaded workers stop voting for scale-up, and all of them over budget reads as insufficient_data. Could this price a clamped round or report saturation instead? Same at L329 and L627.

Comment on lines +200 to +202
decode_query_width: int = 1,
pack_decode_queries: bool = False,
budget_decode_width: int = 1,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three fields that are each 'W or 1' are hard to follow. One verify_width plus two booleans would say the same thing.

Comment on lines +770 to +774
def _configured_decode_width(config: dict[str, Any]) -> int:
speculation = config.get("speculation")
if speculation:
return build_spec_scheme(None, SpeculationConfig(**speculation)).verify_width()
return config.get("nextn", 0) + 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds an AIS scheme with a None model config just to read an int, and adds an SDK import plus a CI stub. Can AIS provenance report the width so Planner doesn't re-derive it?

Comment on lines +154 to +162
width = self._engine_model.decode_query_width
if (
self._history_decode_width is not None
and self._history_decode_width != width
):
# A different verification width describes different measured work.
self._retained_iterations.clear()
self._pending_iterations.clear()
self._history_decode_width = width

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also clears history for fpm_regression and fpm_interpolation, so a speculative_nextn update leaves those models cold. Intended? The title only covers OP queries.

Comment on lines +210 to +219
for name, value in (
("decode_query_width", decode_query_width),
("budget_decode_width", budget_decode_width),
):
if (
isinstance(value, bool)
or not isinstance(value, int)
or not 1 <= value <= _U32_MAX
):
raise ValueError(f"{name} must be a positive u32, got {value}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep the u32 overflow check; drop the isinstance bool/int/dict guards on values from our own typed structs. Also L246-252 and L759-765.

worker_type="decode",
speculation={"kind": "ngram", "params": {"num_speculative_tokens": 3}},
)
expected_ms = oracle.decode_step_latency(4, 126, 2)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do 126 and 2 come from, and [54, 54]? A one-line derivation would make these checkable. Prefer the public query methods over _aggregated_ttft and _estimate_forward_pass_seconds. ([54, 54] is test_engine_query.py L233.)

Comment on lines +116 to +133
For speculative `op_level` models, Planner converts logical decode requests to
the configured verification work at the AIS query boundary. Load metrics and
physical KV capacity remain logical. For vLLM, each attention-DP rank's target
work must fit the published `max_num_batched_tokens`: three draft tokens mean
four verification positions per decode request. These full-round estimates do
not infer tighter scheduler limits or method-specific extra drafting slots.

Current FPMs do not report actual decode query widths. Iterations containing
speculative decode therefore do not calibrate OP estimates; pure-prefill
correction remains available. Regression and measured complete-round FPM models
keep their logical inputs and training. Raw telemetry still updates load
averages. Changing verification width clears retained duration feedback;
same-execution, same-width limit or estimator changes can replay it. Keep each
Planner role tied to a stable worker execution identity.

Regenerate OP corrections in snapshots that older Planner versions tuned from
speculative decode FPMs. Externally calibrated AIS snapshots with packed
telemetry retain their existing convention.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads like PR notes. The guide never defines these 'snapshots' or how to regenerate them. Trim to what an operator has to do.

@JasperChan929
JasperChan929 marked this pull request as draft November 13, 2026 20:36
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 13, 2026 20:54 — with GitHub Actions Inactive

This branch was previously deployed

1 inactive deployment
external_collaborator — ecfa5379 Deployed Nov 13, 2026 by JasperChan929 via ok-to-test #51167
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix planner size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Planner speculative OP queries use logical counts with the packed AIS FPM interface

2 participants