Repository navigation
fix(planner): normalize speculative OP query geometry - #17772
JasperChan929 wants to merge 3 commits into
Conversation
Signed-off-by: wenzhi chen <wc3042@nyu.edu>
|
👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
Signed-off-by: wenzhi chen <wc3042@nyu.edu>
|
No actionable comments were generated in the recent review. WalkthroughThe Planner now resolves speculative decode width for AIS performance estimates, scales OP query inputs, and checks verification work against token budgets. It also excludes decode-containing observations from OP tuning and clears timing history when the active width changes. ChangesSpeculative decode performance modeling
Estimated code review effort: 4 (Complex) | ~45 minutes Severity of issue fixed: Medium Merge Risk:
|
jthomson04
left a comment
There was a problem hiding this comment.
The description says this is unsafe without #15764, but nothing stops the merge button. Can you mark it draft or stack it on that branch until it lands?
| def _estimate_forward_pass_seconds( | ||
| self, metrics_by_rank: list[ForwardPassMetrics] | ||
| ) -> Optional[float]: | ||
| if self._budget_decode_width > 1 and any( |
There was a problem hiding this comment.
When prefill + decode × width exceeds the budget this returns None, and _scale_decision drops None estimates. So the most loaded workers stop voting for scale-up, and all of them over budget reads as insufficient_data. Could this price a clamped round or report saturation instead? Same at L329 and L627.
| decode_query_width: int = 1, | ||
| pack_decode_queries: bool = False, | ||
| budget_decode_width: int = 1, |
There was a problem hiding this comment.
Three fields that are each 'W or 1' are hard to follow. One verify_width plus two booleans would say the same thing.
| def _configured_decode_width(config: dict[str, Any]) -> int: | ||
| speculation = config.get("speculation") | ||
| if speculation: | ||
| return build_spec_scheme(None, SpeculationConfig(**speculation)).verify_width() | ||
| return config.get("nextn", 0) + 1 |
There was a problem hiding this comment.
This builds an AIS scheme with a None model config just to read an int, and adds an SDK import plus a CI stub. Can AIS provenance report the width so Planner doesn't re-derive it?
| width = self._engine_model.decode_query_width | ||
| if ( | ||
| self._history_decode_width is not None | ||
| and self._history_decode_width != width | ||
| ): | ||
| # A different verification width describes different measured work. | ||
| self._retained_iterations.clear() | ||
| self._pending_iterations.clear() | ||
| self._history_decode_width = width |
There was a problem hiding this comment.
This also clears history for fpm_regression and fpm_interpolation, so a speculative_nextn update leaves those models cold. Intended? The title only covers OP queries.
| for name, value in ( | ||
| ("decode_query_width", decode_query_width), | ||
| ("budget_decode_width", budget_decode_width), | ||
| ): | ||
| if ( | ||
| isinstance(value, bool) | ||
| or not isinstance(value, int) | ||
| or not 1 <= value <= _U32_MAX | ||
| ): | ||
| raise ValueError(f"{name} must be a positive u32, got {value}") |
There was a problem hiding this comment.
Keep the u32 overflow check; drop the isinstance bool/int/dict guards on values from our own typed structs. Also L246-252 and L759-765.
| worker_type="decode", | ||
| speculation={"kind": "ngram", "params": {"num_speculative_tokens": 3}}, | ||
| ) | ||
| expected_ms = oracle.decode_step_latency(4, 126, 2) |
There was a problem hiding this comment.
Where do 126 and 2 come from, and [54, 54]? A one-line derivation would make these checkable. Prefer the public query methods over _aggregated_ttft and _estimate_forward_pass_seconds. ([54, 54] is test_engine_query.py L233.)
| For speculative `op_level` models, Planner converts logical decode requests to | ||
| the configured verification work at the AIS query boundary. Load metrics and | ||
| physical KV capacity remain logical. For vLLM, each attention-DP rank's target | ||
| work must fit the published `max_num_batched_tokens`: three draft tokens mean | ||
| four verification positions per decode request. These full-round estimates do | ||
| not infer tighter scheduler limits or method-specific extra drafting slots. | ||
|
|
||
| Current FPMs do not report actual decode query widths. Iterations containing | ||
| speculative decode therefore do not calibrate OP estimates; pure-prefill | ||
| correction remains available. Regression and measured complete-round FPM models | ||
| keep their logical inputs and training. Raw telemetry still updates load | ||
| averages. Changing verification width clears retained duration feedback; | ||
| same-execution, same-width limit or estimator changes can replay it. Keep each | ||
| Planner role tied to a stable worker execution identity. | ||
|
|
||
| Regenerate OP corrections in snapshots that older Planner versions tuned from | ||
| speculative decode FPMs. Externally calibrated AIS snapshots with packed | ||
| telemetry retain their existing convention. |
There was a problem hiding this comment.
This reads like PR notes. The guide never defines these 'snapshots' or how to regenerate them. Trim to what an operator has to do.
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
Overview
Planner sends logical decode counts to AIS's packed OP interface. With speculative ngram K3, one request can reach AIS as
batch_size=1withscale_num_tokens=4, making capacity estimates unavailable; divisible batches can silently price the wrong work. Pack queries after DP partitioning and respect per-rank verification budgets while preserving overloaded workers' scaling votes.Draft: merge requires #15764 / #15758, or an equivalent reviewed consolidation-safety guard. The dependency is still open. Known budget saturation is handled here; genuinely unavailable consolidation estimates still need that separate caller fix.
Key Changes
Core logic: 265 added nonblank, noncomment, nondocstring lines across three performance-query/scaling files; high complexity. No new package, public configuration flag or telemetry field.
engine_query.py: use oneverify_widthand two mode/backend flags. Pack scheduled decode count/KV per rank, preserve logical observations and u32 overflow checks, and reserve full configured vLLM verification work. Public load queries return an explicit saturation status rather thanNone; infeasible capacity candidates are skipped without discarding earlier feasible results. Unsupported decode/aggregated widths larger than a rank's whole budget are rejected; prefill-only workers keep chunked prefill support.load_scaling.py: separate saturated workers from finite latency values. All-saturated groups can scale up under the existing all-workers policy; a healthy/saturated mixture holds. Current and post-consolidation saturation veto shrink, including the aggregate combiner's missing-signal fallback. Duration diagnostics remain finite or unavailable, without fabricated infinity values. Genuine native/modelNoneremains missing data.ais_adapter.py: retain raw logical observations, withhold speculative decode-containing OP iterations whose actual verification widths are unknown, and invalidate duration feedback on a successful width change. Same-width estimator/limit rebuilds replay raw history; transient initialization failures retain it. The operator guide now contains configuration and limitations, with unsupported snapshot-regeneration instructions removed.Validation
Current revision: 1,621 tests pass: 1,490 Planner unit tests, 29 real AIS estimator/profile integration tests and 102 orchestrator tests. The native tests use AISimulate
0.13.0.dev202610290000000087, Python 3.12 and installedai-dynamo-runtime==1.7.0, with offline model metadata and no GPU/model-weight load.Noneloses scale-up signalOrchestratorEngineAdapter.tick: 2→3,scale_upThe six new public native/orchestrator rows run against the unchanged prior PR head
a2ba064d: five fail and the cold-model control passes; all six pass after this revision. The prefill-no-headroom and decode-only consumer rows are included. Individual new native cases take about 10–20 ms locally. Existing independent rank budget, missing-model, empty-queue, failed-rebuild and same-width OP→regression replay controls remain covered. Tests now explain the oracle's126 + 2//2 + 1 = 128context and the per-rank learned prefill mean(8 + 100) / 2 = 54, and use public load queries for the changed prediction assertions.Width invalidation intentionally covers every mode: FPM duration samples lack verification-width identity. Native regression loses readiness until retrained; native interpolation retains its base model but loses learned correction. This revision adds native regression transition evidence; it does not claim a native speculative interpolation width-transition experiment.
These are local CPU/native checks, not hosted CI or measured multi-GPU serving performance. Earlier controlled replay and query-overhead numbers were collected on a prior head and are not claimed as measurements of this revision. Budget saturation is a conservative scaling-pressure signal for an infeasible configured full round, not a measured SLA violation. Configured full rounds still approximate partial/draftless rounds; actual-width decode calibration needs additional telemetry. DP>1 synthetic capacity's existing independent count/KV partitioning defect is outside this fix.
Remaining Review/API Point
AIS87 provenance exposes the resolved config and selected estimator but not resolved verification width; the latest official AIS source checked (
f3713ba3) also lacks that export. The existing canonical SDKbuild_spec_scheme(None, SpeculationConfig(...))is retained to resolve geometry;Noneis supported for this SDK operation. Removing this import/helper and marker-collection stub requires an additive upstream provenance field or getter, followed by a compatible pin update. This reviewer suggestion is therefore not claimed complete. No alternate width formula or nonexistent field is introduced.Issues and Ongoing Work
AI assistance was used for implementation, tests and review.