Repository navigation
fix(router): discard pre-arrival prefill decay credit - #17247
JasperChan929 wants to merge 3 commits into
Conversation
Preserve newly booked prefill work by settling token and duration decay before insertion. Bound discarded progress when completion shifts the clock and clear it on front replacement. Cover late arrivals, unequal modeled rates and completion/missing-duration transitions. Closes ai-dynamo#17240. Signed-off-by: JasperChan929 <wc3042@nyu.edu>
|
👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review. WalkthroughThe prefill tracker now records token completion and elapsed time that exceeded existing work before a later prefill arrived. Snapshots exclude that discarded progress from active-token and remaining-time decay. Anchor changes cap or reset the counters, and regression tests cover insertion and removal cases. ChangesPrefill accounting
Estimated code review effort: 3 (Moderate) | ~25 minutes Severity of issue fixed: Medium Merge Risk:
|
External PR triageBlocker: Next action (@JasperChan929): Reproduce with Supply-chain audit of Status as of 2026-10-28 for head |
jthomson04
left a comment
There was a problem hiding this comment.
This needs a rebase: #18055 moved the tracker to lib/router/slot-tracker/src/prefill_tracker.rs. The file differs from your base by one import line, so the patch should apply as-is.
| @@ -45,6 +45,8 @@ pub(super) struct PrefillLoadSnapshot { | |||
| pub(super) prefill_full_tokens_sum: usize, | |||
| pub(super) anchored_prefill: Option<AnchoredPrefillSnapshot>, | |||
| pub(super) total_modeled_prefill_time_ms: Option<u64>, | |||
| pub(super) discarded_completed_tokens: usize, | |||
There was a problem hiding this comment.
Your table shows projection +8% at 2048 workers, measured before #17480 changed how the queue gate reads load. Please re-measure on current main; if it holds, find out whether it's the 16 extra snapshot bytes or the extra subtractions.
There was a problem hiding this comment.
Thanks for the D arm. It shows the cost is the table entry growing from 128 to 144 B, not the subtractions. WorkerLoadTable already keeps capacities out of the slot for this reason. Could the snapshot stay at 64 B? When every prefill is modeled, snapshot() can add discarded_completed_tokens to prefill_full_tokens_sum and discarded_elapsed_ms to total_modeled_prefill_time_ms, and the readers go back to the baseline formulas. The two forms differ only for a now earlier than the write that set the counter.
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod arrival_regressions { |
There was a problem hiding this comment.
Put these in the existing tests module and reuse prefill_state_ms; modeled is the same helper.
|
|
||
| #[test] | ||
| fn arrival_progress_survives_nonfront_completion_and_resets_with_new_front() { | ||
| for (duration, remaining_tokens, remaining_ms) in [(10, 180, 18), (100, 200, 20)] { |
There was a problem hiding this comment.
A failure here won't say which row broke. Label the cases, and add a line on why the expected values are 180/18 vs 200/20. Same for the table at L942.
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
jthomson04
left a comment
There was a problem hiding this comment.
Re-reviewed at def4851. All three earlier comments are addressed and the accounting is correct. Two should-fix items inline (snapshot size, one untested reset), plus the README conflict with main.
| Ok(remaining_ms - 1), | ||
| "{case}", | ||
| ); | ||
| tracker.remove(&"A".into(), later); |
There was a problem hiding this comment.
If I delete the two reset lines in set_anchor_to_front, I think every test still passes: each assert after the front change is at elapsed 0, where a stale counter is invisible. After removing A, assert 90 tokens / Ok(9) at later + 1 ms.
|
|
||
| Part of the router split (see `lib/router/README.md`). | ||
|
|
||
| ## Modeled prefill load |
There was a problem hiding this comment.
This now conflicts with main: #18608 cut each crate README to one paragraph. I'd take main's README and drop this section; the doc comments on modeled_remaining_prefill_time_ms_at and the tracker fields already say it.
| assert_eq!(tracker.prefill_order.values().next(), Some(&anchor)); | ||
| assert_eq!(tracker.prefill_order.len(), 1); | ||
| } | ||
| #[test] |
There was a problem hiding this comment.
Blank line before #[test].
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
Overview
When an active prefill outlives its predicted duration, adding a new request can incorrectly count time before its arrival as completed work for that request. Discard that excess progress before booking new work, so the new request contributes its full token and modeled-time load.
Key Changes
Approximately 136 added production lines, excluding tests, comments and whitespace; no new dependencies or public API, wire-format or metric changes.
Validation
The bug is accumulated elapsed progress from an older anchor exceeding the work available before a later arrival. Separate token/time settlement is required because the estimates can drain at different times. Mutation-side settlement retains the original anchor and the existing mixed-queue anchor-only policy.
def4851efalls to 101 tokensFive added regression functions use the existing fixtures and explicit logical instants, without sleeps:
new_arrival_keeps_its_work_when_token_and_time_models_drain_separately: overruns, independently drained token/time estimates, and a zero-duration anchor.arrival_progress_survives_nonfront_completion_and_resets_with_new_front: partial/capped anchor shifts, later decay and request-ID reuse; now checks the next millisecond after the front changes.modeled_arrival_is_not_erased_when_unmodeled_work_is_removed: both directions of the modeled/mixed transition, including the 180-to-181 case.folded_snapshot_retains_carry_until_progress_is_subtracted: token and time overflow independently, at arrival and one millisecond later.stale_read_instant_remains_conservative_after_later_arrival: the explicitly conservative stale-instant behavior.All five fail against pinned main
c6906b12d078in an exact-source tracker harness; the existing 17 tracker tests still pass there. The candidate passes all 22 in both that harness and the native crate test target. Separate prior-head controls reproduce the mixed-transition failure and show that the strengthened reset assertion catches the reset-deletion mutant. These harness controls supplement the native crate tests. The new tests are included by the existing Rust CI test target.Native layout and paired CPU measurements
Compared pinned main
c6906b12d078(B) against the candidate patch on that same main (P), with identical Cargo.lock, compiler and probe source. x86_64rustc 1.96.1:PrefillLoadSnapshot64/64 bytes,WorkerLoadSnapshot88/88,WorkerLoadSlot104/104, actual IndexMap dense-entry stride 128/128. Field offsets change; equal sizes do not establish identical code generation or CPU cost. A 64-bit compile-time assertion protects the prefill snapshot size.Intel Core Ultra 9 290HX Plus, WSL2, release builds, pinned logical CPU 0; frequency was not fixed and the host was not isolated. Eight adjacent process pairs per cell, alternating BP/PB, all 64 processes retained. Each operator has 1,000 warmups. One 100-token prefill per logical worker, either a 10ms duration or no duration; reads at logical +1ms, lifecycle mutations at logical 0ms. Projection/lazy parity and drain assertions pass in every process. B/P column medians are independent; the change column is the median of the eight paired ratios, with their full range.
There is measurable reader overhead despite restoring the dense-entry stride: paired reader medians range from -0.35% to +6.32%, with substantial outliers retained above. This is a correctness fix, not a zero-cost claim. The common modeled read uses folded-total subtraction plus overflow handling; mixed reads preserve the anchor-only policy. Settlement adds constant work to tracked arrivals and non-front completion. Folding also runs whenever a snapshot is published, including output-block updates.
The all-busy measurement is the public reader loop plus
BusyThresholds, not the scheduler's full queue gate. Add/free does not measure output-block updates or writer contention. These fixtures do not exercise substantial discarded credit, and provide no serving-throughput, TTFT, cache-miss or RSS result.To reproduce the CPU probe, save the source below as
lib/router/slot-tracker/examples/prefill_cost_probe.rsin separate B/P checkouts. Build each with the same release toolchain, retain separate executables, and run32 unmodeled,32 modeled,2048 unmodeled, and2048 modeledeight times per arm in alternating adjacent pairs, pinned to the same CPU. The extra example is local validation only and is not part of this PR.Public-API probe source
Issues and Ongoing Work
lib/router/slot-tracker; this patch uses that location. perf(router): read worker load lazily in the scheduler queue gate #17480 introduced the lazy worker-load reader, which is included in the measurements above. Neither removes pre-arrival decay credit.Pull Request Checklist
/pr-checklistskill.