Skip to content

fix(router): discard pre-arrival prefill decay credit - #17247

Open
JasperChan929 wants to merge 3 commits into
ai-dynamo:mainfrom
JasperChan929:fix/prefill-arrival-decay-17240
Open

JasperChan929 wants to merge 3 commits into
ai-dynamo:mainfrom
JasperChan929:fix/prefill-arrival-decay-17240

Conversation

@JasperChan929

@JasperChan929 JasperChan929 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Overview

When an active prefill outlives its predicted duration, adding a new request can incorrectly count time before its arrival as completed work for that request. Discard that excess progress before booking new work, so the new request contributes its full token and modeled-time load.

Key Changes

Approximately 136 added production lines, excluding tests, comments and whitespace; no new dependencies or public API, wire-format or metric changes.

  • prefill_tracker.rs: settle excess progress on insertion, bound retained credit when a non-front completion shifts the anchor, and clear it when the front changes. Fold modeled credit into snapshot totals; retain overflow carry until subtraction. A private mode enum keeps the snapshot at 64 bytes and carries the token credit needed by mixed modeled/unmodeled queues.
  • prompt_registry.rs: adapt private test fixtures to the compact snapshot representation. Dense registry entries remain 128 bytes on the measured x86_64 target.
  • multi_worker.rs: describe the corrected modeled-time estimate. The crate README follows main's single-paragraph version from chore(router): remove the layering gate; per-crate README and AGENTS.md baseline #18608; accounting details stay beside the implementation.

Validation

The bug is accumulated elapsed progress from an older anchor exceeding the work available before a later arrival. Separate token/time settlement is required because the estimates can drain at different times. Mutation-side settlement retains the original anchor and the existing mixed-queue anchor-only policy.

Behavior Before With this change
A=100 tokens/10ms at t=0; add B=100/10ms at t=30ms Main reports 0 tokens/0ms for the aggregate 100 tokens/10ms at arrival, 90/9ms at t=31ms
Non-front completion shifts or caps the anchor Retained pre-arrival credit needs bounding to the new elapsed progress Both partial and capped shifts preserve the booked work; decay resumes afterward
Remove the old front, then read 1ms later The old test passed even if both counter-reset assignments were deleted The strengthened assertion requires 90 tokens/9ms; a reset-deletion mutant returns 100 tokens and fails
After a non-front completion, insert an unknown-duration 1-token request into a queue with 180 tokens left Prior PR head def4851e falls to 101 tokens 181 at insertion, 171 one millisecond later, then the anchor-only 101-token floor
Fold near-limit token/time totals Naive saturating addition loses progress before subtraction Carry retained until subtraction preserves the previous explicit-counter result
Read with a caller instant earlier than snapshot settlement Explicit-counter representation bounded the read by raw booked totals Folded totals can conservatively overestimate, including exceeding raw totals; fresh reads remain correct

Five added regression functions use the existing fixtures and explicit logical instants, without sleeps:

  • new_arrival_keeps_its_work_when_token_and_time_models_drain_separately: overruns, independently drained token/time estimates, and a zero-duration anchor.
  • arrival_progress_survives_nonfront_completion_and_resets_with_new_front: partial/capped anchor shifts, later decay and request-ID reuse; now checks the next millisecond after the front changes.
  • modeled_arrival_is_not_erased_when_unmodeled_work_is_removed: both directions of the modeled/mixed transition, including the 180-to-181 case.
  • folded_snapshot_retains_carry_until_progress_is_subtracted: token and time overflow independently, at arrival and one millisecond later.
  • stale_read_instant_remains_conservative_after_later_arrival: the explicitly conservative stale-instant behavior.

All five fail against pinned main c6906b12d078 in an exact-source tracker harness; the existing 17 tracker tests still pass there. The candidate passes all 22 in both that harness and the native crate test target. Separate prior-head controls reproduce the mixed-transition failure and show that the strengthened reset assertion catches the reset-deletion mutant. These harness controls supplement the native crate tests. The new tests are included by the existing Rust CI test target.

Native layout and paired CPU measurements

Compared pinned main c6906b12d078 (B) against the candidate patch on that same main (P), with identical Cargo.lock, compiler and probe source. x86_64 rustc 1.96.1: PrefillLoadSnapshot 64/64 bytes, WorkerLoadSnapshot 88/88, WorkerLoadSlot 104/104, actual IndexMap dense-entry stride 128/128. Field offsets change; equal sizes do not establish identical code generation or CPU cost. A 64-bit compile-time assertion protects the prefill snapshot size.

Intel Core Ultra 9 290HX Plus, WSL2, release builds, pinned logical CPU 0; frequency was not fixed and the host was not isolated. Eight adjacent process pairs per cell, alternating BP/PB, all 64 processes retained. Each operator has 1,000 warmups. One 100-token prefill per logical worker, either a 10ms duration or no duration; reads at logical +1ms, lifecycle mutations at logical 0ms. Projection/lazy parity and drain assertions pass in every process. B/P column medians are independent; the change column is the median of the eight paired ratios, with their full range.

Logical workers Duration hints Operation B median ns P median ns Paired median change [min, max]
32 Unknown Full projection 122.52 129.71 +6.04% [+2.98%, +49.61%]
32 Unknown One lazy lookup 19.33 19.48 +0.88% [-1.13%, +4.23%]
32 Unknown All-busy reader loop 206.80 218.94 +6.27% [+3.52%, +9.92%]
32 Unknown Add/free pair 339.76 351.20 +2.70% [-1.62%, +5.50%]
32 Modeled Full projection 210.34 218.77 +3.84% [+1.61%, +6.09%]
32 Modeled One lazy lookup 20.25 20.70 +2.40% [-4.91%, +6.59%]
32 Modeled All-busy reader loop 344.69 352.91 +1.70% [+0.56%, +3.89%]
32 Modeled Add/free pair 344.26 360.17 +3.71% [-3.93%, +6.97%]
2048 Unknown Full projection 10556.75 10884.52 +2.27% [+0.45%, +8.14%]
2048 Unknown One lazy lookup 19.57 19.37 -0.35% [-1.97%, +2.28%]
2048 Unknown All-busy reader loop 16659.30 17651.72 +6.32% [+2.74%, +8.99%]
2048 Unknown Add/free pair 351.70 342.27 -2.02% [-4.46%, +3.29%]
2048 Modeled Full projection 14705.21 14926.74 +1.64% [-1.07%, +3.45%]
2048 Modeled One lazy lookup 20.46 20.91 +0.95% [-0.06%, +57.71%]
2048 Modeled All-busy reader loop 26253.69 26773.94 +1.65% [-0.60%, +3.97%]
2048 Modeled Add/free pair 364.88 359.17 -2.78% [-3.58%, +2.70%]

There is measurable reader overhead despite restoring the dense-entry stride: paired reader medians range from -0.35% to +6.32%, with substantial outliers retained above. This is a correctness fix, not a zero-cost claim. The common modeled read uses folded-total subtraction plus overflow handling; mixed reads preserve the anchor-only policy. Settlement adds constant work to tracked arrivals and non-front completion. Folding also runs whenever a snapshot is published, including output-block updates.

The all-busy measurement is the public reader loop plus BusyThresholds, not the scheduler's full queue gate. Add/free does not measure output-block updates or writer contention. These fixtures do not exercise substantial discarded credit, and provide no serving-throughput, TTFT, cache-miss or RSS result.

To reproduce the CPU probe, save the source below as lib/router/slot-tracker/examples/prefill_cost_probe.rs in separate B/P checkouts. Build each with the same release toolchain, retain separate executables, and run 32 unmodeled, 32 modeled, 2048 unmodeled, and 2048 modeled eight times per arm in alternating adjacent pairs, pinned to the same CPU. The extra example is local validation only and is not part of this PR.

cargo build --locked --release -p dynamo-slot-tracker --example prefill_cost_probe
taskset -c 0 target/release/examples/prefill_cost_probe 32 modeled
Public-API probe source
// SPDX-License-Identifier: Apache-2.0
// Local, identical-source production-library probe; never part of the PR.

use std::collections::HashMap;
use std::hint::black_box;
use std::time::{Duration, Instant as WallInstant};

use dynamo_router_protocol::worker::{WorkerCapacity, WorkerWithDpRank};
use dynamo_slot_tracker::host::{
    ActiveSequencesMultiWorker, BusyThresholds, NoopSequencePublisher, PrefillLoadHint, SequenceRequest,
};
use tokio::time::Instant;

fn measure(iterations: u64, mut operation: impl FnMut()) -> f64 {
    for _ in 0..1000 {
        operation();
    }
    let start = WallInstant::now();
    for _ in 0..iterations {
        operation();
    }
    start.elapsed().as_nanos() as f64 / iterations as f64
}

fn main() -> anyhow::Result<()> {
    let args: Vec<_> = std::env::args().collect();
    let workers: u64 = args[1].parse()?;
    assert!(workers == 32 || workers == 2048);
    assert!(args[2] == "modeled" || args[2] == "unmodeled");
    let modeled = args[2] == "modeled";
    let scan_iterations = 50_000_000 / workers;
    let point_iterations = 2_000_000;
    let lifecycle_iterations = 250_000;
    let slots = ActiveSequencesMultiWorker::new_without_expiry(
        NoopSequencePublisher,
        64,
        (0..workers).map(|id| (id, (0, 1))).collect::<HashMap<_, _>>(),
        false,
        0,
        "probe",
    );
    let anchor_now = Instant::now();
    let query_now = anchor_now + Duration::from_millis(1);
    let mutation_now = anchor_now;
    let hint = PrefillLoadHint {
        initial_effective_prefill_tokens: 100,
        expected_prefill_duration: modeled.then_some(Duration::from_millis(10)),
    };
    let request = |id: String, worker_id: u64| SequenceRequest {
        request_id: id,
        token_sequence: None,
        track_prefill_tokens: true,
        expected_output_tokens: None,
        prefill_load_hint: Some(hint),
        worker: WorkerWithDpRank::new(worker_id, 0),
        lora_name: None,
    };
    let ranks: Vec<_> = (0..workers).map(|id| WorkerWithDpRank::new(id, 0)).collect();
    let expected_tokens = if modeled { 90 } else { 100 };
    for id in 0..workers {
        slots.add_request(request(format!("anchor-{id}"), id), anchor_now)?;
    }

    let mut projections = slots.project_worker_loads(None, query_now);
    assert_eq!(projections.len(), workers as usize);
    assert!(projections.values().all(|p| p.active_prefill_tokens == expected_tokens
        && p.active_requests == 1 && p.active_decode_blocks == 0));
    let projection_ns = measure(scan_iterations, || {
        slots.project_worker_loads_into(None, black_box(query_now), &mut projections, None);
        black_box(&projections);
    });
    let projection_checksum: usize = projections.values().map(|p| p.active_prefill_tokens).sum();
    assert_eq!(projection_checksum, expected_tokens * workers as usize);

    // Includes reader acquisition/release once per logical query, plus one point read.
    let first_rank = ranks[0];
    let lazy_one_ns = measure(point_iterations, || {
        let reader = slots.worker_load_reader(black_box(query_now));
        let load = reader.get(black_box(first_rank)).unwrap();
        black_box(load);
    });
    let limits = BusyThresholds {
        prefill_busy_threshold: Some(50),
        ..Default::default()
    };
    let capacity = WorkerCapacity::default();
    // Every rank is busy, so this all() traverses all ranks. This is a reader-loop
    // microbenchmark, not the scheduler's complete queue gate.
    let lazy_all_busy_ns = measure(scan_iterations, || {
        let reader = slots.worker_load_reader(black_box(query_now));
        let all_busy = ranks.iter().all(|rank| {
            let load = reader.get(black_box(*rank)).unwrap();
            !limits.has_room(&load, capacity)
        });
        black_box(all_busy);
    });
    {
        let reader = slots.worker_load_reader(query_now);
        for rank in &ranks {
            assert_eq!(reader.get(*rank), projections.get(rank).copied());
        }
    }

    // Fixed logical instants keep both arms in the same pre-drain state even if
    // wall-clock measurement takes longer than the modeled duration.
    let probe = "probe".to_string();
    let add_free_pair_ns = measure(lifecycle_iterations, || {
        slots.add_request(request(probe.clone(), 0), black_box(mutation_now)).unwrap();
        black_box(slots.free(&probe, black_box(mutation_now)).unwrap());
    });
    let after = slots.project_worker_loads(None, query_now);
    assert_eq!(after, projections);
    for id in 0..workers {
        slots.free(&format!("anchor-{id}"), mutation_now)?;
    }
    slots.assert_completely_drained(mutation_now);
    println!("{}", serde_json::json!({
        "workers": workers,
        "modeled": modeled,
        "fixed_query_offset_ms": 1,
        "fixed_mutation_offset_ms": 0,
        "warmup_iterations_per_operator": 1000,
        "scan_iterations": scan_iterations,
        "point_iterations": point_iterations,
        "lifecycle_iterations": lifecycle_iterations,
        "projection_ns": projection_ns,
        "lazy_one_ns": lazy_one_ns,
        "lazy_all_busy_ns": lazy_all_busy_ns,
        "add_free_pair_ns": add_free_pair_ns,
        "projection_checksum": projection_checksum,
        "expected_tokens_per_rank": expected_tokens,
        "lazy_projection_parity": true,
        "post_lifecycle_projection_parity": true,
        "cleanup_drained": true,
    }));
    Ok(())
}

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Preserve newly booked prefill work by settling token and duration decay
before insertion. Bound discarded progress when completion shifts the
clock and clear it on front replacement. Cover late arrivals, unequal
modeled rates and completion/missing-duration transitions.

Closes ai-dynamo#17240.

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 requested a review from a team as a code owner October 24, 2026 16:27
@copy-pr-bot

copy-pr-bot Bot commented Oct 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor fix documentation Improvements or additions to documentation labels Oct 24, 2026
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The prefill tracker now records token completion and elapsed time that exceeded existing work before a later prefill arrived. Snapshots exclude that discarded progress from active-token and remaining-time decay. Anchor changes cap or reset the counters, and regression tests cover insertion and removal cases.

Changes

Prefill accounting

Layer / File(s) Summary
Snapshot counters and decay calculations
lib/kv-router/src/sequences/prefill_tracker.rs, lib/kv-router/src/sequences/multi_worker.rs
Snapshots now include discarded token and elapsed-time counters. Active-token and modeled-time calculations subtract these counters from decay.
Arrival accounting and anchor updates
lib/kv-router/src/sequences/prefill_tracker.rs, lib/kv-router/src/sequences/README.md, lib/kv-router/src/sequences/prompt_registry.rs
Insertion records excess progress before adding a prefill. Anchor shifts cap or reset the counters. Regression tests cover late arrivals and removals, and the documentation describes the accounting behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 9f9a3

No actionable issue remains in the reviewed prefill accounting change; it is ready for normal merge checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "fix(router): discard pre-arrival prefill decay credit" names the main change: the router now discards modeled prefill progress that accrued before a new request arrived. It is specific, sho…
Linked Issues check Passed Issue [#17240] requires modeled prefill accounting to prevent pre-arrival decay from reducing new work, while preserving valid decay for existing work and keeping C queued at the busy threshold. `Pref…
Out of Scope Changes check Passed The changes stay within issue [#17240]. The implementation changes prefill accounting, the tests verify the late-arrival and lifecycle cases, and the README and multi_worker.rs documentation describ…
Description check Passed The description covers the problem, the key code changes by file, validation (test results, CPU measurements, reproduction steps), and issue/reference context. It follows the required template heading…
Full details: Title check

Explanation

The title "fix(router): discard pre-arrival prefill decay credit" names the main change: the router now discards modeled prefill progress that accrued before a new request arrived. It is specific, short, and follows a conventional commit style. It matches the linked issue #17240 and the objective of keeping newly booked work at its full load.

Full details: Linked Issues check

Explanation

Issue [#17240] requires modeled prefill accounting to prevent pre-arrival decay from reducing new work, while preserving valid decay for existing work and keeping C queued at the busy threshold. PrefillLoadTracker::insert now settles pre-arrival token and modeled-time progress before insertion. PrefillLoadSnapshot subtracts only the recorded discarded progress. Removal bounds the counters, and front changes reset them. The three regression tests cover separate token/time depletion, non-front completion, front reset, and mixed modeled/unmodeled work. The documented validation reports that the native admission cases and the full KV-router test suite pass.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue [#17240]. The implementation changes prefill accounting, the tests verify the late-arrival and lifecycle cases, and the README and multi_worker.rs documentation describe the updated formula. The prompt_registry.rs change only updates a private snapshot test fixture for the new counters. No unrelated production behavior, public API, wire field, or dependency change is shown.

Full details: Description check

Explanation

The description covers the problem, the key code changes by file, validation (test results, CPU measurements, reproduction steps), and issue/reference context. It follows the required template headings: Overview, Key Changes, Validation, Issues and Ongoing Work, and the checklist. The Closes line references #17240 and the checklist is marked. The description is long and includes detailed benchmark material, but every required section is substantively filled in.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 1ecc870 and 9f9a385.

Files selected for processing: 4

  • lib/kv-router/src/sequences/README.md
  • lib/kv-router/src/sequences/multi_worker.rs
  • lib/kv-router/src/sequences/prefill_tracker.rs
  • lib/kv-router/src/sequences/prompt_registry.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

External PR triage

Blocker: rust-tests (.) (Pre Merge) fails in dynamo-mocker: generalized_live::tests::attention_dp_releases_completion_only_at_the_shared_boundary and generalized_live::tests::cancellation_does_not_end_a_pass_with_unrelated_pending_output. dynamo-mocker consumes dynamo_kv_router::sequences, which this PR changes, and Pre Merge passes on main. So the failures are most likely caused by this PR.

Next action (@JasperChan929): Reproduce with cargo test -p dynamo-mocker --lib generalized_live, fix, and push. After that, a maintainer from the router codeowners (@ai-dynamo/dynamo-router-codeowners) needs to review and post /ok to test <sha>.

Supply-chain audit of 9f9a38543b: Looks safe to approve. No workflow/CI, dependency, download-URL, script, secret, or binary changes. Changes are limited to Rust and a README in lib/kv-router/src/sequences/.

Status as of 2026-10-28 for head 9f9a38543b. This comment is edited in place on later runs.

Written by Devin

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a rebase: #18055 moved the tracker to lib/router/slot-tracker/src/prefill_tracker.rs. The file differs from your base by one import line, so the patch should apply as-is.

@@ -45,6 +45,8 @@ pub(super) struct PrefillLoadSnapshot {
pub(super) prefill_full_tokens_sum: usize,
pub(super) anchored_prefill: Option<AnchoredPrefillSnapshot>,
pub(super) total_modeled_prefill_time_ms: Option<u64>,
pub(super) discarded_completed_tokens: usize,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your table shows projection +8% at 2048 workers, measured before #17480 changed how the queue gate reads load. Please re-measure on current main; if it holds, find out whether it's the 16 extra snapshot bytes or the extra subtractions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the D arm. It shows the cost is the table entry growing from 128 to 144 B, not the subtractions. WorkerLoadTable already keeps capacities out of the slot for this reason. Could the snapshot stay at 64 B? When every prefill is modeled, snapshot() can add discarded_completed_tokens to prefill_full_tokens_sum and discarded_elapsed_ms to total_modeled_prefill_time_ms, and the readers go back to the baseline formulas. The two forms differ only for a now earlier than the write that set the counter.

}

#[cfg(test)]
mod arrival_regressions {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put these in the existing tests module and reuse prefill_state_ms; modeled is the same helper.


#[test]
fn arrival_progress_survives_nonfront_completion_and_resets_with_new_front() {
for (duration, remaining_tokens, remaining_ms) in [(10, 180, 18), (100, 200, 20)] {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failure here won't say which row broke. Label the cases, and add a line on why the expected values are 180/18 vs 200/20. Same for the table at L942.

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 13, 2026 20:54 — with GitHub Actions Inactive

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at def4851. All three earlier comments are addressed and the accounting is correct. Two should-fix items inline (snapshot size, one untested reset), plus the README conflict with main.

Ok(remaining_ms - 1),
"{case}",
);
tracker.remove(&"A".into(), later);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I delete the two reset lines in set_anchor_to_front, I think every test still passes: each assert after the front change is at elapsed 0, where a stale counter is invisible. After removing A, assert 90 tokens / Ok(9) at later + 1 ms.

Comment thread lib/router/slot-tracker/README.md Outdated

Part of the router split (see `lib/router/README.md`).

## Modeled prefill load

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now conflicts with main: #18608 cut each crate README to one paragraph. I'd take main's README and drop this section; the doc comments on modeled_remaining_prefill_time_ms_at and the tracker fields already say it.

assert_eq!(tracker.prefill_order.values().next(), Some(&anchor));
assert_eq!(tracker.prefill_order.len(), 1);
}
#[test]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blank line before #[test].

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 deployed to external_collaborator November 17, 2026 10:57 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
external_collaborator — ffc363b0 Deployed Nov 17, 2026 by JasperChan929 via ok-to-test #52421
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Newly booked prefill work inherits pre-arrival decay credit

2 participants