Skip to content

fix(router): honor exclusive affinity in queue admission and costing - #17311

Merged
jthomson04 merged 12 commits into
ai-dynamo:mainfrom
JasperChan929:fix/17310-exclusive-affinity-queue
Nov 13, 2026
Merged

jthomson04 merged 12 commits into
ai-dynamo:mainfrom
JasperChan929:fix/17310-exclusive-affinity-queue

Conversation

@JasperChan929

@JasperChan929 JasperChan929 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Overview

An eligible soft-affinity target is exclusive under the builtin default selector, but queue costing and admission can consider workers final selection cannot choose. Aligning them also needs independent affinity waiting lanes, so a soft request waiting for busy B cannot block ordinary traffic that idle A can accept.

Key Changes

Own diff versus main: 9 files, +1,358/-403, including tests and stage documentation. Core counts include visibility/import adaptations at the new crate boundaries.

Core scope versus main c16c2355: 496 added / 285 removed lines, using changed noncomment/nonblank source lines and excluding test/doc/example/bench code and test/testing-only hooks. This is a lexical scope estimate, including braces and attributes.

  • queue.rs / types.rs: share overload/availability snapshots and exclusive-target eligibility across costing, classification, readiness and selection. Preserve main's caller-projected load rows and actor-side freshness repair/fallback checks.
  • admission/queue.rs: separate hard pins, ranked soft targets and worker-only soft targets. Capacity changes and matching arrivals wake the relevant lanes; AFFINITY_RECHECK_INTERVAL discovers snapshot-only eligibility changes, and known fallback heads compete on every dispatch pass. The queue tree documents these wakes.
  • prompt_registry.rs / multi_worker.rs: read fresh host-reject rank counts from the existing derived registry without constructing a projection map. Authoritative sequence ownership is unchanged.

The lexical serving-code count now also excludes the composed policy_queue_tests.rs file, which earlier counts included. Serving code is unchanged by the latest cleanup. The PR retains eight added test functions and three existing-test behavior extensions; existing enqueue/drain/metadata helpers remove 29 lines of repetition while preserving ordering, blocked-lane and cleanup assertions.

Validation

Current head 6ae1d2c7737ddca42220c8d47b932188fa79702d merges main c16c2355 with the reviewed affinity-lane implementation. The merge preserves EDF/class budgets, controlled-delay root expiry, ClassTimers, deferred DRR charging, refresh event handling, cached Fx availability and prepared-row reuse. Exact, ranked-soft and worker-only lanes retain distinct expiry identities. Main #15773's affinity cache/loss metrics are retained inside the existing eligibility snapshot; explicit pins and filtering selectors keep their metric scope. The scoped native suites, all-target Clippy for the five affected crates and repository hooks pass. Main #18055 now owns lanes in dynamo-admission, actor eligibility in dynamo-scheduler, and derived fullness in dynamo-slot-tracker. The original algorithms are retained; no current-head queue or GPU performance measurement is claimed.

Behavior Coverage and result
Ranked or worker-only soft B waits while later plain work can use A exclusive_affinity_queue_waits_for_target_capacity covers both targets; B remains exclusive until eligible capacity returns
B becomes overloaded, unavailable or host-reject-full without commands/capacity notifications One table, exclusive_affinity_timer_falls_back_when_target_becomes_ineligible, advances the paused clock by 100 ms; soft B falls back to A
Hard B and ordinary traffic restricted to B Both stay queued for overload and host rejection; hard B stays parked on availability changes, while ordinary B returns the existing NoEndpoints result when no allowed endpoint is available
Fallback is discovered while A is full, then A frees exclusive_affinity_fallback_claims_other_worker_capacity_before_later_plain admits the older fallback before later plain work
Worker-only wakes and lane bookkeeping Existing worker_update_rechecks_only_that_blocked_lane plus the actor table cover wakes; affinity_lanes_preserve_fallback_and_cleanup retains one rank with dispatch/cancel coverage

853 scoped native unit/integration tests pass; three inherited tests remain ignored (14 binaries across scheduler, admission, slot-tracker, composition router and builtin policies). Counts reflect the crate split; unchanged indexer stages are outside this run. Red/green check: disabling the affinity timer makes the consolidated test fail waiting for fallback; restoring it passes. A separate temporary test on pristine main 46193c63 confirms the ordinary-request NoEndpoints behavior, so this PR does not change that error path to satisfy the broader queued-control expectation in review.

This main merge retains #18538's batched CommandInbox with one actor turn per command, #18528's guarded selected-rank-only active-block correction, and #16078's resident-count snapshot borrowed once at actual selection. The original affinity eligibility/timer remains independent. Main's new COPY-source/hint-policy interfaces are preserved. Both the original affinity regressions and the new inbox, resident-count and decode-block regressions execute in the scoped suite.

This merge also retains #18540's caller-side projection and row fill. Worker configuration is released before projecting load; the actor repairs ranks it wrote since preparation or projects again when the prepared load no longer stands. Reject thresholds retain the full actor pass. The map pool, rank-table version, write-log protocol and original affinity gate scratch are preserved; no projection map is cloned. The write-log and prepared-row booking-freshness regressions, including both full-projection and selective-repair cases, pass.

The review-requested redundant worker-only wake, classification-refresh, lower-level fallback ordering and rank/expiration cleanup variants have been removed. The three eligibility-change tests are one table. The native actor tests complete in milliseconds using paused time. The existing controlled-delay lane-root regression is extended to the same worker's exact, ranked-soft and worker-only lanes, covering blocked/fallback expiry, successor readiness, cancellation, index cleanup and no DRR charge for expiry. It and the selective wake test remain beside the composed profiles; the non-YAML fallback/cleanup unit test now lives in admission. A single aggregate index-count hook is compiled only for test/testing builds to preserve the original assertions.

Validation of this head is CPU-only. The issue's earlier published-release HTTP/GPU reproduction establishes the original trigger; it does not validate GPU serving performance of this head.

Performance limits: cached exclusive requests with host rejection still need an O(ranks) derived-count scan, potentially twice with a classifier. Ordinary/no-target requests skip that scan. Main's prepared-row checks are retained; preserving their provider snapshots through the shared helper adds up to two cheap Arc clones for a prepared request, with no added worker scan. That small cost has not been timed independently. Persistent affinity-backlog polling and serving latency/throughput have not been measured.

Historical queue controls: main 945f785 versus PR head 8e69e68

These measurements were made on PR head 8e69e681 before this main merge, not on 6ae1d2c7. Queue storage/ordering code is unchanged in this merge; no new serving-performance measurement is claimed.

Official policy_queue release benchmarks on Intel(R) Core(TM) Ultra 9 290HX Plus/WSL2, pinned CPU 2. Three adjacent fresh-process pairs per cell use AB/BA/AB; ten Criterion samples, 0.1 s warmup and 0.2 s measurement. All 30 processes passed; every result, including slower cells, is retained. Both binaries use identical lockfile, benchmark source, toolchain and release profile.

Official cell Main median µs Candidate median µs Median paired candidate/main Pair ratio range
policy_queue/drain_shared/128 5.392 4.548 1.018 0.831–1.045
policy_queue/drain_shared/4096 339.899 297.545 0.875 0.825–1.254
policy_queue/drain_one_per_exact_lane/32 1.817 1.944 1.067 1.035–1.084
policy_queue/drain_one_per_exact_lane/512 42.829 49.520 1.156 1.024–1.204
policy_queue/edf_codel_drain_4096_exact/128 576.984 597.022 1.018 1.012–1.097

Arm medians summarize each arm's three process medians; ratios summarize the three per-pair candidate/main values. The exact-lane controls are slower in these short runs (median paired increases about 6.7% and 15.6% at 32/512 lanes). CPU frequency/background load were uncontrolled. These are descriptive heap/index controls, excluding actor eligibility, persistent affinity backlog, timer latency and serving/inference performance. The EDF controlled-delay cell maintains expiry indexes during drain and never expires entries; expiry correctness is covered by the regression above.

Historical CPU/layout measurements: main 4cb0a16 versus previous PR head e3fa2da

These measurements predate the current main merge, including #17465 and #17480. They are historical tradeoff evidence, not performance validation of 6ae1d2c7. Optimized identical-lock binaries, synthetic idle 32/512-worker snapshots, CPU 0, three alternating fresh-process pairs; no exclusive-machine or CPU-frequency control. Cost fixtures return the same cost of 16 for 48 tokens with 32 cached. Criterion queue cells use ten samples, 0.1 s warmup and 0.2 s measurement. All negative cells are retained; there is no equivalence or inference speedup claim.

Workers Request Path Fixed median µs Median paired fixed/main (range)
32 ordinary metadata 0.870 1.041 (0.794–1.265)
32 ordinary classifier_roundtrip 1.706 1.034 (0.701–1.465)
32 soft metadata 0.142 0.188 (0.114–0.267)
32 soft classifier_roundtrip 0.386 0.242 (0.159–0.318)
32 soft_host_reject metadata 2.528 3.649 (2.288–4.416)
32 soft_host_reject classifier_roundtrip 2.466 1.439 (0.876–1.791)
512 ordinary metadata 12.171 1.120 (0.589–1.282)
512 ordinary classifier_roundtrip 24.967 0.934 (0.670–1.211)
512 soft metadata 1.193 0.084 (0.058–0.134)
512 soft classifier_roundtrip 2.906 0.084 (0.077–0.141)
512 soft_host_reject metadata 37.849 2.134 (2.036–5.390)
512 soft_host_reject classifier_roundtrip 37.486 0.996 (0.907–2.071)
Existing queue benchmark Fixed median µs Median paired fixed/main (range)
policy_queue/drain_one_per_exact_lane/32 1.916 1.041 (0.576–1.275)
policy_queue/drain_one_per_exact_lane/512 39.749 0.869 (0.664–1.031)
policy_queue/drain_shared/128 4.897 0.628 (0.508–1.642)
policy_queue/drain_shared/4096 259.307 0.746 (0.557–0.870)
policy_queue/pop_once_exact/32 0.210 1.740 (0.801–1.971)
policy_queue/pop_once_exact/512 0.194 0.825 (0.590–1.397)

Host-reject-aware costing still adds an O(ranks) derived-count scan when an exclusive request has cached-token candidates; classifier costing can perform it twice. It avoids a full projection map, and ordinary/no-target requests and requests with no cached-token candidates skip this additional count scan. This remains a feature-specific CPU tradeoff, not a zero-overhead claim.

On this 64-bit build, map keys remain 16 B and lane heads 48 B. Representative queue entries with a u8 payload remain 72 B in both binaries; the existing compact-header guard remains unchanged. Affinity lanes add a set entry per target and fallback heads add a priority-index entry; no RSS claim is made.

Issues and Ongoing Work

Closes #17310.

Different fallback allowlists can still cause head-of-line blocking within the fallback group; this PR does not add arbitrary constrained-head bypass. The 100 ms interval bounds eligibility recheck cadence, not admission delay under a busy actor or unavailable fallback capacity. Public request and worker-selection APIs are unchanged.

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill, with independent coverage and adversarial/performance review.

AI assistance was used for investigation, implementation and validation.

Historical investigation and provider-cost reproducer (before this queue-lane update)

The material below predates this head and its main merge. Its original-head test counts and complete scheduling-cycle measurements do not validate the current queue implementation; current results are above. It is retained as the earlier review's overload-provider evidence and reproducible investigation history.

Actual overload provider: review follow-up

To address the review question, I repeated CPU measurements with a real process-local Runtime Client and the exact embedded-provider closure move || client.overloaded_instance_ids(), plus the actual native scheduler and shipped default policy. Worker/cache/held-booking state is synthetic; this is not an HTTP or inference benchmark. Both binaries use the same474dependency versions/checksums from the upstream lock. The Runtime Client implementation is unchanged between base and head.

Counting and timing are separate runs. The timing closure has no counters. The matrix uses32/512workers, overload cardinalities0/1/8/128, idle or4/64held-B subjects, and16explicit busy-state barriers in blocked cases. There is one FCFS class, no classifier, overlap refresh or availability provider; overload IDs exclude A/B. Each timing cell has six adjacent process pairs with alternating order, CPU0 on an Intel Core Ultra9 290HX Plus, Linux6.18.33.1/WSL2; no frequency control, exclusive-machine control or deliberate concurrent overload-monitor mutations. No serving code changed in this follow-up.

Provider calls, excluding a common one-call held-request setup in blocked rows:

Scenario Base enqueue/update/drain Head enqueue/update/drain Total base→head
Idle subject 1 / 0 / 0 3 / 0 / 0 1→3
4 held-B subjects,16busy barriers 4 / 0 / 0 9 / 16 / 11 4→36
64 held-B subjects,16busy barriers 64 / 0 / 0 129 / 17 / 191 64→337

The64-subject busy window observed17callbacks across16explicit barriers; window counts include all provider uses, not a one-callback-per-command guarantee. The three idle callbacks correspond to costing, readiness/pop inspection and final selection. A nonempty result is an owned HashSet clone each time; the observed cardinality/capacity pairs are1/3,8/14and128/224. Empty snapshots returnNone.

The isolated microbenchmark calls the actual Client getter and drops the result. It includes snapshot/clock lookup, allocation/clone and drop, and excludes indirect callback dispatch. Head cell medians were23.66–23.74ns for empty snapshots,30.15–30.58ns for1/8IDs, and47.72–48.41ns for128IDs. Extra callback count × hot getter time estimates95.4ns per idle subject at128IDs, or206.5ns per subject in the64-subject/16-update case. These estimates cover the getter component only; they are not total CPU-overhead bounds or production latency.

The full cycle results are retained below, including the larger blocked-case costs. An idle cycle is admit/free plus an explicit queue barrier; a blocked cycle creates the held booking, offers subjects, issues16busy barriers, releases B and drains all subjects.

Workers Overload IDs/capacity Backlog Head getter ns Cycle µs base→head Median paired head/base
32 0/0 0 23.74 4.057→4.225 1.0359
32 0/0 4 23.66 26.873→49.477 1.8471
32 1/3 0 30.53 4.107→4.339 1.0470
32 1/3 4 30.15 27.023→51.156 1.8901
32 8/14 0 30.50 4.110→4.334 1.0519
32 8/14 4 30.58 26.814→50.717 1.8774
512 0/0 0 23.73 16.829→16.877 1.0049
512 0/0 64 23.67 1027.304→2724.700 2.6436
512 1/3 0 30.18 16.828→17.127 1.0155
512 1/3 64 30.20 1068.741→2749.983 2.5698
512 128/224 0 47.72 17.088→17.272 1.0183
512 128/224 64 48.41 1050.679→2761.427 2.6100

The base immediately dispatches to busyB, while the fixed head actually queues4/64subjects and performs capacity checks. Even the empty-set blocked controls are1.8471×/2.6436×, so the total cycle difference is not solely nonempty-set cloning. These are different scheduler-work states, not a same-service-order performance comparison. No inference speedup, slowdown, SLO result, statistical equivalence or universal overhead acceptance is claimed.

The current implementation samples at costing, queued readiness and final selection. Caching one set across an entire queue wait or overlap-refresh await would need a separate freshness contract to preserve overload fallback. This follow-up provides the requested cost evidence while retaining the correctness fix.

Real Client overload-provider reproducer

Use the standalone fixture layout/dependencies embedded in #17310, with upstream base ad1539d8eebddc37121662e9a77aee6295131330 or this PR head 0b3f7e84bcaac29e4992011fade7a79b9d722f0e. Add this dependency under [dependencies] (standard Runtime build prerequisites, including protoc, apply):

dynamo-runtime = { path = "../../.runtime/prefill-scheduler-admission-audit-20261024/source/lib/runtime", default-features = false }

Save the program below as scripts/affinity_queue_scope_audit/src/main.rs; retain the fixture's builtin-policy inclusion in src/lib.rs. Seed its lock from the upstream checkout, build with Rust1.96.1 in release mode, and keep the generated474-package lock identical across variants. Retain separate base/head executables.

Arguments are workers overload_ids backlog busy_updates count|time iterations. Count mode runs once with counters; time mode uses the unchanged embedded callback without counters. Example:

taskset -c 0 ./head-overload 512 128 64 16 count 1
taskset -c 0 ./head-overload 512 128 64 16 time 10

Choose an allowed CPU on another machine. The fixed matrix is32workers ×overload0/1/8 ×backlog0/4 and512workers ×overload0/1/128 ×backlog0/64. Set updates=0/16 and iterations=1000/10 for idle/blocked cases. Run each timing cell as six adjacent base/head process pairs with alternating order. Getter timing retains20samples after3warmups; queue-cycle timing retains10samples after3warmups. Counter runs are separate. The table uses process medians and median paired ratios, retaining all cells.

// SPDX-License-Identifier: Apache-2.0
// Actual Runtime Client overload provider + native scheduler; synthetic workers, no inference.
use std::collections::HashMap;
use std::future::Future;
use std::hint::black_box;
use std::pin::Pin;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::task::{Context, Poll, Waker};
use std::time::{Duration, Instant};

use dynamo_kv_router::protocols::{RoutingConstraints, WorkerWithDpRank};
use dynamo_kv_router::scheduling::{LocalScheduler, NoopOverlapScoresRefresh, OverlapSignals,
    OverloadedWorkerProvider, PolicyProfile, ScheduleMode, ScheduleRequest};
use dynamo_kv_router::test_utils::SimpleWorkerConfig;
use dynamo_kv_router::{ActiveSequencesMultiWorker, KvRouterConfig, NoopSequencePublisher, RouterQueuePolicy};
use dynamo_runtime::distributed::DistributedConfig;
use dynamo_runtime::{DistributedRuntime, Runtime};
use serde_json::{Value, json};
use tokio::sync::watch;
use tokio_util::sync::CancellationToken;

#[derive(Default)]
struct Counts {
    calls: AtomicUsize,
    nonempty: AtomicUsize,
    ids: AtomicUsize,
    capacity: AtomicUsize,
}

impl Counts {
    fn read(&self) -> Value {
        json!({"calls":self.calls.load(Ordering::Relaxed),
            "nonempty_returns":self.nonempty.load(Ordering::Relaxed),
            "returned_ids_total":self.ids.load(Ordering::Relaxed),
            "last_set_capacity":self.capacity.load(Ordering::Relaxed)})
    }
}

type Scheduler = LocalScheduler<NoopSequencePublisher, SimpleWorkerConfig,
    dynamo_kv_router::WorkerSelectionPolicy, NoopOverlapScoresRefresh>;

fn request(id: String, pin: bool) -> ScheduleRequest {
    let a = WorkerWithDpRank::new(0, 0);
    let b = WorkerWithDpRank::new(1, 0);
    let mut overlap = OverlapSignals::default();
    if !pin {
        overlap.effective_cached_tokens.insert(a, 16_384);
        overlap.effective_overlap_blocks.insert(a, 256.0);
        overlap.tier_overlap_blocks.device.insert(a, 256);
    }
    ScheduleRequest {
        mode: ScheduleMode::TrackedWithLifecycle { request_id: id }, token_seq: None,
        block_hashes: None, isl_tokens: 16_448, lora_name: None,
        expected_output_tokens: None, affinity_target: (!pin).then_some(b.into()),
        pinned_worker: pin.then_some(b), allowed_worker_ids: None,
        routing_constraints: RoutingConstraints::default(), router_config_override: None,
        priority_jump: 0.0, strict_priority: 0, policy_class: None, session_context: None,
        overlap, kv_transfer_candidates: None, retain_kv_transfer_chain: false,
        shared_cache_hits: None,
    }
}

fn poll_once<F: Future>(future: Pin<&mut F>) -> Poll<F::Output> {
    future.poll(&mut Context::from_waker(Waker::noop()))
}

async fn cycle(scheduler: &Scheduler, backlog: usize, updates: usize,
               sequence: &mut usize, counts: &Counts, record: bool) -> Value {
    *sequence += 1;
    let key = *sequence;
    let snapshot = || if record { counts.read() } else { Value::Null };
    let before = snapshot();
    if backlog == 0 {
        let id = format!("idle-{key}");
        let result = scheduler.schedule_request_admitted(request(id.clone(), false)).await.unwrap();
        assert_eq!(result.response.best_worker, WorkerWithDpRank::new(1, 0));
        scheduler.free(&id).await.unwrap();
        scheduler.admission_barrier().await.unwrap();
        return if record { json!({"before":before,"after":snapshot(),"subjects":1,
            "pending_after_enqueue":0,"busy_updates":0}) } else { Value::Null };
    }
    let hold = format!("hold-{key}");
    let admitted = scheduler.schedule_request_admitted(request(hold.clone(), true)).await.unwrap();
    assert_eq!(admitted.response.best_worker, WorkerWithDpRank::new(1, 0));
    let after_setup = snapshot();
    let mut pending = Vec::new();
    for index in 0..backlog {
        let id = format!("queued-{key}-{index}");
        let mut future = Box::pin(scheduler.schedule_request_admitted(request(id.clone(), false)));
        assert!(poll_once(future.as_mut()).is_pending());
        pending.push((id, future));
    }
    scheduler.admission_barrier().await.unwrap();
    let after_enqueue = snapshot();
    let pending_after_enqueue = scheduler.pending_count();
    for _ in 0..updates {
        scheduler.admission_barrier().await.unwrap();
    }
    let after_updates = snapshot();
    scheduler.free(&hold).await.unwrap();
    for (id, future) in pending {
        let response = future.await.unwrap();
        assert_eq!(response.response.best_worker, WorkerWithDpRank::new(1, 0));
        scheduler.free(&id).await.unwrap();
    }
    scheduler.admission_barrier().await.unwrap();
    assert_eq!(scheduler.pending_count(), 0);
    if record { json!({"before":before,"after_setup":after_setup,"after_enqueue":after_enqueue,
        "after_updates":after_updates,"after":snapshot(),"subjects":backlog,
        "pending_after_enqueue":pending_after_enqueue,"busy_updates":updates}) } else { Value::Null }
}

#[tokio::main(flavor = "current_thread")]
async fn main() {
    let args: Vec<_> = std::env::args().collect();
    let workers: u64 = args[1].parse().unwrap();
    let overloaded: usize = args[2].parse().unwrap();
    let backlog: usize = args[3].parse().unwrap();
    let updates: usize = args[4].parse().unwrap();
    let count_mode = match args[5].as_str() { "count" => true, "time" => false, _ => panic!("count/time") };
    let iterations: usize = args[6].parse().unwrap();
    assert!(workers >= 2 && overloaded as u64 <= workers - 2 && iterations > 0);
    let runtime = Runtime::from_current().unwrap();
    let distributed = DistributedRuntime::new(runtime.clone(), DistributedConfig::process_local()).await.unwrap();
    let endpoint = distributed.namespace("affinity_overload_review".to_string()).unwrap()
        .component("provider".to_string()).unwrap().endpoint("generate".to_string());
    let client = endpoint.client().await.unwrap();
    let ids: Vec<_> = ((workers - overloaded as u64)..workers).collect();
    client.set_overloaded_instances(&ids);
    let initial = client.overloaded_instance_ids();
    assert_eq!(initial.as_ref().map_or(0, |set| set.len()), overloaded);
    let capacity = initial.as_ref().map_or(0, |set| set.capacity());
    let counts = Arc::new(Counts::default());
    let provider: OverloadedWorkerProvider = if count_mode {
        let counts = Arc::clone(&counts);
        let client = client.clone();
        Arc::new(move || {
            let result = client.overloaded_instance_ids();
            counts.calls.fetch_add(1, Ordering::Relaxed);
            if let Some(set) = &result {
                counts.nonempty.fetch_add(1, Ordering::Relaxed);
                counts.ids.fetch_add(set.len(), Ordering::Relaxed);
                counts.capacity.store(set.capacity(), Ordering::Relaxed);
            }
            result
        })
    } else {
        // Exactly the closure installed by embedded KvRouter, with no counters.
        let client = client.clone();
        Arc::new(move || client.overloaded_instance_ids())
    };
    let slots = Arc::new(ActiveSequencesMultiWorker::new(
        NoopSequencePublisher, 64, (0..workers).map(|id| (id, (0, 1))).collect(),
        false, 0, "overload_review",
    ));
    let configs: HashMap<_, _> = (0..workers).map(|id| (id, SimpleWorkerConfig {
        max_num_batched_tokens: Some(2048), ..Default::default()
    })).collect();
    let (_tx, receiver) = watch::channel(configs);
    let cancel = CancellationToken::new();
    let scheduler = LocalScheduler::new(
        Arc::clone(&slots), receiver, PolicyProfile::synthetic(Some(1.0), RouterQueuePolicy::Fcfs),
        64, affinity_queue_scope_audit::builtin_default::default_policy(KvRouterConfig::default(), "review"),
        None, None::<Arc<NoopOverlapScoresRefresh>>, Some(provider), None,
        Duration::from_secs(60), true, cancel.clone(), "review", false,
    );
    let mut sequence = 0;
    let mut timing_samples = Vec::new();
    let mut provider_samples = Vec::new();
    let phases;
    if count_mode {
        phases = cycle(&scheduler, backlog, updates, &mut sequence, &counts, true).await;
    } else {
        // Direct provider time includes ArcSwap/clock lookup, allocation, clone and drop.
        for sample in 0..23 {
            let start = Instant::now();
            for _ in 0..1000 { drop(black_box(client.overloaded_instance_ids())); }
            if sample >= 3 { provider_samples.push(start.elapsed().as_secs_f64() * 1e9 / 1000.0); }
        }
        let mut last = Value::Null;
        for sample in 0..13 {
            let start = Instant::now();
            for _ in 0..iterations {
                last = cycle(&scheduler, backlog, updates, &mut sequence, &counts, false).await;
            }
            if sample >= 3 { timing_samples.push(start.elapsed().as_secs_f64() * 1e9 / iterations as f64); }
        }
        phases = last;
    }
    assert_eq!(client.overloaded_instance_ids().as_ref().map_or(0, |set| set.len()), overloaded);
    slots.assert_completely_drained(tokio::time::Instant::now());
    assert_eq!(scheduler.pending_count(), 0);
    cancel.cancel();
    runtime.shutdown();
    println!("{}", json!({"workers":workers,"overloaded":overloaded,"set_capacity":capacity,
        "backlog":backlog,"updates":updates,"mode":args[5],"iterations":iterations,
        "cycle_samples_ns":timing_samples,"provider_samples_ns":provider_samples,
        "phases":phases,"counts":counts.read(),"cleaned_up":true,
        "scope":"Real Runtime Client provider and native scheduler; synthetic worker/cache/held booking state; no HTTP/inference"}));
}

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 requested a review from a team as a code owner October 26, 2026 14:22
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added the external-contribution Pull request is from an external contributor label Oct 26, 2026
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

Queue cache-cost calculations and prefill-capacity checks now account for eligible exclusive-affinity targets. When a target is ineligible, the queue falls back to other eligible workers. The same eligibility rule applies to final worker selection.

Changes

Exclusive-affinity queue eligibility

Layer / File(s) Summary
Shared affinity eligibility
lib/kv-router/src/scheduling/types.rs, lib/kv-router/src/scheduling/queue.rs
SchedulingContext::with_eligibility accepts RoutingEligibility directly. The queue stores its overloaded-worker provider and exclusive-affinity selector state, and adds helpers to constrain eligibility for eligible targets.
Queue cache-cost calculation
lib/kv-router/src/scheduling/queue.rs, lib/kv-router/src/scheduling/types.rs
Queue classification and metadata use a shared cache-cost calculation. Cached-token candidates are filtered by worker and data-parallel rank. Tests cover target eligibility and fallback.
Admission and worker selection
lib/kv-router/src/scheduling/queue.rs
Never-queues checks, ready-head checks, and queue draining apply overloaded-worker and eligible-target constraints. Worker selection uses the shared eligibility helper. Tests cover waiting for target capacity, fallback, zero queue limits, and nonexclusive selectors.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 0b3f7

The reviewed queue changes preserve explicit pins and exclude stale cache candidates. No actionable merge-blocking risk remains after normal checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: honoring exclusive affinity during queue admission and costing.
Linked Issues check Passed Issue #17310 requires queue admission and cache costing to match final selection when the builtin default selector makes an eligible soft-affinity target exclusive. The diff at head ad1539d..0b3f7e8…
Out of Scope Changes check Passed Every visible hunk relates to the issue #17310 objective. The changes are in lib/kv-router/src/scheduling/queue.rs and lib/kv-router/src/scheduling/types.rs. They add the exclusive-target eligibil…
Description check Passed The description is long and detailed, and it covers the template sections: Overview, Key Changes, Validation, and Issues and Ongoing Work. It explains that the PR aligns exclusive-affinity eligibility…
Full details: Linked Issues check

Explanation

Issue #17310 requires queue admission and cache costing to match final selection when the builtin default selector makes an eligible soft-affinity target exclusive. The diff at head ad1539d..0b3f7e8 does this:

  • New helper exclusive_affinity_eligibility applies with_affinity_target only when the selector uses exclusive affinity and the target is eligible (affinity_target_is_eligible). Ineligible targets fall back.
  • default_queue_metadata and the classifier path (best_queue_cached_tokens) use this helper, so queue cost uses the target's cache. Example: a cached A with an exclusive target B yields cached_tokens 0.
  • Busy admission (all_workers_prefill_busy_with calls in the ready-head check, the drain path, and pop_next) uses the same helper, so a busy exclusive target blocks admission.
  • Final selection (the eligibility block near the selection path) uses the same helper.
  • overloaded_worker_provider is consulted only for exclusive affinity targets (affinity_overloaded_workers). Nonexclusive selectors get None.
  • best_cached_tokens filters candidates with validate_worker_rank, so cached worker/rank entries must pass the same eligibility.
    The visible tests cover target-specific cache cost, rank matching, waiting for target capacity, fallback when the target is excluded, and nonexclusive behavior. The diff was truncated at 16000 bytes, so I did not read every test. I did not run the tests. The author reports 1116 passing tests and five regressions that fail on the base. Those results are not independently verified here. Explicit pins and caller constraints remain on their existing paths in the visible hunks.
Full details: Out of Scope Changes check

Explanation

Every visible hunk relates to the issue #17310 objective. The changes are in lib/kv-router/src/scheduling/queue.rs and lib/kv-router/src/scheduling/types.rs. They add the exclusive-target eligibility helper, the overloaded-worker snapshot gated on exclusive affinity, the queue-cost and admission call-site updates, and the SchedulingContext::with_eligibility refactor. The types.rs change replaces with_available_workers with with_eligibility and changes best_cached_tokens to use validate_worker_rank. The with_available_workers removal is a crate-private (pub(crate)) change. The tests shown are regressions for the same behavior. One point needs reviewer attention. validate_worker_rank now applies to all selectors, including nonexclusive ones. Cached entries with invalid worker or rank pairs are dropped for those selectors too. This is consistent with the stated validation goal, and I did not find a demonstrated unrelated change. The effect on nonexclusive custom-policy costing is not verified against the full validate_worker_rank definition, which I did not read.

Full details: Description check

Explanation

The description is long and detailed, and it covers the template sections: Overview, Key Changes, Validation, and Issues and Ongoing Work. It explains that the PR aligns exclusive-affinity eligibility across queue costing, admission, readiness, and final selection, and it lists the affected files and the tests that cover each behavior. It links the closed issue (Closes #17310) and includes the Pull Request Checklist, marked as complete. The description is mostly complete and well organized. Its main weakness is length: the Key Changes and Validation sections include large historical benchmark tables and merge-history detail that do not help a reviewer decide on the merge. The Validation section also states that the reported test counts reflect the crate split, which makes the test coverage hard to verify from the description alone. These are presentation issues, not missing required information.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ad1539d and 0b3f7e8.

Files selected for processing: 2

  • lib/kv-router/src/scheduling/queue.rs
  • lib/kv-router/src/scheduling/types.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@dynamo-review-agent dynamo-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions for the author:

  • lib/kv-router/src/scheduling/queue.rs: The supported embedded KV-router installs Client::overloaded_instance_ids, which clones a non-empty overloaded-worker HashSet. This change invokes that provider while costing each exclusive-affinity request and again during readiness/pop checks, potentially repeatedly for queued heads; the supplied CPU measurements explicitly used no overload provider. Could you provide a calculation or measurement for the embedded exclusive-affinity path with a non-empty overload set and representative backlog/worker counts, covering provider-call frequency and cloned-set size?

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator October 26, 2026 16:08 — with GitHub Actions Inactive

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Costing, readiness and selection now agree on the exclusive target, and switching best_cached_tokens to validate_worker_rank is a good tightening. Requesting changes for the head-of-line blocking below. It only shows with queueing on, but that's when this path matters.

Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Keep exclusive soft targets out of the shared queue and preserve fallback
priority when another worker gains capacity. Share eligibility snapshots
across costing, readiness and selection, including current host rejection.

Merge main to resolve classifier and admission interface conflicts.

Refs ai-dynamo#17310.

Signed-off-by: JasperChan929 <wc3042@nyu.edu>

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks right now, and all three earlier points are addressed. Before I approve, please cut the tests down. Several of them cover the same behavior twice, at two levels or across variants that run one code path. Aim for one test per new behavior, at the cheapest level that reaches the new code:

  • Merge exclusive_affinity_timer_falls_back_without_target_capacity_updates, exclusive_affinity_rechecks_host_reject_limits_before_waiting and host_reject_keeps_ordinary_and_hard_requests_in_their_busy_queue into one table over the ineligibility source: overloaded, unavailable, host-reject full. Hold B and queue a soft request to B, a hard pin to B, and an ordinary request restricted to B. Flip the source and advance 100 ms. Assert the soft request lands on A while the other two stay queued.
  • Drop exclusive_worker_affinity_wakes_when_another_rank_frees. The worker_update_rechecks_only_that_blocked_lane extension and the None case of exclusive_affinity_queue_waits_for_target_capacity already cover the worker-only wake. Picking the free rank is existing selector behavior.
  • Drop exclusive_affinity_cost_refresh_uses_current_load_after_classification. validate_classification already recomputes queue metadata on main; the only new part is which clock with_queue_eligibility passes.
  • exclusive_affinity_fallback_claims_other_worker_capacity_before_later_plain and fallback_head_competes_for_other_worker_capacity_before_later_shared_work test the same ordering. Keep the first: it also covers the FallbackBlocked classification in drain_ready.
  • In affinity_lanes_preserve_fallback_and_cleanup, drop the rank loop and the expire arm. Ranked and worker-only lanes share the same bookkeeping, and cancel and expire both go through take_if_entry_in_class → rebuild_worker_heads.

That should roughly halve the test code without losing coverage of anything new.

Separately, lib/kv-router/src/scheduling/AGENTS.md still describes per-class storage as ready_by_worker/blocked_workers keyed by WorkerWithDpRank, with no affinity lanes or fallback heads. Please update the tree. Add a line on how affinity lanes wake (target capacity, an enqueue to the same target, the 100 ms recheck) and note that fallback heads are rechecked on every pass.

Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Consolidate eligibility-change controls and remove duplicate wake, clock,
ordering and cleanup variants. Document affinity lanes and fallback wakes,
and name the 100 ms recheck interval.

Retain main's lazy load reader and prepared-candidate validation when
resolving queue and derived-load-read conflicts.

Refs ai-dynamo#17310.

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator October 30, 2026 10:33 — with GitHub Actions Inactive
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 30, 2026
@JasperChan929

Copy link
Copy Markdown
Contributor Author

@jthomson04 Following up on your test-consolidation review.

Addressed in c837b1e6: I consolidated the three eligibility-change tests, removed the specified duplicate wake/refresh/ordering cases, simplified cleanup coverage to one rank with dispatch/cancel, updated the lane tree and wake documentation, and added AFFINITY_RECHECK_INTERVAL with its bounds explained.

One control expectation differs from your suggestion:

Target B becomes Soft affinity to B Hard pin to B Ordinary, allowed only B
Overloaded Falls back to A Stays queued Stays queued
Host-reject full Falls back to A Stays queued Stays queued
Unavailable Falls back to A Stays queued Returns NoEndpoints

I confirmed the ordinary-request NoEndpoints behavior with an isolated test on pristine main 46193c63 and preserved it here. Did you intend this PR to change that behavior to waiting, or should this control retain main's behavior?

The net added test-section lines decreased from 773 to 501, approximately 35%, counting comments and whitespace relative to each version's main baseline. All specifically identified duplicates were removed. Local validation passes: 1,503 scoped tests, with five inherited ignores.

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 4, 2026 16:53 — with GitHub Actions Inactive

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the trim. The tests now cover one case per new behavior, and the AGENTS.md lane tree and AFFINITY_RECHECK_INTERVAL read well. On your NoEndpoints question: keep main's behavior. Our table was wrong for that case. An ordinary request whose only allowed worker is gone has nothing to wait for.

@jthomson04

Copy link
Copy Markdown
Contributor

/ok to test a33113d

@jthomson04
jthomson04 enabled auto-merge (squash) November 4, 2026 17:50
@devin-ai-integration

devin-ai-integration Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 43324058082 (attempt 1) on 6ae1d2c773

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 43324058082. Updated on every full-CI run of this PR.

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
auto-merge was automatically disabled November 5, 2026 12:49

Head branch was pushed to by a user without write access

@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 5, 2026 12:49 — with GitHub Actions Inactive
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 6, 2026 20:24 — with GitHub Actions Inactive
@jthomson04

Copy link
Copy Markdown
Contributor

/ok to test f401065

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 11, 2026 10:19 — with GitHub Actions Inactive
@jthomson04
jthomson04 temporarily deployed to external_collaborator November 12, 2026 03:29 — with GitHub Actions Inactive
@jthomson04

Copy link
Copy Markdown
Contributor

/ok to test e19add8

Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 12, 2026 15:46 — with GitHub Actions Inactive
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 12, 2026 19:40 — with GitHub Actions Inactive
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
@JasperChan929
JasperChan929 temporarily deployed to external_collaborator November 12, 2026 19:56 — with GitHub Actions Inactive
@jthomson04

Copy link
Copy Markdown
Contributor

/ok to test 6ae1d2c

This branch was previously deployed

1 inactive deployment
external_collaborator — 6ae1d2c7 Deployed Nov 12, 2026 by JasperChan929 via ok-to-test #50652
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor fix size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Queue admission ignores the default selector's exclusive soft-affinity target

2 participants