You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(router): honor exclusive affinity in queue admission and costing - #17311
An eligible soft-affinity target is exclusive under the builtin default selector, but queue costing and admission can consider workers final selection cannot choose. Aligning them also needs independent affinity waiting lanes, so a soft request waiting for busy B cannot block ordinary traffic that idle A can accept.
Key Changes
Own diff versus main: 9 files, +1,358/-403, including tests and stage documentation. Core counts include visibility/import adaptations at the new crate boundaries.
Core scope versus main c16c2355: 496 added / 285 removed lines, using changed noncomment/nonblank source lines and excluding test/doc/example/bench code and test/testing-only hooks. This is a lexical scope estimate, including braces and attributes.
queue.rs / types.rs: share overload/availability snapshots and exclusive-target eligibility across costing, classification, readiness and selection. Preserve main's caller-projected load rows and actor-side freshness repair/fallback checks.
admission/queue.rs: separate hard pins, ranked soft targets and worker-only soft targets. Capacity changes and matching arrivals wake the relevant lanes; AFFINITY_RECHECK_INTERVAL discovers snapshot-only eligibility changes, and known fallback heads compete on every dispatch pass. The queue tree documents these wakes.
prompt_registry.rs / multi_worker.rs: read fresh host-reject rank counts from the existing derived registry without constructing a projection map. Authoritative sequence ownership is unchanged.
The lexical serving-code count now also excludes the composed policy_queue_tests.rs file, which earlier counts included. Serving code is unchanged by the latest cleanup. The PR retains eight added test functions and three existing-test behavior extensions; existing enqueue/drain/metadata helpers remove 29 lines of repetition while preserving ordering, blocked-lane and cleanup assertions.
Validation
Current head 6ae1d2c7737ddca42220c8d47b932188fa79702d merges main c16c2355 with the reviewed affinity-lane implementation. The merge preserves EDF/class budgets, controlled-delay root expiry, ClassTimers, deferred DRR charging, refresh event handling, cached Fx availability and prepared-row reuse. Exact, ranked-soft and worker-only lanes retain distinct expiry identities. Main #15773's affinity cache/loss metrics are retained inside the existing eligibility snapshot; explicit pins and filtering selectors keep their metric scope. The scoped native suites, all-target Clippy for the five affected crates and repository hooks pass. Main #18055 now owns lanes in dynamo-admission, actor eligibility in dynamo-scheduler, and derived fullness in dynamo-slot-tracker. The original algorithms are retained; no current-head queue or GPU performance measurement is claimed.
Behavior
Coverage and result
Ranked or worker-only soft B waits while later plain work can use A
exclusive_affinity_queue_waits_for_target_capacity covers both targets; B remains exclusive until eligible capacity returns
B becomes overloaded, unavailable or host-reject-full without commands/capacity notifications
One table, exclusive_affinity_timer_falls_back_when_target_becomes_ineligible, advances the paused clock by 100 ms; soft B falls back to A
Hard B and ordinary traffic restricted to B
Both stay queued for overload and host rejection; hard B stays parked on availability changes, while ordinary B returns the existing NoEndpoints result when no allowed endpoint is available
Fallback is discovered while A is full, then A frees
exclusive_affinity_fallback_claims_other_worker_capacity_before_later_plain admits the older fallback before later plain work
Worker-only wakes and lane bookkeeping
Existing worker_update_rechecks_only_that_blocked_lane plus the actor table cover wakes; affinity_lanes_preserve_fallback_and_cleanup retains one rank with dispatch/cancel coverage
853 scoped native unit/integration tests pass; three inherited tests remain ignored (14 binaries across scheduler, admission, slot-tracker, composition router and builtin policies). Counts reflect the crate split; unchanged indexer stages are outside this run. Red/green check: disabling the affinity timer makes the consolidated test fail waiting for fallback; restoring it passes. A separate temporary test on pristine main 46193c63 confirms the ordinary-request NoEndpoints behavior, so this PR does not change that error path to satisfy the broader queued-control expectation in review.
This main merge retains #18538's batched CommandInbox with one actor turn per command, #18528's guarded selected-rank-only active-block correction, and #16078's resident-count snapshot borrowed once at actual selection. The original affinity eligibility/timer remains independent. Main's new COPY-source/hint-policy interfaces are preserved. Both the original affinity regressions and the new inbox, resident-count and decode-block regressions execute in the scoped suite.
This merge also retains #18540's caller-side projection and row fill. Worker configuration is released before projecting load; the actor repairs ranks it wrote since preparation or projects again when the prepared load no longer stands. Reject thresholds retain the full actor pass. The map pool, rank-table version, write-log protocol and original affinity gate scratch are preserved; no projection map is cloned. The write-log and prepared-row booking-freshness regressions, including both full-projection and selective-repair cases, pass.
The review-requested redundant worker-only wake, classification-refresh, lower-level fallback ordering and rank/expiration cleanup variants have been removed. The three eligibility-change tests are one table. The native actor tests complete in milliseconds using paused time. The existing controlled-delay lane-root regression is extended to the same worker's exact, ranked-soft and worker-only lanes, covering blocked/fallback expiry, successor readiness, cancellation, index cleanup and no DRR charge for expiry. It and the selective wake test remain beside the composed profiles; the non-YAML fallback/cleanup unit test now lives in admission. A single aggregate index-count hook is compiled only for test/testing builds to preserve the original assertions.
Validation of this head is CPU-only. The issue's earlier published-release HTTP/GPU reproduction establishes the original trigger; it does not validate GPU serving performance of this head.
Performance limits: cached exclusive requests with host rejection still need an O(ranks) derived-count scan, potentially twice with a classifier. Ordinary/no-target requests skip that scan. Main's prepared-row checks are retained; preserving their provider snapshots through the shared helper adds up to two cheap Arc clones for a prepared request, with no added worker scan. That small cost has not been timed independently. Persistent affinity-backlog polling and serving latency/throughput have not been measured.
Historical queue controls: main 945f785 versus PR head 8e69e68
These measurements were made on PR head 8e69e681 before this main merge, not on 6ae1d2c7. Queue storage/ordering code is unchanged in this merge; no new serving-performance measurement is claimed.
Official policy_queue release benchmarks on Intel(R) Core(TM) Ultra 9 290HX Plus/WSL2, pinned CPU 2. Three adjacent fresh-process pairs per cell use AB/BA/AB; ten Criterion samples, 0.1 s warmup and 0.2 s measurement. All 30 processes passed; every result, including slower cells, is retained. Both binaries use identical lockfile, benchmark source, toolchain and release profile.
Official cell
Main median µs
Candidate median µs
Median paired candidate/main
Pair ratio range
policy_queue/drain_shared/128
5.392
4.548
1.018
0.831–1.045
policy_queue/drain_shared/4096
339.899
297.545
0.875
0.825–1.254
policy_queue/drain_one_per_exact_lane/32
1.817
1.944
1.067
1.035–1.084
policy_queue/drain_one_per_exact_lane/512
42.829
49.520
1.156
1.024–1.204
policy_queue/edf_codel_drain_4096_exact/128
576.984
597.022
1.018
1.012–1.097
Arm medians summarize each arm's three process medians; ratios summarize the three per-pair candidate/main values. The exact-lane controls are slower in these short runs (median paired increases about 6.7% and 15.6% at 32/512 lanes). CPU frequency/background load were uncontrolled. These are descriptive heap/index controls, excluding actor eligibility, persistent affinity backlog, timer latency and serving/inference performance. The EDF controlled-delay cell maintains expiry indexes during drain and never expires entries; expiry correctness is covered by the regression above.
Historical CPU/layout measurements: main 4cb0a16 versus previous PR head e3fa2da
These measurements predate the current main merge, including #17465 and #17480. They are historical tradeoff evidence, not performance validation of 6ae1d2c7. Optimized identical-lock binaries, synthetic idle 32/512-worker snapshots, CPU 0, three alternating fresh-process pairs; no exclusive-machine or CPU-frequency control. Cost fixtures return the same cost of 16 for 48 tokens with 32 cached. Criterion queue cells use ten samples, 0.1 s warmup and 0.2 s measurement. All negative cells are retained; there is no equivalence or inference speedup claim.
Workers
Request
Path
Fixed median µs
Median paired fixed/main (range)
32
ordinary
metadata
0.870
1.041 (0.794–1.265)
32
ordinary
classifier_roundtrip
1.706
1.034 (0.701–1.465)
32
soft
metadata
0.142
0.188 (0.114–0.267)
32
soft
classifier_roundtrip
0.386
0.242 (0.159–0.318)
32
soft_host_reject
metadata
2.528
3.649 (2.288–4.416)
32
soft_host_reject
classifier_roundtrip
2.466
1.439 (0.876–1.791)
512
ordinary
metadata
12.171
1.120 (0.589–1.282)
512
ordinary
classifier_roundtrip
24.967
0.934 (0.670–1.211)
512
soft
metadata
1.193
0.084 (0.058–0.134)
512
soft
classifier_roundtrip
2.906
0.084 (0.077–0.141)
512
soft_host_reject
metadata
37.849
2.134 (2.036–5.390)
512
soft_host_reject
classifier_roundtrip
37.486
0.996 (0.907–2.071)
Existing queue benchmark
Fixed median µs
Median paired fixed/main (range)
policy_queue/drain_one_per_exact_lane/32
1.916
1.041 (0.576–1.275)
policy_queue/drain_one_per_exact_lane/512
39.749
0.869 (0.664–1.031)
policy_queue/drain_shared/128
4.897
0.628 (0.508–1.642)
policy_queue/drain_shared/4096
259.307
0.746 (0.557–0.870)
policy_queue/pop_once_exact/32
0.210
1.740 (0.801–1.971)
policy_queue/pop_once_exact/512
0.194
0.825 (0.590–1.397)
Host-reject-aware costing still adds an O(ranks) derived-count scan when an exclusive request has cached-token candidates; classifier costing can perform it twice. It avoids a full projection map, and ordinary/no-target requests and requests with no cached-token candidates skip this additional count scan. This remains a feature-specific CPU tradeoff, not a zero-overhead claim.
On this 64-bit build, map keys remain 16 B and lane heads 48 B. Representative queue entries with a u8 payload remain 72 B in both binaries; the existing compact-header guard remains unchanged. Affinity lanes add a set entry per target and fallback heads add a priority-index entry; no RSS claim is made.
Main refactor(router): split dynamo-kv-router into vocabulary, stage, and kernel crates #18055 splits the router into stages and a scheduler kernel. This merge keeps that ownership and its public stage operations, lazy load/prepared-row contracts, testing hooks and dependency layering. Affinity tests use the existing resolved-profile helper rather than importing YAML into the kernel.
Different fallback allowlists can still cause head-of-line blocking within the fallback group; this PR does not add arbitrary constrained-head bypass. The 100 ms interval bounds eligibility recheck cadence, not admission delay under a busy actor or unavailable fallback capacity. Public request and worker-selection APIs are unchanged.
Pull Request Checklist
I used Dynamo's /pr-checklist skill, with independent coverage and adversarial/performance review.
AI assistance was used for investigation, implementation and validation.
Historical investigation and provider-cost reproducer (before this queue-lane update)
The material below predates this head and its main merge. Its original-head test counts and complete scheduling-cycle measurements do not validate the current queue implementation; current results are above. It is retained as the earlier review's overload-provider evidence and reproducible investigation history.
Actual overload provider: review follow-up
To address the review question, I repeated CPU measurements with a real process-local Runtime Client and the exact embedded-provider closure move || client.overloaded_instance_ids(), plus the actual native scheduler and shipped default policy. Worker/cache/held-booking state is synthetic; this is not an HTTP or inference benchmark. Both binaries use the same474dependency versions/checksums from the upstream lock. The Runtime Client implementation is unchanged between base and head.
Counting and timing are separate runs. The timing closure has no counters. The matrix uses32/512workers, overload cardinalities0/1/8/128, idle or4/64held-B subjects, and16explicit busy-state barriers in blocked cases. There is one FCFS class, no classifier, overlap refresh or availability provider; overload IDs exclude A/B. Each timing cell has six adjacent process pairs with alternating order, CPU0 on an Intel Core Ultra9 290HX Plus, Linux6.18.33.1/WSL2; no frequency control, exclusive-machine control or deliberate concurrent overload-monitor mutations. No serving code changed in this follow-up.
Provider calls, excluding a common one-call held-request setup in blocked rows:
Scenario
Base enqueue/update/drain
Head enqueue/update/drain
Total base→head
Idle subject
1 / 0 / 0
3 / 0 / 0
1→3
4 held-B subjects,16busy barriers
4 / 0 / 0
9 / 16 / 11
4→36
64 held-B subjects,16busy barriers
64 / 0 / 0
129 / 17 / 191
64→337
The64-subject busy window observed17callbacks across16explicit barriers; window counts include all provider uses, not a one-callback-per-command guarantee. The three idle callbacks correspond to costing, readiness/pop inspection and final selection. A nonempty result is an owned HashSet clone each time; the observed cardinality/capacity pairs are1/3,8/14and128/224. Empty snapshots returnNone.
The isolated microbenchmark calls the actual Client getter and drops the result. It includes snapshot/clock lookup, allocation/clone and drop, and excludes indirect callback dispatch. Head cell medians were23.66–23.74ns for empty snapshots,30.15–30.58ns for1/8IDs, and47.72–48.41ns for128IDs. Extra callback count × hot getter time estimates95.4ns per idle subject at128IDs, or206.5ns per subject in the64-subject/16-update case. These estimates cover the getter component only; they are not total CPU-overhead bounds or production latency.
The full cycle results are retained below, including the larger blocked-case costs. An idle cycle is admit/free plus an explicit queue barrier; a blocked cycle creates the held booking, offers subjects, issues16busy barriers, releases B and drains all subjects.
Workers
Overload IDs/capacity
Backlog
Head getter ns
Cycle µs base→head
Median paired head/base
32
0/0
0
23.74
4.057→4.225
1.0359
32
0/0
4
23.66
26.873→49.477
1.8471
32
1/3
0
30.53
4.107→4.339
1.0470
32
1/3
4
30.15
27.023→51.156
1.8901
32
8/14
0
30.50
4.110→4.334
1.0519
32
8/14
4
30.58
26.814→50.717
1.8774
512
0/0
0
23.73
16.829→16.877
1.0049
512
0/0
64
23.67
1027.304→2724.700
2.6436
512
1/3
0
30.18
16.828→17.127
1.0155
512
1/3
64
30.20
1068.741→2749.983
2.5698
512
128/224
0
47.72
17.088→17.272
1.0183
512
128/224
64
48.41
1050.679→2761.427
2.6100
The base immediately dispatches to busyB, while the fixed head actually queues4/64subjects and performs capacity checks. Even the empty-set blocked controls are1.8471×/2.6436×, so the total cycle difference is not solely nonempty-set cloning. These are different scheduler-work states, not a same-service-order performance comparison. No inference speedup, slowdown, SLO result, statistical equivalence or universal overhead acceptance is claimed.
The current implementation samples at costing, queued readiness and final selection. Caching one set across an entire queue wait or overlap-refresh await would need a separate freshness contract to preserve overload fallback. This follow-up provides the requested cost evidence while retaining the correctness fix.
Real Client overload-provider reproducer
Use the standalone fixture layout/dependencies embedded in #17310, with upstream base ad1539d8eebddc37121662e9a77aee6295131330 or this PR head 0b3f7e84bcaac29e4992011fade7a79b9d722f0e. Add this dependency under [dependencies] (standard Runtime build prerequisites, including protoc, apply):
Save the program below as scripts/affinity_queue_scope_audit/src/main.rs; retain the fixture's builtin-policy inclusion in src/lib.rs. Seed its lock from the upstream checkout, build with Rust1.96.1 in release mode, and keep the generated474-package lock identical across variants. Retain separate base/head executables.
Arguments are workers overload_ids backlog busy_updates count|time iterations. Count mode runs once with counters; time mode uses the unchanged embedded callback without counters. Example:
Choose an allowed CPU on another machine. The fixed matrix is32workers ×overload0/1/8 ×backlog0/4 and512workers ×overload0/1/128 ×backlog0/64. Set updates=0/16 and iterations=1000/10 for idle/blocked cases. Run each timing cell as six adjacent base/head process pairs with alternating order. Getter timing retains20samples after3warmups; queue-cycle timing retains10samples after3warmups. Counter runs are separate. The table uses process medians and median paired ratios, retaining all cells.
👋 Hi JasperChan929! Thank you for contributing to ai-dynamo/dynamo.
Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.
No actionable comments were generated in the recent review.
Walkthrough
Queue cache-cost calculations and prefill-capacity checks now account for eligible exclusive-affinity targets. When a target is ineligible, the queue falls back to other eligible workers. The same eligibility rule applies to final worker selection.
SchedulingContext::with_eligibility accepts RoutingEligibility directly. The queue stores its overloaded-worker provider and exclusive-affinity selector state, and adds helpers to constrain eligibility for eligible targets.
Queue classification and metadata use a shared cache-cost calculation. Cached-token candidates are filtered by worker and data-parallel rank. Tests cover target eligibility and fallback.
Admission and worker selection lib/kv-router/src/scheduling/queue.rs
Never-queues checks, ready-head checks, and queue draining apply overloaded-worker and eligible-target constraints. Worker selection uses the shared eligibility helper. Tests cover waiting for target capacity, fallback, zero queue limits, and nonexclusive selectors.
The reviewed queue changes preserve explicit pins and exclude stale cache candidates. No actionable merge-blocking risk remains after normal checks.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title clearly and concisely describes the main change: honoring exclusive affinity during queue admission and costing.
Linked Issues check
Issue #17310 requires queue admission and cache costing to match final selection when the builtin default selector makes an eligible soft-affinity target exclusive. The diff at head ad1539d..0b3f7e8…
Out of Scope Changes check
Every visible hunk relates to the issue #17310 objective. The changes are in lib/kv-router/src/scheduling/queue.rs and lib/kv-router/src/scheduling/types.rs. They add the exclusive-target eligibil…
Description check
The description is long and detailed, and it covers the template sections: Overview, Key Changes, Validation, and Issues and Ongoing Work. It explains that the PR aligns exclusive-affinity eligibility…
Full details: Linked Issues check
Explanation
Issue #17310 requires queue admission and cache costing to match final selection when the builtin default selector makes an eligible soft-affinity target exclusive. The diff at head ad1539d..0b3f7e8 does this:
New helper exclusive_affinity_eligibility applies with_affinity_target only when the selector uses exclusive affinity and the target is eligible (affinity_target_is_eligible). Ineligible targets fall back.
default_queue_metadata and the classifier path (best_queue_cached_tokens) use this helper, so queue cost uses the target's cache. Example: a cached A with an exclusive target B yields cached_tokens 0.
Busy admission (all_workers_prefill_busy_with calls in the ready-head check, the drain path, and pop_next) uses the same helper, so a busy exclusive target blocks admission.
Final selection (the eligibility block near the selection path) uses the same helper.
overloaded_worker_provider is consulted only for exclusive affinity targets (affinity_overloaded_workers). Nonexclusive selectors get None.
best_cached_tokens filters candidates with validate_worker_rank, so cached worker/rank entries must pass the same eligibility.
The visible tests cover target-specific cache cost, rank matching, waiting for target capacity, fallback when the target is excluded, and nonexclusive behavior. The diff was truncated at 16000 bytes, so I did not read every test. I did not run the tests. The author reports 1116 passing tests and five regressions that fail on the base. Those results are not independently verified here. Explicit pins and caller constraints remain on their existing paths in the visible hunks.
Full details: Out of Scope Changes check
Explanation
Every visible hunk relates to the issue #17310 objective. The changes are in lib/kv-router/src/scheduling/queue.rs and lib/kv-router/src/scheduling/types.rs. They add the exclusive-target eligibility helper, the overloaded-worker snapshot gated on exclusive affinity, the queue-cost and admission call-site updates, and the SchedulingContext::with_eligibility refactor. The types.rs change replaces with_available_workers with with_eligibility and changes best_cached_tokens to use validate_worker_rank. The with_available_workers removal is a crate-private (pub(crate)) change. The tests shown are regressions for the same behavior. One point needs reviewer attention. validate_worker_rank now applies to all selectors, including nonexclusive ones. Cached entries with invalid worker or rank pairs are dropped for those selectors too. This is consistent with the stated validation goal, and I did not find a demonstrated unrelated change. The effect on nonexclusive custom-policy costing is not verified against the full validate_worker_rank definition, which I did not read.
Full details: Description check
Explanation
The description is long and detailed, and it covers the template sections: Overview, Key Changes, Validation, and Issues and Ongoing Work. It explains that the PR aligns exclusive-affinity eligibility across queue costing, admission, readiness, and final selection, and it lists the affected files and the tests that cover each behavior. It links the closed issue (Closes #17310) and includes the Pull Request Checklist, marked as complete. The description is mostly complete and well organized. Its main weakness is length: the Key Changes and Validation sections include large historical benchmark tables and merge-history detail that do not help a reviewer decide on the merge. The Validation section also states that the reported test counts reflect the crate split, which makes the test coverage hard to verify from the description alone. These are presentation issues, not missing required information.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Reviewing files that changed from the base of the PR and between ad1539d and 0b3f7e8.
Files selected for processing: 2
lib/kv-router/src/scheduling/queue.rs
lib/kv-router/src/scheduling/types.rs
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Questions for the author:
lib/kv-router/src/scheduling/queue.rs: The supported embedded KV-router installs Client::overloaded_instance_ids, which clones a non-empty overloaded-worker HashSet. This change invokes that provider while costing each exclusive-affinity request and again during readiness/pop checks, potentially repeatedly for queued heads; the supplied CPU measurements explicitly used no overload provider. Could you provide a calculation or measurement for the embedded exclusive-affinity path with a non-empty overload set and representative backlog/worker counts, covering provider-call frequency and cloned-set size?
The reason will be displayed to describe this comment to others. Learn more.
Costing, readiness and selection now agree on the exclusive target, and switching best_cached_tokens to validate_worker_rank is a good tightening. Requesting changes for the head-of-line blocking below. It only shows with queueing on, but that's when this path matters.
Keep exclusive soft targets out of the shared queue and preserve fallback
priority when another worker gains capacity. Share eligibility snapshots
across costing, readiness and selection, including current host rejection.
Merge main to resolve classifier and admission interface conflicts.
Refs ai-dynamo#17310.
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
The reason will be displayed to describe this comment to others. Learn more.
The fix looks right now, and all three earlier points are addressed. Before I approve, please cut the tests down. Several of them cover the same behavior twice, at two levels or across variants that run one code path. Aim for one test per new behavior, at the cheapest level that reaches the new code:
Merge exclusive_affinity_timer_falls_back_without_target_capacity_updates, exclusive_affinity_rechecks_host_reject_limits_before_waiting and host_reject_keeps_ordinary_and_hard_requests_in_their_busy_queue into one table over the ineligibility source: overloaded, unavailable, host-reject full. Hold B and queue a soft request to B, a hard pin to B, and an ordinary request restricted to B. Flip the source and advance 100 ms. Assert the soft request lands on A while the other two stay queued.
Drop exclusive_worker_affinity_wakes_when_another_rank_frees. The worker_update_rechecks_only_that_blocked_lane extension and the None case of exclusive_affinity_queue_waits_for_target_capacity already cover the worker-only wake. Picking the free rank is existing selector behavior.
Drop exclusive_affinity_cost_refresh_uses_current_load_after_classification. validate_classification already recomputes queue metadata on main; the only new part is which clock with_queue_eligibility passes.
exclusive_affinity_fallback_claims_other_worker_capacity_before_later_plain and fallback_head_competes_for_other_worker_capacity_before_later_shared_work test the same ordering. Keep the first: it also covers the FallbackBlocked classification in drain_ready.
In affinity_lanes_preserve_fallback_and_cleanup, drop the rank loop and the expire arm. Ranked and worker-only lanes share the same bookkeeping, and cancel and expire both go through take_if_entry_in_class → rebuild_worker_heads.
That should roughly halve the test code without losing coverage of anything new.
Separately, lib/kv-router/src/scheduling/AGENTS.md still describes per-class storage as ready_by_worker/blocked_workers keyed by WorkerWithDpRank, with no affinity lanes or fallback heads. Please update the tree. Add a line on how affinity lanes wake (target capacity, an enqueue to the same target, the 100 ms recheck) and note that fallback heads are rechecked on every pass.
Consolidate eligibility-change controls and remove duplicate wake, clock,
ordering and cleanup variants. Document affinity lanes and fallback wakes,
and name the 100 ms recheck interval.
Retain main's lazy load reader and prepared-candidate validation when
resolving queue and derived-load-read conflicts.
Refs ai-dynamo#17310.
Signed-off-by: JasperChan929 <wc3042@nyu.edu>
Addressed in c837b1e6: I consolidated the three eligibility-change tests, removed the specified duplicate wake/refresh/ordering cases, simplified cleanup coverage to one rank with dispatch/cancel, updated the lane tree and wake documentation, and added AFFINITY_RECHECK_INTERVAL with its bounds explained.
One control expectation differs from your suggestion:
Target B becomes
Soft affinity to B
Hard pin to B
Ordinary, allowed only B
Overloaded
Falls back to A
Stays queued
Stays queued
Host-reject full
Falls back to A
Stays queued
Stays queued
Unavailable
Falls back to A
Stays queued
Returns NoEndpoints
I confirmed the ordinary-request NoEndpoints behavior with an isolated test on pristine main 46193c63 and preserved it here. Did you intend this PR to change that behavior to waiting, or should this control retain main's behavior?
The net added test-section lines decreased from 773 to 501, approximately 35%, counting comments and whitespace relative to each version's main baseline. All specifically identified duplicates were removed. Local validation passes: 1,503 scoped tests, with five inherited ignores.
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the trim. The tests now cover one case per new behavior, and the AGENTS.md lane tree and AFFINITY_RECHECK_INTERVAL read well. On your NoEndpoints question: keep main's behavior. Our table was wrong for that case. An ordinary request whose only allowed worker is gone has nothing to wait for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
An eligible soft-affinity target is exclusive under the builtin default selector, but queue costing and admission can consider workers final selection cannot choose. Aligning them also needs independent affinity waiting lanes, so a soft request waiting for busy B cannot block ordinary traffic that idle A can accept.
Key Changes
Own diff versus main: 9 files, +1,358/-403, including tests and stage documentation. Core counts include visibility/import adaptations at the new crate boundaries.
Core scope versus main
c16c2355: 496 added / 285 removed lines, using changed noncomment/nonblank source lines and excluding test/doc/example/bench code and test/testing-only hooks. This is a lexical scope estimate, including braces and attributes.AFFINITY_RECHECK_INTERVALdiscovers snapshot-only eligibility changes, and known fallback heads compete on every dispatch pass. The queue tree documents these wakes.The lexical serving-code count now also excludes the composed
policy_queue_tests.rsfile, which earlier counts included. Serving code is unchanged by the latest cleanup. The PR retains eight added test functions and three existing-test behavior extensions; existing enqueue/drain/metadata helpers remove 29 lines of repetition while preserving ordering, blocked-lane and cleanup assertions.Validation
Current head
6ae1d2c7737ddca42220c8d47b932188fa79702dmerges mainc16c2355with the reviewed affinity-lane implementation. The merge preserves EDF/class budgets, controlled-delay root expiry, ClassTimers, deferred DRR charging, refresh event handling, cached Fx availability and prepared-row reuse. Exact, ranked-soft and worker-only lanes retain distinct expiry identities. Main #15773's affinity cache/loss metrics are retained inside the existing eligibility snapshot; explicit pins and filtering selectors keep their metric scope. The scoped native suites, all-target Clippy for the five affected crates and repository hooks pass. Main #18055 now owns lanes indynamo-admission, actor eligibility indynamo-scheduler, and derived fullness indynamo-slot-tracker. The original algorithms are retained; no current-head queue or GPU performance measurement is claimed.exclusive_affinity_queue_waits_for_target_capacitycovers both targets; B remains exclusive until eligible capacity returnsexclusive_affinity_timer_falls_back_when_target_becomes_ineligible, advances the paused clock by 100 ms; soft B falls back to ANoEndpointsresult when no allowed endpoint is availableexclusive_affinity_fallback_claims_other_worker_capacity_before_later_plainadmits the older fallback before later plain workworker_update_rechecks_only_that_blocked_laneplus the actor table cover wakes;affinity_lanes_preserve_fallback_and_cleanupretains one rank with dispatch/cancel coverage853 scoped native unit/integration tests pass; three inherited tests remain ignored (14 binaries across scheduler, admission, slot-tracker, composition router and builtin policies). Counts reflect the crate split; unchanged indexer stages are outside this run. Red/green check: disabling the affinity timer makes the consolidated test fail waiting for fallback; restoring it passes. A separate temporary test on pristine main
46193c63confirms the ordinary-requestNoEndpointsbehavior, so this PR does not change that error path to satisfy the broader queued-control expectation in review.This main merge retains #18538's batched
CommandInboxwith one actor turn per command, #18528's guarded selected-rank-only active-block correction, and #16078's resident-count snapshot borrowed once at actual selection. The original affinity eligibility/timer remains independent. Main's new COPY-source/hint-policy interfaces are preserved. Both the original affinity regressions and the new inbox, resident-count and decode-block regressions execute in the scoped suite.This merge also retains #18540's caller-side projection and row fill. Worker configuration is released before projecting load; the actor repairs ranks it wrote since preparation or projects again when the prepared load no longer stands. Reject thresholds retain the full actor pass. The map pool, rank-table version, write-log protocol and original affinity gate scratch are preserved; no projection map is cloned. The write-log and prepared-row booking-freshness regressions, including both full-projection and selective-repair cases, pass.
The review-requested redundant worker-only wake, classification-refresh, lower-level fallback ordering and rank/expiration cleanup variants have been removed. The three eligibility-change tests are one table. The native actor tests complete in milliseconds using paused time. The existing controlled-delay lane-root regression is extended to the same worker's exact, ranked-soft and worker-only lanes, covering blocked/fallback expiry, successor readiness, cancellation, index cleanup and no DRR charge for expiry. It and the selective wake test remain beside the composed profiles; the non-YAML fallback/cleanup unit test now lives in admission. A single aggregate index-count hook is compiled only for test/testing builds to preserve the original assertions.
Validation of this head is CPU-only. The issue's earlier published-release HTTP/GPU reproduction establishes the original trigger; it does not validate GPU serving performance of this head.
Performance limits: cached exclusive requests with host rejection still need an O(ranks) derived-count scan, potentially twice with a classifier. Ordinary/no-target requests skip that scan. Main's prepared-row checks are retained; preserving their provider snapshots through the shared helper adds up to two cheap
Arcclones for a prepared request, with no added worker scan. That small cost has not been timed independently. Persistent affinity-backlog polling and serving latency/throughput have not been measured.Historical queue controls: main 945f785 versus PR head 8e69e68
These measurements were made on PR head
8e69e681before this main merge, not on6ae1d2c7. Queue storage/ordering code is unchanged in this merge; no new serving-performance measurement is claimed.Official
policy_queuerelease benchmarks on Intel(R) Core(TM) Ultra 9 290HX Plus/WSL2, pinned CPU 2. Three adjacent fresh-process pairs per cell use AB/BA/AB; ten Criterion samples, 0.1 s warmup and 0.2 s measurement. All 30 processes passed; every result, including slower cells, is retained. Both binaries use identical lockfile, benchmark source, toolchain and release profile.policy_queue/drain_shared/128policy_queue/drain_shared/4096policy_queue/drain_one_per_exact_lane/32policy_queue/drain_one_per_exact_lane/512policy_queue/edf_codel_drain_4096_exact/128Arm medians summarize each arm's three process medians; ratios summarize the three per-pair candidate/main values. The exact-lane controls are slower in these short runs (median paired increases about 6.7% and 15.6% at 32/512 lanes). CPU frequency/background load were uncontrolled. These are descriptive heap/index controls, excluding actor eligibility, persistent affinity backlog, timer latency and serving/inference performance. The EDF controlled-delay cell maintains expiry indexes during drain and never expires entries; expiry correctness is covered by the regression above.
Historical CPU/layout measurements: main 4cb0a16 versus previous PR head e3fa2da
These measurements predate the current main merge, including #17465 and #17480. They are historical tradeoff evidence, not performance validation of
6ae1d2c7. Optimized identical-lock binaries, synthetic idle 32/512-worker snapshots, CPU 0, three alternating fresh-process pairs; no exclusive-machine or CPU-frequency control. Cost fixtures return the same cost of 16 for 48 tokens with 32 cached. Criterion queue cells use ten samples, 0.1 s warmup and 0.2 s measurement. All negative cells are retained; there is no equivalence or inference speedup claim.policy_queue/drain_one_per_exact_lane/32policy_queue/drain_one_per_exact_lane/512policy_queue/drain_shared/128policy_queue/drain_shared/4096policy_queue/pop_once_exact/32policy_queue/pop_once_exact/512Host-reject-aware costing still adds an O(ranks) derived-count scan when an exclusive request has cached-token candidates; classifier costing can perform it twice. It avoids a full projection map, and ordinary/no-target requests and requests with no cached-token candidates skip this additional count scan. This remains a feature-specific CPU tradeoff, not a zero-overhead claim.
On this 64-bit build, map keys remain 16 B and lane heads 48 B. Representative queue entries with a u8 payload remain 72 B in both binaries; the existing compact-header guard remains unchanged. Affinity lanes add a set entry per target and fallback heads add a priority-index entry; no RSS claim is made.
Issues and Ongoing Work
Closes #17310.
Different fallback allowlists can still cause head-of-line blocking within the fallback group; this PR does not add arbitrary constrained-head bypass. The 100 ms interval bounds eligibility recheck cadence, not admission delay under a busy actor or unavailable fallback capacity. Public request and worker-selection APIs are unchanged.
Pull Request Checklist
/pr-checklistskill, with independent coverage and adversarial/performance review.AI assistance was used for investigation, implementation and validation.
Historical investigation and provider-cost reproducer (before this queue-lane update)
The material below predates this head and its main merge. Its original-head test counts and complete scheduling-cycle measurements do not validate the current queue implementation; current results are above. It is retained as the earlier review's overload-provider evidence and reproducible investigation history.
Actual overload provider: review follow-up
To address the review question, I repeated CPU measurements with a real process-local
Runtime Clientand the exact embedded-provider closuremove || client.overloaded_instance_ids(), plus the actual native scheduler and shipped default policy. Worker/cache/held-booking state is synthetic; this is not an HTTP or inference benchmark. Both binaries use the same474dependency versions/checksums from the upstream lock. The Runtime Client implementation is unchanged between base and head.Counting and timing are separate runs. The timing closure has no counters. The matrix uses32/512workers, overload cardinalities0/1/8/128, idle or4/64held-B subjects, and16explicit busy-state barriers in blocked cases. There is one FCFS class, no classifier, overlap refresh or availability provider; overload IDs exclude A/B. Each timing cell has six adjacent process pairs with alternating order, CPU0 on an Intel Core Ultra9 290HX Plus, Linux6.18.33.1/WSL2; no frequency control, exclusive-machine control or deliberate concurrent overload-monitor mutations. No serving code changed in this follow-up.
Provider calls, excluding a common one-call held-request setup in blocked rows:
The64-subject busy window observed17callbacks across16explicit barriers; window counts include all provider uses, not a one-callback-per-command guarantee. The three idle callbacks correspond to costing, readiness/pop inspection and final selection. A nonempty result is an owned
HashSetclone each time; the observed cardinality/capacity pairs are1/3,8/14and128/224. Empty snapshots returnNone.The isolated microbenchmark calls the actual Client getter and drops the result. It includes snapshot/clock lookup, allocation/clone and drop, and excludes indirect callback dispatch. Head cell medians were23.66–23.74ns for empty snapshots,30.15–30.58ns for1/8IDs, and47.72–48.41ns for128IDs. Extra callback count × hot getter time estimates95.4ns per idle subject at128IDs, or206.5ns per subject in the64-subject/16-update case. These estimates cover the getter component only; they are not total CPU-overhead bounds or production latency.
The full cycle results are retained below, including the larger blocked-case costs. An idle cycle is admit/free plus an explicit queue barrier; a blocked cycle creates the held booking, offers subjects, issues16busy barriers, releases B and drains all subjects.
The base immediately dispatches to busyB, while the fixed head actually queues4/64subjects and performs capacity checks. Even the empty-set blocked controls are1.8471×/2.6436×, so the total cycle difference is not solely nonempty-set cloning. These are different scheduler-work states, not a same-service-order performance comparison. No inference speedup, slowdown, SLO result, statistical equivalence or universal overhead acceptance is claimed.
The current implementation samples at costing, queued readiness and final selection. Caching one set across an entire queue wait or overlap-refresh await would need a separate freshness contract to preserve overload fallback. This follow-up provides the requested cost evidence while retaining the correctness fix.
Real Client overload-provider reproducer
Use the standalone fixture layout/dependencies embedded in #17310, with upstream base
ad1539d8eebddc37121662e9a77aee6295131330or this PR head0b3f7e84bcaac29e4992011fade7a79b9d722f0e. Add this dependency under[dependencies](standard Runtime build prerequisites, including protoc, apply):Save the program below as
scripts/affinity_queue_scope_audit/src/main.rs; retain the fixture's builtin-policy inclusion insrc/lib.rs. Seed its lock from the upstream checkout, build with Rust1.96.1 in release mode, and keep the generated474-package lock identical across variants. Retain separate base/head executables.Arguments are
workers overload_ids backlog busy_updates count|time iterations. Count mode runs once with counters; time mode uses the unchanged embedded callback without counters. Example:taskset -c 0 ./head-overload 512 128 64 16 count 1 taskset -c 0 ./head-overload 512 128 64 16 time 10Choose an allowed CPU on another machine. The fixed matrix is32workers ×overload0/1/8 ×backlog0/4 and512workers ×overload0/1/128 ×backlog0/64. Set updates=0/16 and iterations=1000/10 for idle/blocked cases. Run each timing cell as six adjacent base/head process pairs with alternating order. Getter timing retains20samples after3warmups; queue-cycle timing retains10samples after3warmups. Counter runs are separate. The table uses process medians and median paired ratios, retaining all cells.