Skip to content

perf(router): keep prepared rows when ranks are past the reject thresholds - #18009

Merged
jthomson04 merged 3 commits into
mainfrom
jthomson04/prepared-rows-full-ranks
Nov 5, 2026
Merged

jthomson04 merged 3 commits into
mainfrom
jthomson04/prepared-rows-full-ranks

Conversation

@jthomson04

@jthomson04 jthomson04 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Since #17465, callers build the candidate rows before a request reaches the scheduler actor, and the actor only fills in loads. But the actor threw the prepared rows away whenever any rank was past the host's reject thresholds (active_decode_blocks_threshold, active_prefill_tokens_threshold[_frac]), and rebuilt every row itself. With real thresholds, some rank is nearly always full, so every admission paid the full eligibility scan on the actor. Now the actor keeps the prepared rows and drops only the rows of workers whose every rank is full.

2,048 workers, agentx workload main this PR
Busy thresholds on: booked req/s at saturation (median of 3) 7,172 10,013 (+40%)
Busy thresholds on: actor kcycles per booking 437 314 (−28%)
Busy thresholds on: admission p99 at a fixed 4.6k req/s 2,764 ms 2.1 ms
No thresholds: booked req/s (mean of 2 pairs) 12,353 12,300

Setup: selection_replay_bench (#17959) on one Grace node, thresholds --active-decode-blocks-threshold 0.25 --active-prefill-tokens-threshold-frac 1.0, arms alternated on the same CPUs.

Key Changes

  • Prepared rows survive full ranks. collect_prepared_candidates takes the actor's eligibility and drops rows of fully full workers. The kept rows are the same rows the rebuild produced, in the same order. An admission where every worker is full still returns AllEligibleWorkersOverloaded.
  • Each prepared row carries its worker's DP size (CandidateData.dp_size, which fits in existing padding; the row stays 128 bytes). The full-worker check is one probe of the full-rank map, with no config lookup.
  • The best eligible cached prefix (max_raw_cached_tokens) is recounted over the kept rows only when a dropped row held it.
  • Requests with an exclusive affinity target or KV-transfer candidates still rebuild once a rank is full, since preparation could not know which ranks are full.

Validation

  • New queue tests:
    • a DP=2 worker with one full rank stays eligible;
    • a fully full worker is dropped;
    • when every worker is full, the request is rejected;
    • an exclusive-affinity target that is full triggers a rebuild;
    • the cached-prefix recount.
  • cargo test -p dynamo-kv-router, and cargo clippy -p dynamo-kv-router --all-targets [--features bench] -- -D warnings.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Improvements
    • Candidate information can now be reused in more scheduling scenarios, reducing unnecessary preparation while preserving routing behavior for affinity-targeted and KV-transfer requests.
    • Workers with all ranks at capacity are excluded from eligible choices when appropriate.
    • Cached-prefix calculations are updated when full workers are excluded, helping selection reflect the remaining candidates.

@jthomson04

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor
Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

Prepared candidate rows now retain worker data-parallel size. Selection can remove rows for workers whose ranks are all full, subject to request conditions, and update cached-token tracking.

Changes

Prepared Selection

Layer / File(s) Summary
Worker fullness metadata
lib/kv-router/src/plugins/worker_selection/inputs.rs, lib/kv-router/src/scheduling/filter.rs, lib/kv-router/src/scheduling/selector/mod.rs, lib/kv-router/src/scheduling/selector/policy.rs
Candidate rows retain data-parallel size. Routing eligibility provides checks for whether all ranks are full and whether full workers are skipped.
Prepared row filtering
lib/kv-router/src/scheduling/selector/mod.rs, lib/kv-router/src/scheduling/selector/policy.rs
Prepared selection removes rows for fully loaded workers when eligibility skips full workers. It recalculates the tracked maximum cached-token count when a removed row held the maximum and reports whether any candidates remain.
Prepared queue reuse
lib/kv-router/src/scheduling/queue.rs
Prepared rows remain eligible for reuse when full-worker filtering is allowed. Tests cover multi-rank workers, exclusive affinity targets, and cached-prefix recalculation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to a20f0

Prepared selection retains eligible rows while filtering fully loaded workers. No merge-blocking risk was established.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title "perf(router): keep prepared rows when ranks are past the reject thresholds" matches the main change. The PR stops the scheduler actor from discarding caller-prepared candidate rows when ran…
Description check Passed The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, and it includes the Pull Request Checklist block. The Overview explains…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "perf(router): keep prepared rows when ranks are past the reject thresholds" matches the main change. The PR stops the scheduler actor from discarding caller-prepared candidate rows when ranks exceed the reject thresholds. It drops only rows for workers whose every rank is full. The title is specific, uses a conventional commit prefix, and is under 75 characters. It does not name the affinity and KV-transfer rebuild exceptions, but that detail is not required.

Full details: Description check

Explanation

The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, and it includes the Pull Request Checklist block. The Overview explains the problem and the fix, and it gives benchmark numbers. Key Changes lists the behavior changes, including the DP size field, the cached-prefix recount, and the rebuild exceptions. Validation lists the new queue tests and the cargo test and clippy commands. The Issues section has no linked issue. The template line 'Closes #XXXX' was replaced by nothing, so the advisory PR Issue Link check may flag it. The checklist item is unchecked.

Resolution

Optional: add a 'Closes #NNNN' line under Issues and Ongoing Work, or state that no issue applies. The PR Issue Link check is advisory now but becomes required when release/1.7.0 is cut. Also consider checking the '/pr-checklist' item if the skill was run.

Full details: Docstring Coverage

Explanation

Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 10 functions across 5 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between a80115b and a20f055.

Files selected for processing: 5
  • lib/kv-router/src/plugins/worker_selection/inputs.rs
  • lib/kv-router/src/scheduling/filter.rs
  • lib/kv-router/src/scheduling/queue.rs
  • lib/kv-router/src/scheduling/selector/mod.rs
  • lib/kv-router/src/scheduling/selector/policy.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@jthomson04
jthomson04 force-pushed the jthomson04/prepared-rows-full-ranks branch from a20f055 to 6767a1c Compare November 4, 2026 05:41
@devin-ai-integration

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 41780841256 (attempt 1) on 6767a1c1c2

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Other Jobs
dynamo-runtime ❌ 2 ✅ 5
planner ❌ 2 ✅ 3

Failure details

4 jobs failed, all with the same 2 tests in components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py: the --router-mode parser rejects power-of-two (allowed choices: round-robin, random, kv, direct, least-loaded, device-aware-weighted). No cancelled jobs.

❌ planner / CPU Test cpu, arm64: router-mode 'power-of-two' rejected (2 planner tests)

Job: planner / CPU Test cpu, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126415485252 -R ai-dynamo/dynamo --log-failed

FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
  args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E   argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 4171 passed, 29 skipped in 538.02s ============

test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.

❌ planner / CPU Test cpu, amd64: router-mode 'power-of-two' rejected (2 planner tests)

Job: planner / CPU Test cpu, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126415485349 -R ai-dynamo/dynamo --log-failed

FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
  args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E   argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 4171 passed, 29 skipped in 452.11s ============

test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.

❌ dynamo-runtime / test / sequential cuda13.0, amd64: router-mode 'power-of-two' rejected (2 planner tests)

Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126416721585 -R ai-dynamo/dynamo --log-failed

FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
  args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E   argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 7778 passed, 170 skipped, 2854 deselected in 1604.47s ============

test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.

❌ dynamo-runtime / test / sequential cuda13.0, arm64: router-mode 'power-of-two' rejected (2 planner tests)

Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126416721592 -R ai-dynamo/dynamo --log-failed

FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
  args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E   argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 7778 passed, 170 skipped, 2854 deselected in 1919.98s ============

test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.

For agents
{
 "pr": 18009,
 "run_id": 41780841256,
 "run_attempt": 1,
 "head_sha": "6767a1c1c2a5ce217dfbe7e957223e371a952e36",
 "failures": [
  {
   "job": "planner / CPU Test cpu, arm64",
   "job_id": 126415485252,
   "failed_step": "Run CPU-only tests (parallelized)",
   "signature": "--router-mode: invalid choice: 'power-of-two'",
   "tests": [
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
   ],
   "log_cmd": "gh run view --job 126415485252 -R ai-dynamo/dynamo --log-failed"
  },
  {
   "job": "planner / CPU Test cpu, amd64",
   "job_id": 126415485349,
   "failed_step": "Run CPU-only tests (parallelized)",
   "signature": "--router-mode: invalid choice: 'power-of-two'",
   "tests": [
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
   ],
   "log_cmd": "gh run view --job 126415485349 -R ai-dynamo/dynamo --log-failed"
  },
  {
   "job": "dynamo-runtime / test / sequential cuda13.0, amd64",
   "job_id": 126416721585,
   "failed_step": "Run CPU-only tests (parallelized)",
   "signature": "--router-mode: invalid choice: 'power-of-two'",
   "tests": [
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
   ],
   "log_cmd": "gh run view --job 126416721585 -R ai-dynamo/dynamo --log-failed"
  },
  {
   "job": "dynamo-runtime / test / sequential cuda13.0, arm64",
   "job_id": 126416721592,
   "failed_step": "Run CPU-only tests (parallelized)",
   "signature": "--router-mode: invalid choice: 'power-of-two'",
   "tests": [
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
    "components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
   ],
   "log_cmd": "gh run view --job 126416721592 -R ai-dynamo/dynamo --log-failed"
  }
 ]
}

Posted automatically by Devin for run 41780841256. Updated on every full-CI run of this PR.

@jthomson04
jthomson04 marked this pull request as ready for review November 4, 2026 17:30
@jthomson04
jthomson04 requested a review from a team as a code owner November 4, 2026 17:30
@jthomson04
jthomson04 force-pushed the jthomson04/prepared-rows-full-ranks branch from 6767a1c to d2a1678 Compare November 4, 2026 17:36
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test d2a1678

…holds

The scheduler actor discarded the caller's prepared candidate rows whenever
any rank was past the host's reject thresholds, and rebuilt every row
itself. Under real busy thresholds some rank is nearly always full, so every
admission paid the full eligibility scan on the actor.

The actor now keeps the prepared rows and drops only the rows of workers
whose every rank is full. Prepared rows carry the worker's data-parallel
size so the check needs no config lookup. The kept rows are the rows the
actor would build, in the same order, and the best eligible cached prefix is
recounted when a dropped row held it. Requests with an exclusive affinity
target or KV transfer candidates still rebuild, since preparation could not
know which ranks are full.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the jthomson04/prepared-rows-full-ranks branch from d2a1678 to 8e74c62 Compare November 4, 2026 21:31
@jthomson04
jthomson04 enabled auto-merge (squash) November 4, 2026 21:51
jthomson04 and others added 2 commits November 4, 2026 16:04
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…y queue test

#17725 added a queue test that calls counting_policy with one argument;
this PR gave it an exclusive_affinity flag. Pass false, the old behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 merged commit 4f71537 into main Nov 5, 2026
198 of 201 checks passed
@jthomson04
jthomson04 deleted the jthomson04/prepared-rows-full-ranks branch November 5, 2026 03:54
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 5, 2026
Fourth upstream merge for the split v2 PR: the five commits past
02331f0. Four conflicts, all from ai-dynamo#18009, placed in the owner crates;
upstream behavior whole, the stack's moves intact, no shims.

- ai-dynamo#18009 keep prepared rows when ranks are past the reject thresholds:
  is_worker_full(dp_size)/skips_full_workers in the kernel's filter.rs;
  the actor keeps prepared rows and the kernel's select_with_policy drops
  the rows of workers whose every rank is full before the policy adopts
  them (PreparedRows::retain in worker-selection; the policy recounts the
  best cached prefix from the rows it adopts and reports whether any is
  left); prepared rows carry dp_size (u16, so the 128-byte row holds) set
  in the kernel's CandidatePreparerExt::prepare; the three new kernel
  queue tests and the relocated test helpers updated.
- ai-dynamo#18108 CRTC decode stores appended at the leaf tail under one lock:
  lib/router/kv-indexer concurrent_radix_tree_compressed (auto-merged).
- ai-dynamo#16258 vLLM backend defaults for omitted token limits: lib/llm
  preprocessor, lib/sidecar/vllm, components, docs, tests (auto-merged).
- ai-dynamo#18152 planner ARIMA forecasts, ai-dynamo#16426 operator crash-loop: outside the
  router (auto-merged).

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants