Repository navigation
perf(router): keep prepared rows when ranks are past the reject thresholds - #18009
Conversation
|
@coderabbitai full review |
Action performed
|
|
No actionable comments were generated in the recent review. WalkthroughPrepared candidate rows now retain worker data-parallel size. Selection can remove rows for workers whose ranks are all full, subject to request conditions, and update cached-token tracking. ChangesPrepared Selection
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk:
|
a20f055 to
6767a1c
Compare
❌ Dynamo PR CI failed — run 41780841256 (attempt 1) on
|
| Other | Jobs |
|---|---|
| dynamo-runtime | ❌ 2 ✅ 5 |
| planner | ❌ 2 ✅ 3 |
Failure details
4 jobs failed, all with the same 2 tests in components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py: the --router-mode parser rejects power-of-two (allowed choices: round-robin, random, kv, direct, least-loaded, device-aware-weighted). No cancelled jobs.
❌ planner / CPU Test cpu, arm64: router-mode 'power-of-two' rejected (2 planner tests)
Job: planner / CPU Test cpu, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126415485252 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 4171 passed, 29 skipped in 538.02s ============
test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.
❌ planner / CPU Test cpu, amd64: router-mode 'power-of-two' rejected (2 planner tests)
Job: planner / CPU Test cpu, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126415485349 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 4171 passed, 29 skipped in 452.11s ============
test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.
❌ dynamo-runtime / test / sequential cuda13.0, amd64: router-mode 'power-of-two' rejected (2 planner tests)
Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126416721585 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 7778 passed, 170 skipped, 2854 deselected in 1604.47s ============
test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.
❌ dynamo-runtime / test / sequential cuda13.0, arm64: router-mode 'power-of-two' rejected (2 planner tests)
Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126416721592 -R ai-dynamo/dynamo --log-failed
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two] - AssertionError: P4: cannot parse the frontend args: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
FAILED components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3] - SystemExit: 2
args = ['--model=Qwen/Qwen3-32B', '--router-mode=power-of-two']
E argparse.ArgumentError: argument --router-mode: invalid choice: 'power-of-two' (choose from 'round-robin', 'random', 'kv', 'direct', 'least-loaded', 'device-aware-weighted')
============ 2 failed, 7778 passed, 170 skipped, 2854 deselected in 1919.98s ============
test_effective_argv[...power_of_two] and test_worker_router_mode_matches_the_real_worker_parser[args3-env3] both pass --router-mode power-of-two to the real frontend/worker argument parser, and the parser does not list power-of-two as a valid choice. Auto test retries failed the same way, so this is deterministic, not an infrastructure issue.
For agents
{
"pr": 18009,
"run_id": 41780841256,
"run_attempt": 1,
"head_sha": "6767a1c1c2a5ce217dfbe7e957223e371a952e36",
"failures": [
{
"job": "planner / CPU Test cpu, arm64",
"job_id": 126415485252,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "--router-mode: invalid choice: 'power-of-two'",
"tests": [
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
],
"log_cmd": "gh run view --job 126415485252 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "planner / CPU Test cpu, amd64",
"job_id": 126415485349,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "--router-mode: invalid choice: 'power-of-two'",
"tests": [
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
],
"log_cmd": "gh run view --job 126415485349 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "dynamo-runtime / test / sequential cuda13.0, amd64",
"job_id": 126416721585,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "--router-mode: invalid choice: 'power-of-two'",
"tests": [
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
],
"log_cmd": "gh run view --job 126416721585 -R ai-dynamo/dynamo --log-failed"
},
{
"job": "dynamo-runtime / test / sequential cuda13.0, arm64",
"job_id": 126416721592,
"failed_step": "Run CPU-only tests (parallelized)",
"signature": "--router-mode: invalid choice: 'power-of-two'",
"tests": [
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_effective_argv[command7-args7-power_of_two]",
"components/src/dynamo/planner/tests/unit/test_live_retune_frontend_config.py::test_worker_router_mode_matches_the_real_worker_parser[args3-env3]"
],
"log_cmd": "gh run view --job 126416721592 -R ai-dynamo/dynamo --log-failed"
}
]
}Posted automatically by Devin for run 41780841256. Updated on every full-CI run of this PR.
6767a1c to
d2a1678
Compare
|
/ok to test d2a1678 |
…holds The scheduler actor discarded the caller's prepared candidate rows whenever any rank was past the host's reject thresholds, and rebuilt every row itself. Under real busy thresholds some rank is nearly always full, so every admission paid the full eligibility scan on the actor. The actor now keeps the prepared rows and drops only the rows of workers whose every rank is full. Prepared rows carry the worker's data-parallel size so the check needs no config lookup. The kept rows are the rows the actor would build, in the same order, and the best eligible cached prefix is recounted when a dropped row held it. Requests with an exclusive affinity target or KV transfer candidates still rebuild, since preparation could not know which ranks are full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
d2a1678 to
8e74c62
Compare
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
…y queue test #17725 added a queue test that calls counting_policy with one argument; this PR gave it an exclusive_affinity flag. Pass false, the old behavior. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Fourth upstream merge for the split v2 PR: the five commits past 02331f0. Four conflicts, all from ai-dynamo#18009, placed in the owner crates; upstream behavior whole, the stack's moves intact, no shims. - ai-dynamo#18009 keep prepared rows when ranks are past the reject thresholds: is_worker_full(dp_size)/skips_full_workers in the kernel's filter.rs; the actor keeps prepared rows and the kernel's select_with_policy drops the rows of workers whose every rank is full before the policy adopts them (PreparedRows::retain in worker-selection; the policy recounts the best cached prefix from the rows it adopts and reports whether any is left); prepared rows carry dp_size (u16, so the 128-byte row holds) set in the kernel's CandidatePreparerExt::prepare; the three new kernel queue tests and the relocated test helpers updated. - ai-dynamo#18108 CRTC decode stores appended at the leaf tail under one lock: lib/router/kv-indexer concurrent_radix_tree_compressed (auto-merged). - ai-dynamo#16258 vLLM backend defaults for omitted token limits: lib/llm preprocessor, lib/sidecar/vllm, components, docs, tests (auto-merged). - ai-dynamo#18152 planner ARIMA forecasts, ai-dynamo#16426 operator crash-loop: outside the router (auto-merged). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Overview
Since #17465, callers build the candidate rows before a request reaches the scheduler actor, and the actor only fills in loads. But the actor threw the prepared rows away whenever any rank was past the host's reject thresholds (
active_decode_blocks_threshold,active_prefill_tokens_threshold[_frac]), and rebuilt every row itself. With real thresholds, some rank is nearly always full, so every admission paid the full eligibility scan on the actor. Now the actor keeps the prepared rows and drops only the rows of workers whose every rank is full.Setup:
selection_replay_bench(#17959) on one Grace node, thresholds--active-decode-blocks-threshold 0.25 --active-prefill-tokens-threshold-frac 1.0, arms alternated on the same CPUs.Key Changes
collect_prepared_candidatestakes the actor's eligibility and drops rows of fully full workers. The kept rows are the same rows the rebuild produced, in the same order. An admission where every worker is full still returnsAllEligibleWorkersOverloaded.CandidateData.dp_size, which fits in existing padding; the row stays 128 bytes). The full-worker check is one probe of the full-rank map, with no config lookup.max_raw_cached_tokens) is recounted over the kept rows only when a dropped row held it.Validation
cargo test -p dynamo-kv-router, andcargo clippy -p dynamo-kv-router --all-targets [--features bench] -- -D warnings.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.🤖 Generated with Claude Code
Summary by CodeRabbit