Repository navigation
feat(router): add pluggable queue admission policy - #13019
ishandhanani wants to merge 5 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
ishandhanani
left a comment
There was a problem hiding this comment.
Full review found five correctness or integration blockers and two hot-path concerns. CI and the current happy-path tests pass, but these cases need fixes before merge.
9215e1d to
3e3000a
Compare
3e3000a to
0af2ac6
Compare
WalkthroughChangesThe scheduler now supports queue-admission policies with deferred requests, worker eligibility, lifecycle events, reconciliation, and completion or abort reporting. Request cleanup passes worker ownership and final context-token usage. A Criterion benchmark measures admission scheduling across worker counts. Queue admission
Estimated code review effort: 4 (Complex) | ~60 minutes Mergeability Score: 🟡 Moderate · up to This PR adds policy lifecycle reporting and admission control, but slot-release failures can prevent terminal events from reaching the policy, leaving requests accounted for indefinitely and potentially disrupting wake-up or fairness state. Tracked requests may also bypass admission, so policies can see incomplete partition load. Merge should wait for these paths to be fixed or explicitly accepted. 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Comment |
There was a problem hiding this comment.
Full review at cc594734. This is not ready for approval because the standalone lifecycle still has four concrete failure paths.
The current open threads also cover the required public-API documentation and the incorrect is_available documentation. I did not duplicate those comments.
Validation:
cargo test -p dynamo-kv-router --lib: 870 passed, 2 ignored.cargo test -p dynamo-llm push_router --lib: 29 passed.- The external
router-sandboxThunderAgent tests passed: 10 tests plus clippy. - CI currently has 67 passed, 1 pending, and 0 failed checks.
The handwritten diff contains 1,596 changed lines. This is slightly more than the reviewability threshold. A clean split is: terminal lifecycle modeling and host plumbing first; admission storage, public integration, documentation, example, and benchmark second.
There was a problem hiding this comment.
Rereviewed the exact head 5b18d33075d3b0a7bd2600f12fbfe0ca7b64bdf4. I found no remaining actionable correctness, lifecycle, or performance issue.
Verified contracts:
- Terminal admission state survives worker removal and slot-release errors.
- Standalone reservations distinguish Completed from Aborted and roll back post-booking endpoint failures.
- Multi-partition release finds the owning partition while missing/repeated releases remain idempotent.
- Queue admission rejects duplicate request IDs across the full active lifecycle, including deferred and bypassed requests, and emits at most one terminal policy event.
- The canonical session policy matches the ThunderAgent session-serialization need.
- Stale worker-qualified terminal calls remain deliberate no-ops to protect against request-ID reuse/ABA.
Validation:
cargo test -p dynamo-kv-router --features standalone-selection --lib --quiet: 969 passed, 2 ignored.cargo test -p dynamo-llm push_router --lib --quiet: 29 passed.- Canonical custom-policy example: 2 passed.
- ThunderAgent reference crate against this Git head: 10 passed.
- Teleport 64-core run: 726/726
dynamo-mockertests passed. - try6767 feature-complete mocker/indexer/AIC rerun: 5/5 passed; the broader matrix completed 48 passed and 6 skipped.
- Targeted clippy passed except for the unrelated pre-existing dead-code warning in
services/indexer/registry.rs. - All review threads are resolved; full CI is running on this exact head.
Non-blocking reviewability note: this remains an XXL change at 2,383 added/deleted lines. If further work lands before merge, keep it separate; future changes of this shape should split the policy API, lifecycle plumbing, and docs/example/benchmark surfaces.
ishandhanani
left a comment
There was a problem hiding this comment.
Canonical queue-admission overhead benchmark. This supersedes the earlier performance numbers on this PR.
Method: TRY-67676, one pinned CPU, alternating enabled/disabled order. Each operation schedules one pinned request and completes it. Times are medians across rounds; the delta is the median paired delta.
The no-TA case installs a stateless TrivialReadyPolicy. It always returns Ready, observes dispatch and completion, and stores no state. This case measures the full plugin lifecycle without ThunderAgent logic.
| Policy | Workers | Disabled | Enabled | Paired delta |
|---|---|---|---|---|
Trivial Ready policy (no TA) |
32 | 4.769 us | 7.589 us | +2.834 us (+59.4%) |
Trivial Ready policy (no TA) |
1,024 | 11.954 us | 14.953 us | +3.025 us (+25.3%) |
Trivial Ready policy (no TA) |
10,000 | 110.77 us | 118.29 us | +6.960 us (+6.3%) |
| ThunderAgent | 32 | 5.133 us | 9.258 us | +4.120 us (+80.3%) |
| ThunderAgent | 1,024 | 19.425 us | 23.092 us | +3.663 us (+18.9%) |
| ThunderAgent | 10,000 | 157.98 us | 162.79 us | +4.840 us (+3.1%) |
The trivial policy used seven paired rounds. ThunderAgent used three paired rounds. Compare enabled and disabled within each policy; the two policies use separate benchmark binaries and baselines.
The no-TA full-lifecycle seam adds about 3 us through 1,024 workers. At 10,000 workers it adds 7.0 us, or 6.3%. ThunderAgent adds 3.7-4.8 us from 1,024 through 10,000 workers, or 3.1% at 10,000.
ishandhanani
left a comment
There was a problem hiding this comment.
Rereviewed the current head after the new Python-binding finding. SelectionService.abort_reservation now matches the Rust and HTTP surfaces, the type stub is updated, and the feature-enabled binding test exercises both terminal paths. Validation: the select-service Python extension built successfully, 20 KV binding tests passed, strict bindings clippy passed, and formatting/diff checks passed. No remaining actionable finding in the follow-up diff; the PR description remains the canonical benchmark record.
39e7566 to
6dacb3b
Compare
5923529 to
6dacb3b
Compare
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
d0c31f8 to
3134471
Compare
ishandhanani
left a comment
There was a problem hiding this comment.
Full rereview of the rewritten queue-admission stack is complete. No remaining material correctness, lifecycle, compatibility, or unchanged-topology hot-path finding was identified.
The stack is now reviewable by responsibility: #13019 is the dormant public API foundation, #13394 is the queue-actor host plus example and benchmark, and #13378 bridges terminal reservation state and live response progress.
Verified the deferred-abort and final-watch-close regressions, API compatibility for the prior module and overload-provider surface, policy-disabled behavior, and retry ordering. Local validation includes 997 KV-router library tests, 12 response-path lifecycle tests, LLM check, strict default KV-router clippy, formatting, and diff checks.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This introduces the standalone queue-admission policy API and moves policy-queue types under
scheduling::queue_admission. It establishes a standalone foundation for session-aware policies without activating a policy in the scheduler yet.How This Was Implemented
scheduling::queue_admissionand update Dynamo’s internal imports to the new module.Walkthrough
Mental model
The API separates an external policy’s accounting from Dynamo’s request and worker ownership. A policy can decide
Bypass,Ready, orDeferand later wake a request by opaque admission ID; it cannot mutate Dynamo’s queue or choose a worker directly.flowchart LR A[External policy] -->|decision or wake ID| B[Queue-admission API] B -->|implemented by next PR| C[Scheduler host] C --> D[Existing worker selection]Boundaries and limitations
This foundation changes no scheduler behavior by itself. The following PR activates the host inside the queue actor; the final stacked PR bridges terminal reservation state and response progress. The former
scheduling::policy_queuemodule path is intentionally removed; the public queue-admission surface lives underscheduling::queue_admission.Validation
cargo test -p dynamo-kv-router --lib— 885 passed.cargo clippy -p dynamo-kv-router --lib -- -D warnings.cargo fmt --all --checkandgit diff --check.