Skip to content

feat(router): add pluggable queue admission policy - #13019

Closed
ishandhanani wants to merge 5 commits into
mainfrom
idhanani/dyn-3231-session-aware-policy-crate
Closed

ishandhanani wants to merge 5 commits into
mainfrom
idhanani/dyn-3231-session-aware-policy-crate

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

This introduces the standalone queue-admission policy API and moves policy-queue types under scheduling::queue_admission. It establishes a standalone foundation for session-aware policies without activating a policy in the scheduler yet.

How This Was Implemented

  • Define the object-safe policy decisions, request views, worker snapshots, admission IDs, and lifecycle event vocabulary.
  • Move policy queue primitives into scheduling::queue_admission and update Dynamo’s internal imports to the new module.
  • Keep the foundation dormant behind the next stacked host PR, with narrow documented allowances only for host-only methods.
Walkthrough

Mental model

The API separates an external policy’s accounting from Dynamo’s request and worker ownership. A policy can decide Bypass, Ready, or Defer and later wake a request by opaque admission ID; it cannot mutate Dynamo’s queue or choose a worker directly.

flowchart LR
    A[External policy] -->|decision or wake ID| B[Queue-admission API]
    B -->|implemented by next PR| C[Scheduler host]
    C --> D[Existing worker selection]
Loading

Boundaries and limitations

This foundation changes no scheduler behavior by itself. The following PR activates the host inside the queue actor; the final stacked PR bridges terminal reservation state and response progress. The former scheduling::policy_queue module path is intentionally removed; the public queue-admission surface lives under scheduling::queue_admission.

Validation

  • cargo test -p dynamo-kv-router --lib — 885 passed.
  • cargo clippy -p dynamo-kv-router --lib -- -D warnings.
  • cargo fmt --all --check and git diff --check.

@github-actions github-actions Bot added the feat label Aug 11, 2026
@github-actions github-actions Bot added the router Relates to routing, KV-aware routing, etc. label Aug 11, 2026
@datadog-official

This comment has been minimized.

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full review found five correctness or integration blockers and two hot-path concerns. CI and the current happy-path tests pass, but these cases need fixes before merge.

Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Comment thread lib/kv-router/src/scheduling/local.rs Resolved
Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
Comment thread lib/kv-router/src/scheduling/queue.rs Resolved
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 11, 2026
@github-actions

github-actions Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-session-aware-policy-crate branch from 9215e1d to 3e3000a Compare August 11, 2026 18:55
Base automatically changed from idhanani/dyn-3712-worker-policy-api-hardening to main August 11, 2026 22:27
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-session-aware-policy-crate branch from 3e3000a to 0af2ac6 Compare August 11, 2026 23:19
@ishandhanani
ishandhanani marked this pull request as ready for review August 13, 2026 05:29
@ishandhanani
ishandhanani requested a review from a team as a code owner August 13, 2026 05:29
@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

The scheduler now supports queue-admission policies with deferred requests, worker eligibility, lifecycle events, reconciliation, and completion or abort reporting. Request cleanup passes worker ownership and final context-token usage. A Criterion benchmark measures admission scheduling across worker counts.

Queue admission

Layer / File(s) Summary
Admission contracts and policy transfer
lib/kv-router/src/scheduling/queue_admission/mod.rs, lib/kv-router/src/scheduling/selector/*, lib/kv-router/src/scheduling/mod.rs, lib/kv-router/src/lib.rs
Defines queue-admission requests, workers, decisions, events, policies, and selector policy transfer.
Deferred admission queue state
lib/kv-router/src/scheduling/policy_queue.rs
Stores deferred entries, processes wake-up events, updates queue accounting, and tests reconciliation admission.
Scheduler admission lifecycle
lib/kv-router/src/scheduling/queue.rs, lib/kv-router/src/scheduling/local.rs
Integrates admission with scheduling, worker state, bookings, reconciliation, completion, abort, rejection, and lifecycle tests.
Request cleanup and reservation integration
lib/kv-router/src/services/selection/*, lib/llm/src/kv_router.rs, lib/llm/src/kv_router/scheduler.rs, lib/llm/src/kv_router/push_router/request_guard.rs
Uses lifecycle-aware reservations and propagates worker-validated completion, abort, and context-token data.
Admission scaling benchmark
lib/kv-router/Cargo.toml, lib/kv-router/benches/policy_queue_admission.rs
Adds a Criterion benchmark for tracked scheduling and worker-pinned completion with 32, 1,024, and 10,000 workers.

Estimated code review effort: 4 (Complex) | ~60 minutes

Mergeability Score: 🟡 Moderate · up to cc594

This PR adds policy lifecycle reporting and admission control, but slot-release failures can prevent terminal events from reaching the policy, leaving requests accounted for indefinitely and potentially disrupting wake-up or fairness state. Tracked requests may also bypass admission, so policies can see incomplete partition load. Merge should wait for these paths to be fixed or explicitly accepted.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description explains the API, implementation, boundaries, and validation, but it omits the required Related Issues section and reviewer-start guidance. Add the required Related Issues section with an issue link or no-issue confirmation, and identify the files where reviewers should start.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely identifies the main change: adding a pluggable queue admission policy.

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani ishandhanani left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full review at cc594734. This is not ready for approval because the standalone lifecycle still has four concrete failure paths.

The current open threads also cover the required public-API documentation and the incorrect is_available documentation. I did not duplicate those comments.

Validation:

  • cargo test -p dynamo-kv-router --lib: 870 passed, 2 ignored.
  • cargo test -p dynamo-llm push_router --lib: 29 passed.
  • The external router-sandbox ThunderAgent tests passed: 10 tests plus clippy.
  • CI currently has 67 passed, 1 pending, and 0 failed checks.

The handwritten diff contains 1,596 changed lines. This is slightly more than the reviewability threshold. A clean split is: terminal lifecycle modeling and host plumbing first; admission storage, public integration, documentation, example, and benchmark second.

Comment thread lib/kv-router/src/scheduling/local.rs Resolved
@ishandhanani
ishandhanani requested a review from a team as a code owner August 13, 2026 07:35
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani ishandhanani left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rereviewed the exact head 5b18d33075d3b0a7bd2600f12fbfe0ca7b64bdf4. I found no remaining actionable correctness, lifecycle, or performance issue.

Verified contracts:

  • Terminal admission state survives worker removal and slot-release errors.
  • Standalone reservations distinguish Completed from Aborted and roll back post-booking endpoint failures.
  • Multi-partition release finds the owning partition while missing/repeated releases remain idempotent.
  • Queue admission rejects duplicate request IDs across the full active lifecycle, including deferred and bypassed requests, and emits at most one terminal policy event.
  • The canonical session policy matches the ThunderAgent session-serialization need.
  • Stale worker-qualified terminal calls remain deliberate no-ops to protect against request-ID reuse/ABA.

Validation:

  • cargo test -p dynamo-kv-router --features standalone-selection --lib --quiet: 969 passed, 2 ignored.
  • cargo test -p dynamo-llm push_router --lib --quiet: 29 passed.
  • Canonical custom-policy example: 2 passed.
  • ThunderAgent reference crate against this Git head: 10 passed.
  • Teleport 64-core run: 726/726 dynamo-mocker tests passed.
  • try6767 feature-complete mocker/indexer/AIC rerun: 5/5 passed; the broader matrix completed 48 passed and 6 skipped.
  • Targeted clippy passed except for the unrelated pre-existing dead-code warning in services/indexer/registry.rs.
  • All review threads are resolved; full CI is running on this exact head.

Non-blocking reviewability note: this remains an XXL change at 2,383 added/deleted lines. If further work lands before merge, keep it separate; future changes of this shape should split the policy API, lifecycle plumbing, and docs/example/benchmark surfaces.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Canonical queue-admission overhead benchmark. This supersedes the earlier performance numbers on this PR.

Method: TRY-67676, one pinned CPU, alternating enabled/disabled order. Each operation schedules one pinned request and completes it. Times are medians across rounds; the delta is the median paired delta.

The no-TA case installs a stateless TrivialReadyPolicy. It always returns Ready, observes dispatch and completion, and stores no state. This case measures the full plugin lifecycle without ThunderAgent logic.

Policy Workers Disabled Enabled Paired delta
Trivial Ready policy (no TA) 32 4.769 us 7.589 us +2.834 us (+59.4%)
Trivial Ready policy (no TA) 1,024 11.954 us 14.953 us +3.025 us (+25.3%)
Trivial Ready policy (no TA) 10,000 110.77 us 118.29 us +6.960 us (+6.3%)
ThunderAgent 32 5.133 us 9.258 us +4.120 us (+80.3%)
ThunderAgent 1,024 19.425 us 23.092 us +3.663 us (+18.9%)
ThunderAgent 10,000 157.98 us 162.79 us +4.840 us (+3.1%)

The trivial policy used seven paired rounds. ThunderAgent used three paired rounds. Compare enabled and disabled within each policy; the two policies use separate benchmark binaries and baselines.

The no-TA full-lifecycle seam adds about 3 us through 1,024 workers. At 10,000 workers it adds 7.0 us, or 6.3%. ThunderAgent adds 3.7-4.8 us from 1,024 through 10,000 workers, or 3.1% at 10,000.

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rereviewed the current head after the new Python-binding finding. SelectionService.abort_reservation now matches the Rust and HTTP surfaces, the type stub is updated, and the feature-enabled binding test exercises both terminal paths. Validation: the select-service Python extension built successfully, 20 KV binding tests passed, strict bindings clippy passed, and formatting/diff checks passed. No remaining actionable finding in the follow-up diff; the PR description remains the canonical benchmark record.

devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani requested review from a team as code owners August 13, 2026 22:15
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-session-aware-policy-crate branch from 39e7566 to 6dacb3b Compare August 16, 2026 19:30
devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-session-aware-policy-crate branch from 5923529 to 6dacb3b Compare August 17, 2026 18:18
devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-session-aware-policy-crate branch from d0c31f8 to 3134471 Compare August 17, 2026 21:47

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full rereview of the rewritten queue-admission stack is complete. No remaining material correctness, lifecycle, compatibility, or unchanged-topology hot-path finding was identified.

The stack is now reviewable by responsibility: #13019 is the dormant public API foundation, #13394 is the queue-actor host plus example and benchmark, and #13378 bridges terminal reservation state and live response progress.

Verified the deferred-abort and final-watch-close regressions, API compatibility for the prior module and overload-provider surface, policy-disabled behavior, and retry ordering. Local validation includes 997 KV-router library tests, 12 response-path lifecycle tests, LLM check, strict default KV-router clippy, formatting, and diff checks.

devin-ai-integration[bot]

This comment was marked as resolved.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread lib/kv-router/src/scheduling/mod.rs Resolved
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment thread lib/kv-router/src/scheduling/mod.rs Resolved
@ishandhanani

Copy link
Copy Markdown
Contributor Author

Superseded by the fresh Classify → Order stack in #13800, #13801, and #13802. The earlier QueueAdmission protocol and its benchmark results no longer describe the proposed design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat router Relates to routing, KV-aware routing, etc. size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant