Skip to content

feat(router): bridge queue admission lifecycle - #13378

Closed
ishandhanani wants to merge 2 commits into
idhanani/dyn-3231-queue-admission-hostfrom
idhanani/dyn-3231-live-progress
Closed

ishandhanani wants to merge 2 commits into
idhanani/dyn-3231-queue-admission-hostfrom
idhanani/dyn-3231-live-progress

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

This completes the queue-admission lifecycle by exposing terminal reservation events through Selection Service and carrying live response progress to the policy. It makes ThunderAgent’s session accounting observe dispatch, success, abort, and bounded output growth without changing the no-policy path.

How This Was Implemented

  • Add terminal reservation operations and typed failures to the Selection Service and Python binding.
  • Carry policy-owned request guards through dispatch and response handling so terminal release is complete before a retry is observable.
  • Advance optional live progress only at output-block boundaries; final completion context remains authoritative.
  • Add selection and response-path regressions for deferred cancellation, terminal results, stream failure cleanup, and migration.
Walkthrough

Mental model

The host PR makes a policy decision and owns deferred requests. This PR closes that loop once a request crosses the router boundary: Selection Service reports terminal reservation state, and the response path optionally publishes monotonic output growth.

sequenceDiagram
    participant P as Admission policy
    participant Q as Queue actor
    participant S as Selection Service
    participant R as Response path
    P->>Q: Ready or Defer
    Q->>S: select and reserve
    S-->>Q: reservation and progress updater
    Q->>R: dispatch with request guard
    R->>P: progress at output-block boundary
    R->>Q: completed or aborted
    Q->>P: terminal lifecycle event
Loading

Request lifecycle

A deferred reservation that is aborted before dispatch is removed from host-owned deferred storage, gets an Aborted policy event, and completes the original selection with a typed terminal error. On dispatch or stream failure, RequestGuard::abort() is awaited before the error reaches the retry manager; its Drop cleanup only covers consumer disconnect and cannot start a migration retry.

Progress readers and updaters are created only for policy-managed requests. The response path updates the reader monotonically at existing output-block boundaries, while terminal Completed contains the exact final input-plus-output context.

Boundaries and limitations

This PR does not add a response-body callback to policies, change retry selection semantics, or alter regular requests that do not configure queue admission. Live progress is best-effort between dispatch and the terminal event, not a substitute for terminal context.

Validation

  • cargo test -p dynamo-kv-router --features standalone-selection --lib — 997 passed.
  • cargo test -p dynamo-llm --lib kv_router::push_router::tests -- --nocapture — 12 passed.
  • cargo check -p dynamo-llm --quiet.
  • cargo clippy -p dynamo-kv-router --lib -- -D warnings.

@datadog-official

datadog-official Bot commented Aug 17, 2026 •

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

PR | frontend / Build multi-arch cpu   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Failed to load manifest for workspace member '/dynamo/examples/router/session-aware-queue-admission-policy'. No such file or directory while building libdynamo_llm.

PR | backend-status-check   View in Datadog   GitHub Actions

📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: idhanani/dyn-3231-live-progress

PR | frontend / Build multi-arch cpu
Commit: edd3df9dbde57eff131a839874aa0cbd32983892
Error (code / build):
Failed to load manifest for workspace member '/dynamo/examples/router/session-aware-queue-admission-policy'. No such file or directory while building libdynamo_llm.
CI job: https://github.com/ai-dynamo/dynamo/actions/runs/32073006400/job/95521267331

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 0.00%
• Overall Coverage: 44.85% (-5.84%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: edd3df9 | Docs | Datadog PR Page | Give us feedback!

@github-actions github-actions Bot added router Relates to routing, KV-aware routing, etc. feat labels Aug 17, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-live-progress branch 2 times, most recently from 6f7f1c8 to 928e916 Compare August 17, 2026 19:10
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3231-live-progress branch from 928e916 to edd3df9 Compare August 17, 2026 21:48
@ishandhanani ishandhanani changed the title feat(router): add live admission progress feat(router): bridge queue admission lifecycle Aug 17, 2026
@ishandhanani
ishandhanani changed the base branch from idhanani/dyn-3231-session-aware-policy-crate to idhanani/dyn-3231-queue-admission-host August 17, 2026 21:48
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@ishandhanani

Copy link
Copy Markdown
Contributor Author

Superseded by the fresh Classify → Order stack in #13800, #13801, and #13802. Closing this dependent response-path layer so review stays on the revised design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant