Skip to content

feat(router): add do_not_queue admission control - #12192

Open
nicole-lihui wants to merge 1 commit into
ai-dynamo:mainfrom
nicole-lihui:feat/10573-do-not-queue
Open

nicole-lihui wants to merge 1 commit into
ai-dynamo:mainfrom
nicole-lihui:feat/10573-do-not-queue

Conversation

@nicole-lihui

@nicole-lihui nicole-lihui commented Jul 27, 2026 •

Copy link
Copy Markdown

Summary

Implement do_not_queue for #10573 against main at f80421807. An opted-in request now enters the normal policy queue and participates in class, deadline, priority, DRR, and worker eligibility decisions. After that admission turn drains, the router retracts the request only if it still needs to wait for worker capacity. Retraction restores global and class pending counters and disarms its lifecycle lease; the rejection reports the class load remaining after retraction. Ordinary requests continue to queue.

  • Preserve nvext.do_not_queue and raw RouterRequest::New.do_not_queue, defaulting to false. Propagate the field through frontend, selection, standalone selection, and EPP. Advisory selection ignores it because advisory operations do not enter queue admission.
  • Preserve structured raw RouterResponse::DoNotQueue. HTTP paths expose the distinct router.do_not_queue reason, RateLimited/429, and safe class and pending load details. Existing queue-limit, deadline, capacity, and no-worker outcomes retain their own behavior.
  • Update the existing nvext documentation and all affected custom-policy example, builtin scorer, and benchmark constructors.

This option governs only the router pending queue. It does not control backend engine queues or guarantee zero end-to-end waiting. Existing high-level positional routing APIs retain their default behavior; the PR does not extend the C API.

Validation

  • macOS: cargo test -p dynamo-kv-router --no-default-features --features standalone-selection do_not_queue --lib (14 passed), plus all 51 scheduler queue tests without default features.
  • macOS: targeted dynamo-llm --no-default-features tests cover semantic error mapping, raw router idle/busy behavior, and real unary and streaming HTTP 429 responses. Targeted dynamo-ext-proc tests cover integrated and standalone EPP mappings. The HTTP and raw router integration tests used a localhost-capable test environment.
  • cargo check passed for dynamo-ext-proc, dynamo-custom-policy-builtin, and the two affected custom-policy examples. cargo fmt --all -- --check and git diff --check passed. Docs lint passed with 0 errors and 8 existing warnings.
  • cargo clippy passed for dynamo-kv-router, dynamo-llm, and dynamo-ext-proc with --no-default-features --all-targets -- -D warnings -A unused-variables -A clippy::derivable-impls. The two allowances cover pre-existing macOS warnings in dynamo-runtime and dynamo-mocker.
  • policy_queue Criterion benchmark, macOS dev profile, 20 samples, same benchmark source on main and this PR: normal pop_once_exact/1 869.16 ns → 879.60 ns; single rejected-tail retraction from 1024 queued requests 128.26 µs → 129.90 µs. These measurements cover queue primitives, not end-to-end router or HTTP latency; differences are small relative to run-to-run noise.

Review entry points

  1. lib/kv-router/src/scheduling/queue.rs: normal enqueue/drain, post-drain retraction, counter and lease cleanup.
  2. lib/kv-router/src/protocols.rs, lib/kv-router/src/services/selection/, and lib/llm/src/kv_router.rs: wire compatibility and admission propagation.
  3. Runtime semantic error, frontend HTTP, EPP, and regression tests.

Closes #10573

@copy-pr-bot

copy-pr-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator July 27, 2026 10:50 — with GitHub Actions Inactive
@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator July 27, 2026 10:50 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi nicole-lihui! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. labels Jul 27, 2026
@nicole-lihui nicole-lihui changed the title Feat/10573 do not queue feat: add nvext.do_not_queue Jul 27, 2026
@github-actions github-actions Bot added the feat label Jul 27, 2026
@nicole-lihui nicole-lihui changed the title feat: add nvext.do_not_queue feat(router): add nvext.do_not_queue Jul 27, 2026
@nicole-lihui nicole-lihui changed the title feat(router): add nvext.do_not_queue feat(router): add nvext.do_not_queue backpressure handling Jul 27, 2026
@nicole-lihui
nicole-lihui marked this pull request as ready for review July 28, 2026 09:33
@nicole-lihui
nicole-lihui requested review from a team as code owners July 28, 2026 09:33

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

The request-level do_not_queue flag is added to request extensions, routing hints, scheduler requests, and routing options. Busy queue-bound requests now return DoNotQueue, which propagates as backpressure and HTTP 429 responses, with dedicated metrics, tests, and documentation.

Do-not-queue admission

Layer / File(s) Summary
Request and routing contracts
lib/llm/src/protocols/common/*, lib/llm/src/preprocessor.rs, lib/kv-router/src/scheduling/types.rs
Request extensions, routing hints, scheduling requests, and routing options carry the new flag.
Scheduler admission behavior
lib/kv-router/src/scheduling/*, lib/llm/src/kv_router/push_router.rs, lib/mocker/...
Queue admission rejects busy do-not-queue requests with KvSchedulerError::DoNotQueue, while immediate admission and default queueing remain supported.
Router option propagation
lib/llm/src/kv_router.rs, lib/llm/src/kv_router/prefill_router/*, lib/llm/src/kv_router/push_router/selection.rs, lib/bindings/c/src/lib.rs
Routing APIs consolidate options and forward do_not_queue through prefill, decode, and scheduler selection.
Backpressure response mapping
deploy/inference-gateway/ext-proc/src/*, lib/kv-router/src/services/selection/error.rs, lib/llm/src/http/service/openai.rs
Do-not-queue errors are classified as backpressure and mapped to HTTP 429 responses.
Backpressure metrics and documentation
lib/llm/src/kv_router/metrics.rs, lib/llm/src/kv_router/scheduler.rs, docs/components/frontend/nvext.md
Dedicated rejection metrics and client-facing nvext.do_not_queue documentation are added.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes match the linked issue’s goals: opt-in fast-fail admission, 429 mapping, preserved defaults, metrics, and documentation.
Out of Scope Changes check ✅ Passed The modified files all support do_not_queue propagation, backpressure handling, metrics, docs, or related test coverage.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description check ✅ Passed The description clearly explains the feature, behavior, propagation, validation, review entry points, and linked issue. It uses different headings from the template, but it provides the required infor…
Title check ✅ Passed The title clearly summarizes the primary change: adding do_not_queue admission control to the router.
✨ Finishing Touches
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch feat/10573-do-not-queue

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/kv-router/src/services/selection/core/mod.rs`:
- Line 732: Update schedule_selection and SelectionOperation to carry the
request’s do_not_queue value from both selection request types, and assign that
propagated value instead of hard-coding false. Add a saturated-worker test
verifying do_not_queue=true is rejected with backpressure while the default
false behavior still queues.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4f4d2d79-69fd-44ed-ad6e-50b74dff8de9

📥 Commits

Reviewing files that changed from the base of the PR and between 769ee07 and 369ea61.

📒 Files selected for processing (23)
  • deploy/inference-gateway/ext-proc/src/epp.rs
  • deploy/inference-gateway/ext-proc/src/picker.rs
  • deploy/inference-gateway/ext-proc/src/server.rs
  • docs/components/frontend/nvext.md
  • lib/bindings/c/src/lib.rs
  • lib/kv-router/src/scheduling/local.rs
  • lib/kv-router/src/scheduling/policy.rs
  • lib/kv-router/src/scheduling/queue.rs
  • lib/kv-router/src/scheduling/selector.rs
  • lib/kv-router/src/scheduling/types.rs
  • lib/kv-router/src/services/selection/core/mod.rs
  • lib/kv-router/src/services/selection/error.rs
  • lib/llm/src/http/service/openai.rs
  • lib/llm/src/kv_router.rs
  • lib/llm/src/kv_router/metrics.rs
  • lib/llm/src/kv_router/prefill_router/query.rs
  • lib/llm/src/kv_router/push_router.rs
  • lib/llm/src/kv_router/push_router/selection.rs
  • lib/llm/src/kv_router/scheduler.rs
  • lib/llm/src/preprocessor.rs
  • lib/llm/src/protocols/common/extensions.rs
  • lib/llm/src/protocols/common/preprocessor.rs
  • lib/mocker/src/replay/offline/extensions/kv_router/mod.rs

@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator July 29, 2026 08:46 — with GitHub Actions Inactive
@datadog-official

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

Pre Merge | pre-merge-status-check   View in Datadog   GitHub Actions

Pre Merge | rust-clippy (.)   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 710465e | Docs | Datadog PR Page | Give us feedback!

@github-actions

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions Bot added the Stale label Aug 31, 2026
@nicole-lihui
nicole-lihui force-pushed the feat/10573-do-not-queue branch from 710465e to ba0be65 Compare September 1, 2026 09:45
@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator September 1, 2026 09:45 — with GitHub Actions Inactive
@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator September 1, 2026 09:56 — with GitHub Actions Inactive
@nicole-lihui
nicole-lihui force-pushed the feat/10573-do-not-queue branch from 8d3f7c6 to a51407d Compare September 15, 2026 07:05
@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator September 15, 2026 07:05 — with GitHub Actions Inactive
Signed-off-by: nicole-lihui <nicole.li@daocloud.io>
@nicole-lihui
nicole-lihui force-pushed the feat/10573-do-not-queue branch from a51407d to 16b7939 Compare September 23, 2026 07:02
@nicole-lihui
nicole-lihui temporarily deployed to external_collaborator September 23, 2026 07:02 — with GitHub Actions Inactive
@nicole-lihui nicole-lihui changed the title feat(router): add nvext.do_not_queue backpressure handling feat(router): add do_not_queue admission control Sep 23, 2026
@devin-ai-integration

devin-ai-integration Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

External PR triage — 2026-11-03

Blocker: failing checks caused by this PR, plus new merge conflicts with main. Both fixes belong to the author.

Next action, owned by @nicole-lihui:

  1. Rebase onto current main and resolve the three kv-router conflicts.
  2. Add do_not_queue to every SelectAndReserveRequest initializer, including lib/router-plugins/builtin/tests/.
  3. Get cargo test and cargo clippy --all-targets passing for the workspace, then push.

Then: @PeaBrane (@ai-dynamo/dynamo-router-codeowners; team review already requested) re-checks the audit on the new head, posts /ok to test <new head SHA>, and reviews.

Supply-chain audit (head 16b7939722, unchanged): safe to approve. 37 Rust and doc files across router, llm, EPP ext-proc and examples. There are no workflow/CI edits, no Cargo.toml/Cargo.lock changes, no download URLs (only a localhost doc URL), no scripts, no secret access and no binaries. Re-audit the rebased diff.


Automated external-PR triage. This comment is edited in place on each run. No CI authorization or approval was issued. Written by Devin

@github-actions github-actions Bot removed the Stale label Nov 11, 2026

This branch was previously deployed

1 inactive deployment
external_collaborator — 16b79397 Deployed Sep 23, 2026 by nicole-lihui via ok-to-test #22276
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CONTRIBUTION]: RouterBackpressureReason::DoNotQueue

1 participant