Repository navigation
feat(router): add do_not_queue admission control - #12192
nicole-lihui wants to merge 1 commit into
Conversation
|
👋 Hi nicole-lihui! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
nvext.do_not_queuenvext.do_not_queue
nvext.do_not_queuenvext.do_not_queue backpressure handling
WalkthroughChangesThe request-level Do-not-queue admission
Estimated code review effort: 4 (Complex) | ~45 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches⚔️ Resolve merge conflicts
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/kv-router/src/services/selection/core/mod.rs`:
- Line 732: Update schedule_selection and SelectionOperation to carry the
request’s do_not_queue value from both selection request types, and assign that
propagated value instead of hard-coding false. Add a saturated-worker test
verifying do_not_queue=true is rejected with backpressure while the default
false behavior still queues.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 4f4d2d79-69fd-44ed-ad6e-50b74dff8de9
📒 Files selected for processing (23)
deploy/inference-gateway/ext-proc/src/epp.rsdeploy/inference-gateway/ext-proc/src/picker.rsdeploy/inference-gateway/ext-proc/src/server.rsdocs/components/frontend/nvext.mdlib/bindings/c/src/lib.rslib/kv-router/src/scheduling/local.rslib/kv-router/src/scheduling/policy.rslib/kv-router/src/scheduling/queue.rslib/kv-router/src/scheduling/selector.rslib/kv-router/src/scheduling/types.rslib/kv-router/src/services/selection/core/mod.rslib/kv-router/src/services/selection/error.rslib/llm/src/http/service/openai.rslib/llm/src/kv_router.rslib/llm/src/kv_router/metrics.rslib/llm/src/kv_router/prefill_router/query.rslib/llm/src/kv_router/push_router.rslib/llm/src/kv_router/push_router/selection.rslib/llm/src/kv_router/scheduler.rslib/llm/src/preprocessor.rslib/llm/src/protocols/common/extensions.rslib/llm/src/protocols/common/preprocessor.rslib/mocker/src/replay/offline/extensions/kv_router/mod.rs
|
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
710465e to
ba0be65
Compare
8d3f7c6 to
a51407d
Compare
Signed-off-by: nicole-lihui <nicole.li@daocloud.io>
a51407d to
16b7939
Compare
nvext.do_not_queue backpressure handling
External PR triage — 2026-11-03Blocker: failing checks caused by this PR, plus new merge conflicts with
Next action, owned by @nicole-lihui:
Then: @PeaBrane (@ai-dynamo/dynamo-router-codeowners; team review already requested) re-checks the audit on the new head, posts Supply-chain audit (head Automated external-PR triage. This comment is edited in place on each run. No CI authorization or approval was issued. Written by Devin |
Summary
Implement
do_not_queuefor #10573 against main atf80421807. An opted-in request now enters the normal policy queue and participates in class, deadline, priority, DRR, and worker eligibility decisions. After that admission turn drains, the router retracts the request only if it still needs to wait for worker capacity. Retraction restores global and class pending counters and disarms its lifecycle lease; the rejection reports the class load remaining after retraction. Ordinary requests continue to queue.nvext.do_not_queueand rawRouterRequest::New.do_not_queue, defaulting tofalse. Propagate the field through frontend, selection, standalone selection, and EPP. Advisory selection ignores it because advisory operations do not enter queue admission.RouterResponse::DoNotQueue. HTTP paths expose the distinctrouter.do_not_queuereason,RateLimited/429, and safe class and pending load details. Existing queue-limit, deadline, capacity, and no-worker outcomes retain their own behavior.This option governs only the router pending queue. It does not control backend engine queues or guarantee zero end-to-end waiting. Existing high-level positional routing APIs retain their default behavior; the PR does not extend the C API.
Validation
cargo test -p dynamo-kv-router --no-default-features --features standalone-selection do_not_queue --lib(14 passed), plus all 51 scheduler queue tests without default features.dynamo-llm --no-default-featurestests cover semantic error mapping, raw router idle/busy behavior, and real unary and streaming HTTP 429 responses. Targeteddynamo-ext-proctests cover integrated and standalone EPP mappings. The HTTP and raw router integration tests used a localhost-capable test environment.cargo checkpassed fordynamo-ext-proc,dynamo-custom-policy-builtin, and the two affected custom-policy examples.cargo fmt --all -- --checkandgit diff --checkpassed. Docs lint passed with 0 errors and 8 existing warnings.cargo clippypassed fordynamo-kv-router,dynamo-llm, anddynamo-ext-procwith--no-default-features --all-targets -- -D warnings -A unused-variables -A clippy::derivable-impls. The two allowances cover pre-existing macOS warnings indynamo-runtimeanddynamo-mocker.policy_queueCriterion benchmark, macOS dev profile, 20 samples, same benchmark source on main and this PR: normalpop_once_exact/1869.16 ns → 879.60 ns; single rejected-tail retraction from 1024 queued requests 128.26 µs → 129.90 µs. These measurements cover queue primitives, not end-to-end router or HTTP latency; differences are small relative to run-to-run noise.Review entry points
lib/kv-router/src/scheduling/queue.rs: normal enqueue/drain, post-drain retraction, counter and lease cleanup.lib/kv-router/src/protocols.rs,lib/kv-router/src/services/selection/, andlib/llm/src/kv_router.rs: wire compatibility and admission propagation.Closes #10573