Repository navigation
Conversation
✅ Dynamo PR CI passed — run 43203554746 (attempt 1) on
|
dcf6af0 to
74f984a
Compare
7e6214c to
7681707
Compare
7681707 to
021c77c
Compare
kv_fetch registers with [(HintHost::worker_selection("kvcr"),
HintDefault::On)], so a kvcr role whose worker_selection entry lists no
`hints` sends its committed copies, and best-effort fetches when kvcr
commits none. Other worker-selection policies run no hints and can't list
kv_fetch; the default policy no longer sends best-effort fetches.
kv_fetch now registers in register() next to kvcr instead of in
with_defaults, so a host that links no kvcr doesn't name an unknown host.
kv_fetch::default_constructor() is gone: the default is built by
kv_fetch's provider from empty parameters. The selection-core test host
registers its stand-in fetch the same way.
Signed-off-by: Karen Chung <karenc@nvidia.com>
0c01925 to
5735dd1
Compare
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
jthomson04
left a comment
There was a problem hiding this comment.
examples/backends/vllm/deploy/kvcr/agg.yaml and agg-memory-service.yaml run DYN_ROUTER_MODE=kv with no policy file. On main they get a kv.fetch on every booking; after this PR they get none, and nothing fails. Could they select kvcr, with a release note telling KVCR users to do the same? Or should kv_fetch stay on by default for default?
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
Fixed in c796827 by setting |
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
Pull Request is not mergeable
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch Signed-off-by: Karen Chung <karenc@nvidia.com>
…e-decode-selection Resolve the conflict in lib/router-plugins/builtin/src/lib.rs with ai-dynamo#17647 and ai-dynamo#17360: keep kvcr, kv_fetch and their tests, and add kv_capacity_aware back to the crate doc, the module list, register() and the tests. No other change was needed: the policy builds against the worker-selection interface after ai-dynamo#16078 and ai-dynamo#17647, and uses no scheduler module that ai-dynamo#18582 made private. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuehua Liu <liujohn@nvidia.com>
Overview
Moves the router's own KV fetch hint onto
kv_fetch, a builtin hint policy registered for thekvcrworker-selection policy and on by default there.kvcrgets a sender for the copies it commits without configuration, and listinghintsretunes or replaces it. Other worker-selection policies, including the default, send no fetch hints unless the role listskv_fetchinhints, where main sends best-effort fetches.Release note: a deployment that relies on KV fetch hints, such as KVCR, should select
kvcr, for example withDYN_ROUTER_WORKER_SELECTION_POLICY=kvcron the Frontend. The KVCR examples underexamples/backends/vllm/deploy/kvcr/now do.Third PR in the stack: #17359 → #17647 → this PR.
Flow after this PR
flowchart LR Y["User-provided config, includes: <br/>worker_selection.prefill: kvcr<br/>"] --> R0["Router, once per role<br/>at startup: resolve_role<br/>sets RouterResponsibilities"] R0 --> R1["Router, each request:<br/>candidate workers,<br/>each with its<br/>p2p copy source"] --> P subgraph PLG["Builtin worker-selection plugin: kvcr"] P["score: default cost,<br/>crediting copyable<br/>blocks, then pick"] end P --> R2["Router: book the<br/>picked worker, pass<br/>kv_fetch the peer it is<br/>committed to copy from"] --> H subgraph HPL["Builtin hint policy plugin: kv_fetch"] H["build_hints returns a<br/>kv.fetch@1.0 hint<br/>payload from that peer"] end H --> R3["Router: put the payload<br/>in the KvHint envelope,<br/>or release the booking<br/>if kv_fetch returned<br/>none"] classDef you fill:#e8f0fe,stroke:#1a56db,color:#1e3a8a classDef plugin fill:#e8f5e9,stroke:#2e7d32,color:#14532d,stroke-width:2px classDef host fill:#fff4e5,stroke:#b45309,color:#78350f class Y you class P,H plugin class R0,R1,R2,R3 hostBlue: what you write. Green: plugin code. Orange: what the router does on its own.
Summary
kv_fetchhint policy. Added viaKvFetchinlib/router-plugins/builtin/src/kv_fetch.rs, which replaces thekv.fetchthe router built itself: it sends a committed copy, or else a best-effortkv.fetch@1.0when the fetch adds at leastmin_additional_blocks.kv_fetchis on by default underkvcr. Added viaregisterinlib/router-plugins/builtin/src/kv_fetch.rs, called next tokvcrinlib/router-plugins/builtin/src/lib.rs:kv_fetchregisterskvcras its one default host. A role under any other worker-selection policy, such as a custom one whose scorer readsKV_COPY, runs it by listing it inhints. A host that links nokvcrhas nokv_fetch.kvcr. Added viaDYN_ROUTER_WORKER_SELECTION_POLICY=kvcron the Frontend inexamples/backends/vllm/deploy/kvcr/agg.yamlandagg-memory-service.yaml, so they keep sending fetch hints.KV_FETCH_ACTION_TYPE("kv.fetch"),KV_FETCH_ACTION_VERSION("1.0") andKvSourceLocationsPayloadmove from router core'slib/kv-router/src/kv_hints.rsintokv_fetch.rs, which builds the action viaKvSourceLocationsPayload::into_action.kv_hints.rskeeps only the envelope.dynamo-llmre-exports the payload at its old path, so engines are unchanged.Key Changes
Core logic: about 130 added and 75 removed lines, excluding tests, docs, comments and blank lines. About 20 of the added lines are KVCR's
kv.fetchformat, moved fromkv_hints.rstokv_fetch.rs.Review first:
lib/router-plugins/builtin/src/kv_fetch.rs:99(KvFetch::build_hints): the committed and best-effort branches, and which failure fails a booking.lib/router-plugins/builtin/src/kv_fetch.rs:135(register):kv_fetchregisterskvcras its default host.lib/kv-router/src/plugins/resolve.rs:174(inresolve_role): bookings compute a copy source only when a hint policy reads one.Validation
On this PR's tree:
cargo fmt --check;cargo clippy -D warningsfordynamo-kv-router(also withstandalone-selection,metrics,bench),dynamo-router-policies,dynamo-llmanddynamo-mocker; and the tests ofdynamo-kv-router(1,473 withstandalone-selection,metrics),dynamo-router-policies,dynamo-llm'skv_router::(498) anddynamo-mocker(317). End to end againstdynamo.mockerworkers on CPU:--router-mode kv, no policy filekvcrwithkv_fetchlistedtests/router/test_router_e2e_with_mockers.pyandtests/router/test_policy_class.pyagainstdynamo.mockerworkers on the full stack: 51 passed, and 5 skipped by their existing CI-flakiness marker.Selection cost, from Criterion benches in the release profile on one machine, with the three builds interleaved per benchmark group. A rerun of three alternating passes showed the remaining differences are within run-to-run noise (±2%):
main@e4aeea8cf4768170772bmainselection_coreselect, reserve and free, 8 workersNot covered:
lib/kv-router/benches/selection_core.rsbuilds its core without hint policies.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit
Summary
kv_fetchas the default hint policy for roles without an explicit configuration. It sends fetch hints for committed copies and otherwise fetches on a best-effort basis when the source can provide enough additional blocks.