Skip to content

feat(router): move the KV fetch hint onto a kv_fetch hint policy - #17360

Merged
karen-sy merged 26 commits into
karenc/router-kvcr-fetch-scorerfrom
karenc/router-hint-policy-kv-fetch
Nov 12, 2026
Merged

karen-sy merged 26 commits into
karenc/router-kvcr-fetch-scorerfrom
karenc/router-hint-policy-kv-fetch

Conversation

@karen-sy

@karen-sy karen-sy commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Moves the router's own KV fetch hint onto kv_fetch, a builtin hint policy registered for the kvcr worker-selection policy and on by default there. kvcr gets a sender for the copies it commits without configuration, and listing hints retunes or replaces it. Other worker-selection policies, including the default, send no fetch hints unless the role lists kv_fetch in hints, where main sends best-effort fetches.

Release note: a deployment that relies on KV fetch hints, such as KVCR, should select kvcr, for example with DYN_ROUTER_WORKER_SELECTION_POLICY=kvcr on the Frontend. The KVCR examples under examples/backends/vllm/deploy/kvcr/ now do.

Third PR in the stack: #17359 → #17647 → this PR.

Flow after this PR

flowchart LR
    Y["User-provided config, includes: <br/>worker_selection.prefill: kvcr<br/>"] --> R0["Router, once per role<br/>at startup: resolve_role<br/>sets RouterResponsibilities"]
    R0 --> R1["Router, each request:<br/>candidate workers,<br/>each with its<br/>p2p copy source"] --> P
    subgraph PLG["Builtin worker-selection plugin: kvcr"]
        P["score: default cost,<br/>crediting copyable<br/>blocks, then pick"]
    end
    P --> R2["Router: book the<br/>picked worker, pass<br/>kv_fetch the peer it is<br/>committed to copy from"] --> H
    subgraph HPL["Builtin hint policy plugin: kv_fetch"]
        H["build_hints returns a<br/>kv.fetch@1.0 hint<br/>payload from that peer"]
    end
    H --> R3["Router: put the payload<br/>in the KvHint envelope,<br/>or release the booking<br/>if kv_fetch returned<br/>none"]

    classDef you fill:#e8f0fe,stroke:#1a56db,color:#1e3a8a
    classDef plugin fill:#e8f5e9,stroke:#2e7d32,color:#14532d,stroke-width:2px
    classDef host fill:#fff4e5,stroke:#b45309,color:#78350f
    class Y you
    class P,H plugin
    class R0,R1,R2,R3 host
Loading

Blue: what you write. Green: plugin code. Orange: what the router does on its own.

Summary

  • KV fetch hint moves out of router core into a kv_fetch hint policy. Added via KvFetch in lib/router-plugins/builtin/src/kv_fetch.rs, which replaces the kv.fetch the router built itself: it sends a committed copy, or else a best-effort kv.fetch@1.0 when the fetch adds at least min_additional_blocks.
  • kv_fetch is on by default under kvcr. Added via register in lib/router-plugins/builtin/src/kv_fetch.rs, called next to kvcr in lib/router-plugins/builtin/src/lib.rs: kv_fetch registers kvcr as its one default host. A role under any other worker-selection policy, such as a custom one whose scorer reads KV_COPY, runs it by listing it in hints. A host that links no kvcr has no kv_fetch.
  • The KVCR examples select kvcr. Added via DYN_ROUTER_WORKER_SELECTION_POLICY=kvcr on the Frontend in examples/backends/vllm/deploy/kvcr/agg.yaml and agg-memory-service.yaml, so they keep sending fetch hints.
  • KVCR's fetch format moves into the plugin. KV_FETCH_ACTION_TYPE ("kv.fetch"), KV_FETCH_ACTION_VERSION ("1.0") and KvSourceLocationsPayload move from router core's lib/kv-router/src/kv_hints.rs into kv_fetch.rs, which builds the action via KvSourceLocationsPayload::into_action. kv_hints.rs keeps only the envelope. dynamo-llm re-exports the payload at its old path, so engines are unchanged.
## kvcr picks the worker with copy credit; kv_fetch, on by default under kvcr, sends the copies.
worker_selection:
  prefill: kvcr
  decode: dynamo-default-cost-fn          # default routing, no hints
## Listing hints replaces the defaults, so list kv_fetch next to any opt-in hint.
worker_selection:
  prefill:
    type: kvcr
    hints:
      - type: kv_fetch
      - type: <future_opt_in_hint>        # placeholder: any hint registered as off by default
## No hints. kvcr then needs copy credit off, so it commits no copies.
worker_selection:
  prefill:
    type: kvcr
    parameters:
      shared_cache_multiplier: 0.0
    hints: []

Key Changes

Core logic: about 130 added and 75 removed lines, excluding tests, docs, comments and blank lines. About 20 of the added lines are KVCR's kv.fetch format, moved from kv_hints.rs to kv_fetch.rs.

Review first:

  • lib/router-plugins/builtin/src/kv_fetch.rs:99 (KvFetch::build_hints): the committed and best-effort branches, and which failure fails a booking.
  • lib/router-plugins/builtin/src/kv_fetch.rs:135 (register): kv_fetch registers kvcr as its default host.
  • lib/kv-router/src/plugins/resolve.rs:174 (in resolve_role): bookings compute a copy source only when a hint policy reads one.

Validation

On this PR's tree: cargo fmt --check; cargo clippy -D warnings for dynamo-kv-router (also with standalone-selection,metrics,bench), dynamo-router-policies, dynamo-llm and dynamo-mocker; and the tests of dynamo-kv-router (1,473 with standalone-selection,metrics), dynamo-router-policies, dynamo-llm's kv_router:: (498) and dynamo-mocker (317). End to end against dynamo.mocker workers on CPU:

Case Result
--router-mode kv, no policy file Serves; the default policy runs no hints
kvcr with kv_fetch listed Serves

tests/router/test_router_e2e_with_mockers.py and tests/router/test_policy_class.py against dynamo.mocker workers on the full stack: 51 passed, and 5 skipped by their existing CI-flakiness marker.

Selection cost, from Criterion benches in the release profile on one machine, with the three builds interleaved per benchmark group. A rerun of three alternating passes showed the remaining differences are within run-to-run noise (±2%):

Benchmark main @ e4aeea8cf4 previous stack head @ 768170772b this stack vs main
selection_core select, reserve and free, 8 workers 17.74 µs 18.04 µs 17.36 µs −2.1%
default selection, 2 workers 101 ns 137 ns 99 ns −1.7%
default selection, 32 workers 688 ns 716 ns 696 ns +1.1%
default selection, 1,024 workers 21.53 µs 21.30 µs 21.26 µs −1.3%
default selection, 10,000 workers 331.24 µs 339.40 µs 315.69 µs −4.7%
default policy wrapper, 2 workers 97 ns 135 ns 98 ns +0.7%

Not covered:

  • GPU or engine runs.
  • Benchmarks of the hint stage on every booking: lib/kv-router/benches/selection_core.rs builds its core without hint policies.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

Summary

  • New Features
    • Added kv_fetch as the default hint policy for roles without an explicit configuration. It sends fetch hints for committed copies and otherwise fetches on a best-effort basis when the source can provide enough additional blocks.
    • Added a configurable threshold for best-effort fetching; it defaults to one additional block.
  • Behavior Changes
    • Fetch hints are now generated by hint policies rather than added automatically by the router. Explicit role policies replace the default.
  • Documentation
    • Updated guidance on configuring, customizing, or disabling hints by role and on handling committed copies.

@devin-ai-integration

devin-ai-integration Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 43203554746 (attempt 1) on 0eed404a47

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 43203554746. Updated on every full-CI run of this PR.

@karen-sy
karen-sy force-pushed the karenc/router-hint-policy-kv-fetch branch from dcf6af0 to 74f984a Compare October 30, 2026 01:58
@github-actions github-actions Bot added documentation Improvements or additions to documentation router Relates to routing, KV-aware routing, etc. labels Oct 30, 2026
@karen-sy
karen-sy removed this pull request from stack #17361 October 30, 2026 02:04
@karen-sy
karen-sy changed the base branch from karenc/router-hint-policy to karenc/router-kvcr-fetch-scorer October 30, 2026 02:04
@karen-sy
karen-sy added this pull request to stack #17648 October 30, 2026 02:04
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

@karen-sy
karen-sy force-pushed the karenc/router-hint-policy-kv-fetch branch from 7e6214c to 7681707 Compare November 4, 2026 22:25
@karen-sy
karen-sy force-pushed the karenc/router-hint-policy-kv-fetch branch from 7681707 to 021c77c Compare November 6, 2026 05:10
@github-actions github-actions Bot added backend::vllm Relates to the vllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Nov 6, 2026
@karen-sy karen-sy changed the title feat(router): move the KV fetch hint onto a kv_fetch hint policy feat(router): KVCR fetch refactored into a built-in hint policy Nov 6, 2026
@karen-sy
karen-sy marked this pull request as ready for review November 6, 2026 05:27
@karen-sy
karen-sy requested review from a team as code owners November 6, 2026 05:27
kv_fetch registers with [(HintHost::worker_selection("kvcr"),
HintDefault::On)], so a kvcr role whose worker_selection entry lists no
`hints` sends its committed copies, and best-effort fetches when kvcr
commits none. Other worker-selection policies run no hints and can't list
kv_fetch; the default policy no longer sends best-effort fetches.

kv_fetch now registers in register() next to kvcr instead of in
with_defaults, so a host that links no kvcr doesn't name an unknown host.
kv_fetch::default_constructor() is gone: the default is built by
kv_fetch's provider from empty parameters. The selection-core test host
registers its stand-in fetch the same way.

Signed-off-by: Karen Chung <karenc@nvidia.com>
@karen-sy
karen-sy force-pushed the karenc/router-hint-policy-kv-fetch branch from 0c01925 to 5735dd1 Compare November 10, 2026 01:06
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
@karen-sy
karen-sy requested a review from jthomson04 November 10, 2026 17:40
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

examples/backends/vllm/deploy/kvcr/agg.yaml and agg-memory-service.yaml run DYN_ROUTER_MODE=kv with no policy file. On main they get a kv.fetch on every booking; after this PR they get none, and nothing fails. Could they select kvcr, with a release note telling KVCR users to do the same? Or should kv_fetch stay on by default for default?

…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
@karen-sy

Copy link
Copy Markdown
Contributor Author

examples/backends/vllm/deploy/kvcr/agg.yaml and agg-memory-service.yaml run DYN_ROUTER_MODE=kv with no policy file. On main they get a kv.fetch on every booking; after this PR they get none, and nothing fails. Could they select kvcr, with a release note telling KVCR users to do the same? Or should kv_fetch stay on by default for default?

Fixed in c796827 by setting DYN_ROUTER_WORKER_SELECTION_POLICY=kvcr on the Frontend in both examples. kvcr's multiplier defaults to 0 there, so they keep main's best-effort kv.fetch on every booking. The PR description now has a release note telling KVCR users to select kvcr.

…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
Signed-off-by: Karen Chung <karenc@nvidia.com>

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work

@karen-sy
karen-sy enabled stacked auto-merge (squash) November 10, 2026 23:42
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
stacked auto-merge was automatically disabled November 11, 2026 02:07

Pull Request is not mergeable

…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
@karen-sy
karen-sy enabled stacked auto-merge (squash) November 11, 2026 05:42
…t-policy-kv-fetch

Signed-off-by: Karen Chung <karenc@nvidia.com>
@karen-sy
karen-sy merged commit 087f8d0 into main Nov 12, 2026
128 checks passed
@karen-sy
karen-sy deleted the karenc/router-hint-policy-kv-fetch branch November 12, 2026 11:42
john-liu-dev added a commit to john-liu-dev/dynamo that referenced this pull request Nov 16, 2026
…e-decode-selection

Resolve the conflict in lib/router-plugins/builtin/src/lib.rs with
ai-dynamo#17647 and ai-dynamo#17360: keep kvcr, kv_fetch and their tests, and add
kv_capacity_aware back to the crate doc, the module list, register()
and the tests. No other change was needed: the policy builds against
the worker-selection interface after ai-dynamo#16078 and ai-dynamo#17647, and uses no
scheduler module that ai-dynamo#18582 made private.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuehua Liu <liujohn@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants