Skip to content

refactor(router): define typed KV hint contract - #13134

Merged
ishandhanani merged 14 commits into
mainfrom
idhanani/dyn-3877-define-hint-contract-between-dynamo-engine
Sep 8, 2026
Merged

ishandhanani merged 14 commits into
mainfrom
idhanani/dyn-3877-define-hint-contract-between-dynamo-engine

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

This change separates caller-controlled RoutingHints from one versioned KvHint message attached as kv_hint between Dynamo and inference engines, aligned with vLLM RFC #53421. The open message lets engines consume independently versioned KV actions without another top-level schema change or a name collision with placement hints.

CLOSES: DYN-3877

How This Was Implemented

  • Added KvHint { protocol_version, message_id, actions } and generic KvHintAction { action_id, action_type, action_version, payload } transport types.
  • Kept action payloads opaque while retaining a typed constructor for kv.fetch@1.0.
  • Kept capability advertisement separate from the request message and preserved the existing router_hint* discovery wire names until they can be renamed with KVCC.
  • Updated post-selection router composition to attach one kv_hint message only for supported targets.
  • Kept the current vLLM bridge passing the complete message through kv_transfer_params.kv_hint.
Walkthrough

Mental model

The engine advertises which actions it supports. Dynamo uses that metadata after worker selection, builds one versioned KV hint message containing independent actions, and passes the complete message to the selected engine.

flowchart LR
    A["Engine advertises KV support"] --> B["Frontend records capability metadata"]
    B --> C["Router selects a worker"]
    C --> D["Dynamo builds KV actions"]
    D --> E["PreprocessedRequest.kv_hint"]
    E --> F["vLLM sets kv_transfer_params.kv_hint"]
    F --> G["KVCC receives the complete message"]
Loading

State and ordering

The message uses protocol version 0.1. Every action carries action_id, action_type, action_version, and an opaque JSON-object payload, matching the vLLM RFC and its Python/Rust transport types.

KvHintAction is deliberately open rather than a closed enum, so a future action type or version can cross the transport without changing the top-level message. Dynamo still constructs the current kv.fetch@1.0 action through a typed helper and shared constants. Its payload contains one source control endpoint and the root-aligned source block hashes.

The router keeps the existing source-selection behavior. It chooses a compatible source with a longer cached prefix after selecting the target worker.

Request lifecycle

Model registration publishes the existing router_hint, router_hint_worker_type, and router_hint_source_control_endpoints metadata. Capability negotiation is not part of KvHint.

After worker selection, kv_hint_for_selection() builds the supported action list. RoutingHost attaches the message to PreprocessedRequest.kv_hint, and the vLLM adapter preserves the complete message while installing the connector's request-time transfer parameters.

The vLLM adapter does not unwrap or interpret individual actions. It copies the message into the existing private kv_transfer_params.kv_hint field so KVCC can consume the same versioned contract.

Boundaries and limitations

This change does not alter worker selection policy or the caller-controlled RoutingHints contract. It does not add dereference or other KV actions, and there is no response or acknowledgment path; existing KV events provide later visibility into cache behavior.

The first-class vLLM request path is being introduced separately in vLLM PR #53423. Until that lands and Dynamo updates its vLLM dependency, this PR intentionally retains the existing private bridge instead of depending on unmerged vLLM code.

The existing router_hint* discovery names are intentionally transitional. TODOs keep their later rename coupled to the matching KVCC change; this PR does not add aliases or dual publication.

Validation

  • cargo test -p dynamo-kv-router kv_hints --lib (5 passed)
  • cargo test -p dynamo-llm kv_hint --lib (2 passed)
  • cargo test -p dynamo-llm protocols::common::preprocessor::tests --lib (7 passed)
  • cargo check -p dynamo-backend-common
  • cargo fmt --all -- --check
  • ruff check on the changed Python files
  • python -m py_compile on the changed Python files
  • Not run: vLLM pytest requires Dynamo and vLLM packages, which are not installed in this worktree's .venv.

Summary by CodeRabbit

  • New Features

    • Added typed KV-cache migration hints for routing requests.
    • Workers can now advertise migration capabilities, worker types, and source-control endpoints.
    • Routing can select suitable workers and attach migration transfer plans automatically.
    • Added support for migration hints across supported vLLM workflows.
  • Bug Fixes

    • Preserved existing transfer settings while filtering untrusted request fields.
  • Tests

    • Expanded coverage for endpoint selection, validation, IPv6 formatting, and migration hint serialization.

@github-actions github-actions Bot added refactor backend::vllm Relates to the vllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` router Relates to routing, KV-aware routing, etc. labels Aug 12, 2026
@ishandhanani
ishandhanani marked this pull request as ready for review August 12, 2026 22:44
@ishandhanani
ishandhanani requested review from a team as code owners August 12, 2026 22:44
devin-ai-integration[bot]

This comment was marked as resolved.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4ea8096f-571d-43aa-99f2-f43716a05b6c

📥 Commits

Reviewing files that changed from the base of the PR and between 427046c and d96ece5.

📒 Files selected for processing (18)
  • components/src/dynamo/common/constants.py
  • components/src/dynamo/vllm/handlers.py
  • components/src/dynamo/vllm/kv_hints.py
  • components/src/dynamo/vllm/main.py
  • components/src/dynamo/vllm/tests/test_kv_hints.py
  • components/src/dynamo/vllm/tests/test_vllm_legacy_lora.py
  • components/src/dynamo/vllm/tests/test_vllm_unit.py
  • lib/backend-common/src/engine.rs
  • lib/backend-common/src/lib.rs
  • lib/kv-router/src/kv_hints.rs
  • lib/kv-router/src/lib.rs
  • lib/kv-router/src/protocols.rs
  • lib/kv-router/src/router_hint.rs
  • lib/llm/src/kv_router.rs
  • lib/llm/src/kv_router/push_router.rs
  • lib/llm/src/kv_router/push_router/selection.rs
  • lib/llm/src/local_model/runtime_config.rs
  • lib/llm/src/protocols/common/preprocessor.rs

Walkthrough

The PR replaces router-hint metadata and payloads with typed KV MIGRATE hints. Routing selects migration sources, preprocessing stores typed hints, vLLM publishes migration metadata, and request handling converts transfer plans for the connector.

Changes

KV MIGRATE Hint Migration

Layer / File(s) Summary
Typed hint contracts and request representation
lib/kv-router/src/kv_hints.rs, lib/kv-router/src/protocols.rs, lib/llm/src/protocols/common/preprocessor.rs, lib/backend-common/src/..., lib/kv-router/src/router_hint.rs
Adds typed KvHints, MigrateHint, and KvTransferPlan structures. Renames migration metadata APIs. Stores hints directly on PreprocessedRequest and removes legacy router-hint conversion.
Runtime metadata publication and capability setup
components/src/dynamo/common/constants.py, components/src/dynamo/vllm/kv_hints.py, components/src/dynamo/vllm/main.py, components/src/dynamo/vllm/handlers.py, lib/llm/src/local_model/runtime_config.rs, components/src/dynamo/vllm/tests/*
Publishes KV MIGRATE capability, worker type, and source-control endpoints. Updates capability filtering, endpoint validation, DP handling, and related tests.
Migration hint selection and routing propagation
lib/llm/src/kv_router.rs, lib/llm/src/kv_router/push_router.rs, lib/llm/src/kv_router/push_router/selection.rs
Selects eligible migration workers and source endpoints. Builds nested transfer plans and propagates kv_hints through routed outcomes and dispatch.
vLLM request translation
components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/tests/test_vllm_unit.py
Translates kv_hints.migrate.transfer_plan into the connector router_hint field while preserving existing KV-transfer parameters.

Estimated code review effort: 4 (Complex) | ~45 minutes

Mergeability Score: ⚪ Minimal · up to d96ec

This change formalizes the router-to-engine KV hint contract while preserving MIGRATE behavior through the backend boundary; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 56.25% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: defining a typed KV hint contract.
Description check ✅ Passed The description is detailed, on-topic, and covers the implementation, limitations, related issue, and validation. It does not include an explicit “Where should the reviewer start?” section, but the mi…
Full details: Description check

Explanation

The description is detailed, on-topic, and covers the implementation, limitations, related issue, and validation. It does not include an explicit “Where should the reviewer start?” section, but the missing section is non-critical because the walkthrough identifies the main flow and relevant changes.

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@datadog-official

datadog-official Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 84.21%
• Overall Coverage: 47.16% (-1.68%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e7cfafd | Docs | Datadog PR Page | Give us feedback!

@ishandhanani
ishandhanani requested a review from a team as a code owner August 12, 2026 23:16
devin-ai-integration[bot]

This comment was marked as resolved.

@karen-sy

Copy link
Copy Markdown
Contributor

Linking KvHint-driven kv management RFCs vllm-project/vllm#51428 (vLLM) and sgl-project/sglang#27574 (SGLang) for cross-reference purposes

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ AI-assisted review. The current v1 execution path looks sound, and I found no confirmed runtime blocker. The inline notes are contract-design and regression-oracle concerns; please put human eyes on them before merging.

Comment thread lib/kv-router/src/kv_hints.rs Resolved
Comment thread lib/kv-router/src/kv_hints.rs Resolved
Comment thread lib/llm/src/kv_router/push_router.rs Resolved
devin-ai-integration[bot]

This comment was marked as resolved.

@ishandhanani
ishandhanani requested review from a team as code owners August 13, 2026 23:10
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 13, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-3877-define-hint-contract-between-dynamo-engine branch from 2cca71f to a4bdb9b Compare August 26, 2026 19:47
ishandhanani and others added 7 commits August 26, 2026 20:34
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ine-hint-contract-between-dynamo-engine

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@dagil-nvidia dagil-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving (docs-codeowners slice, with a full pass behind it): all gates green (CODEOWNERS drift 0 at 613 rules, strict coverage, 93 matcher tests, 26 skills, docs linter proven with a known-positive probe) and CI at head is 84 pass / 0 fail. The two deleted doc rows are genuinely obsolete, not lost truth: this PR removes the whole kv_hints() / evict_session() chain end to end, a repo-wide grep finds zero leftovers of any removed identifier, the surviving accessor table matches exactly what still exists on SessionContext, and the capability folds into session_final, which stays. No other doc references the removed API - including the 14 router docs relocated by main's restructure - and the one remaining mention is the v1.3.0 release-notes entry, which is a historical record. The rename was verified mechanical across ~40 files, wire compatibility for the discovery keys is intentionally preserved, and all three Cargo.locks carry the serde_json promotion.

Note: the head advanced during review; the delta (the publish_source_endpoints kwarg rename at all three call sites, plus the merge from main) was re-verified against the current head.

Three non-blocking notes:

  1. components/src/dynamo/vllm/tests/test_vllm_unit.py:1179 - the rewritten test dropped the old negative guard: nothing now pins that a caller-supplied extra_args["kv_transfer_params"] (including a spoofed kv_hint) never reaches SamplingParams. Correct today by construction, but the routing-host strip that used to enforce it is gone too, so one test feeding a spoofed hint and asserting it does not land is the remaining fence worth adding.
  2. lib/llm/src/kv_router.rs:1090 - KvHint::new(message_id.unwrap_or_default(), ...) emits an empty message id when context is None; unreachable on attach paths today, but a UUID fallback or debug assert keeps the message-identity promise honest for future callers.
  3. Stale "TRANSFER" vocabulary survives in a handful of doc comments and docstrings, and in one operator-visible warning (kv_router.rs:1395) where "TRANSFER" maps to no config key or doc term an operator can look up; "P2P KV transfer hint" (as the Python error strings already say) reads right.

Pre-existing, already noted in-thread: the disagg decode path drops decode-targeted hints (same shape as the old design), and the 13 build-time-generated API nav entries.

Automated review pass; severity escalation stays with the human reviewers.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ine-hint-contract-between-dynamo-engine

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

# Conflicts:
#	lib/llm/src/protocols/common/preprocessor.rs
…ine-hint-contract-between-dynamo-engine

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

# Conflicts:
#	lib/llm/src/kv_router/routing_host/kv_selection.rs
…ine-hint-contract-between-dynamo-engine

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

# Conflicts:
#	lib/llm/src/kv_router.rs
#	lib/llm/src/kv_router/routing_host/kv_selection.rs
@oandreeva-nv

Copy link
Copy Markdown
Contributor

should be merged with : ai-dynamo/kvcr#18

…ine-hint-contract-between-dynamo-engine

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

@oandreeva-nv oandreeva-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm for kvcr compatibility, please fix ci

@ishandhanani
ishandhanani enabled auto-merge (squash) September 8, 2026 17:00
…ine-hint-contract-between-dynamo-engine

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani merged commit 8b030e0 into main Sep 8, 2026
124 checks passed
@ishandhanani
ishandhanani deleted the idhanani/dyn-3877-define-hint-contract-between-dynamo-engine branch September 8, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` refactor router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants