Repository navigation
refactor(router): define typed KV hint contract - #13134
ishandhanani merged 14 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (18)
WalkthroughThe PR replaces router-hint metadata and payloads with typed KV MIGRATE hints. Routing selects migration sources, preprocessing stores typed hints, vLLM publishes migration metadata, and request handling converts transfer plans for the connector. ChangesKV MIGRATE Hint Migration
Estimated code review effort: 4 (Complex) | ~45 minutes Mergeability Score: ⚪ Minimal · up to This change formalizes the router-to-engine KV hint contract while preserving MIGRATE behavior through the backend boundary; no actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description is detailed, on-topic, and covers the implementation, limitations, related issue, and validation. It does not include an explicit “Where should the reviewer start?” section, but the missing section is non-critical because the walkthrough identifies the main flow and relevant changes.
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Comment |
|
🎯 Code Coverage (details) 🔗 Commit SHA: e7cfafd | Docs | Datadog PR Page | Give us feedback! |
|
Linking KvHint-driven kv management RFCs vllm-project/vllm#51428 (vLLM) and sgl-project/sglang#27574 (SGLang) for cross-reference purposes |
PeaBrane
left a comment
There was a problem hiding this comment.
✅ AI-assisted review. The current v1 execution path looks sound, and I found no confirmed runtime blocker. The inline notes are contract-design and regression-oracle concerns; please put human eyes on them before merging.
294f1ac to
2cca71f
Compare
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
2cca71f to
a4bdb9b
Compare
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ine-hint-contract-between-dynamo-engine Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
dagil-nvidia
left a comment
There was a problem hiding this comment.
Approving (docs-codeowners slice, with a full pass behind it): all gates green (CODEOWNERS drift 0 at 613 rules, strict coverage, 93 matcher tests, 26 skills, docs linter proven with a known-positive probe) and CI at head is 84 pass / 0 fail. The two deleted doc rows are genuinely obsolete, not lost truth: this PR removes the whole kv_hints() / evict_session() chain end to end, a repo-wide grep finds zero leftovers of any removed identifier, the surviving accessor table matches exactly what still exists on SessionContext, and the capability folds into session_final, which stays. No other doc references the removed API - including the 14 router docs relocated by main's restructure - and the one remaining mention is the v1.3.0 release-notes entry, which is a historical record. The rename was verified mechanical across ~40 files, wire compatibility for the discovery keys is intentionally preserved, and all three Cargo.locks carry the serde_json promotion.
Note: the head advanced during review; the delta (the publish_source_endpoints kwarg rename at all three call sites, plus the merge from main) was re-verified against the current head.
Three non-blocking notes:
components/src/dynamo/vllm/tests/test_vllm_unit.py:1179- the rewritten test dropped the old negative guard: nothing now pins that a caller-suppliedextra_args["kv_transfer_params"](including a spoofedkv_hint) never reachesSamplingParams. Correct today by construction, but the routing-host strip that used to enforce it is gone too, so one test feeding a spoofed hint and asserting it does not land is the remaining fence worth adding.lib/llm/src/kv_router.rs:1090-KvHint::new(message_id.unwrap_or_default(), ...)emits an empty message id when context is None; unreachable on attach paths today, but a UUID fallback or debug assert keeps the message-identity promise honest for future callers.- Stale "TRANSFER" vocabulary survives in a handful of doc comments and docstrings, and in one operator-visible warning (
kv_router.rs:1395) where "TRANSFER" maps to no config key or doc term an operator can look up; "P2P KV transfer hint" (as the Python error strings already say) reads right.
Pre-existing, already noted in-thread: the disagg decode path drops decode-targeted hints (same shape as the old design), and the 13 build-time-generated API nav entries.
Automated review pass; severity escalation stays with the human reviewers.
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ine-hint-contract-between-dynamo-engine Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> # Conflicts: # lib/llm/src/protocols/common/preprocessor.rs
…ine-hint-contract-between-dynamo-engine Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> # Conflicts: # lib/llm/src/kv_router/routing_host/kv_selection.rs
…ine-hint-contract-between-dynamo-engine Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> # Conflicts: # lib/llm/src/kv_router.rs # lib/llm/src/kv_router/routing_host/kv_selection.rs
|
should be merged with : ai-dynamo/kvcr#18 |
…ine-hint-contract-between-dynamo-engine Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
oandreeva-nv
left a comment
There was a problem hiding this comment.
lgtm for kvcr compatibility, please fix ci
…ine-hint-contract-between-dynamo-engine Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This change separates caller-controlled
RoutingHintsfrom one versionedKvHintmessage attached askv_hintbetween Dynamo and inference engines, aligned with vLLM RFC #53421. The open message lets engines consume independently versioned KV actions without another top-level schema change or a name collision with placement hints.CLOSES: DYN-3877
How This Was Implemented
KvHint { protocol_version, message_id, actions }and genericKvHintAction { action_id, action_type, action_version, payload }transport types.kv.fetch@1.0.router_hint*discovery wire names until they can be renamed with KVCC.kv_hintmessage only for supported targets.kv_transfer_params.kv_hint.Walkthrough
Mental model
The engine advertises which actions it supports. Dynamo uses that metadata after worker selection, builds one versioned KV hint message containing independent actions, and passes the complete message to the selected engine.
flowchart LR A["Engine advertises KV support"] --> B["Frontend records capability metadata"] B --> C["Router selects a worker"] C --> D["Dynamo builds KV actions"] D --> E["PreprocessedRequest.kv_hint"] E --> F["vLLM sets kv_transfer_params.kv_hint"] F --> G["KVCC receives the complete message"]State and ordering
The message uses protocol version
0.1. Every action carriesaction_id,action_type,action_version, and an opaque JSON-object payload, matching the vLLM RFC and its Python/Rust transport types.KvHintActionis deliberately open rather than a closed enum, so a future action type or version can cross the transport without changing the top-level message. Dynamo still constructs the currentkv.fetch@1.0action through a typed helper and shared constants. Its payload contains one source control endpoint and the root-aligned source block hashes.The router keeps the existing source-selection behavior. It chooses a compatible source with a longer cached prefix after selecting the target worker.
Request lifecycle
Model registration publishes the existing
router_hint,router_hint_worker_type, androuter_hint_source_control_endpointsmetadata. Capability negotiation is not part ofKvHint.After worker selection,
kv_hint_for_selection()builds the supported action list.RoutingHostattaches the message toPreprocessedRequest.kv_hint, and the vLLM adapter preserves the complete message while installing the connector's request-time transfer parameters.The vLLM adapter does not unwrap or interpret individual actions. It copies the message into the existing private
kv_transfer_params.kv_hintfield so KVCC can consume the same versioned contract.Boundaries and limitations
This change does not alter worker selection policy or the caller-controlled
RoutingHintscontract. It does not add dereference or other KV actions, and there is no response or acknowledgment path; existing KV events provide later visibility into cache behavior.The first-class vLLM request path is being introduced separately in vLLM PR #53423. Until that lands and Dynamo updates its vLLM dependency, this PR intentionally retains the existing private bridge instead of depending on unmerged vLLM code.
The existing
router_hint*discovery names are intentionally transitional. TODOs keep their later rename coupled to the matching KVCC change; this PR does not add aliases or dual publication.Validation
cargo test -p dynamo-kv-router kv_hints --lib(5 passed)cargo test -p dynamo-llm kv_hint --lib(2 passed)cargo test -p dynamo-llm protocols::common::preprocessor::tests --lib(7 passed)cargo check -p dynamo-backend-commoncargo fmt --all -- --checkruff checkon the changed Python filespython -m py_compileon the changed Python files.venv.Summary by CodeRabbit
New Features
Bug Fixes
Tests