Repository navigation
feat(kv): hints! - #7
ishandhanani wants to merge 6 commits into
Conversation
8629042 to
e5135af
Compare
e5135af to
962faf4
Compare
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
962faf4 to
95a3ff0
Compare
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
|
This consolidates the router-to-engine KV interface around a single selected-worker action envelope, rather than cache-manager-specific request fields. The envelope is
What we observed in the compaction replay is important for policy: DEREF was correct but did not improve performance. On the 4xH100 concurrency-4 run, backend KV hit rate fell 2.60 percentage points, physical evict/input rose 14.78 percent, TTFT p50 rose 5.98 percent, and TTFT p95 rose 23.72 percent. DEREF only changes logical ownership; under pressure that changed the candidates and caused shared system/tool prefix KV to be refilled later. It is therefore a supported lifecycle mechanism, but not enabled as an optimization policy today. DEMOTE and PREFETCH are intentionally mechanisms, not claims of a current policy win. The previous Mooncake write-back POC established the storage path; this implementation makes the safety boundary explicit. PREFETCH has no claimed benchmark win yet and should be driven by a controller only when it has a concrete upcoming request and target worker. The next work is controller/admission policy and telemetry, not more one-off cache mutations. |
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
This PR adds
kv.deref@1.0to the versionedKvHints.actionscontract from ai-dynamo/dynamo#13134 and lowers it through Dynamo's SGLang sidecar. This lets agent compaction and session-final requests control cache lifetime without changing worker selection or sending unsupported actions.CLOSES: DYN-13010
How This Was Implemented
CURRENT_SUCCESSfor a final session request andNEXT_SUCCESSfor a compaction request;DYN_DISABLE_AGENT_COMPACTION_DEREF=1suppresses only the compaction action.session_idand the dereference payload into the native SGLang gRPC request, while registering only the SGLang capabilities reported by discovery.Walkthrough
Mental model
Agent lifecycle metadata describes why a request exists. Dynamo converts that metadata into a backend action, performs normal worker selection, and attaches the action only after it knows that the selected worker supports the matching contract version.
flowchart LR A["Agent headers"] --> B["AgentContext"] B --> C["KvHints action list"] C --> D["Normal worker selection"] D --> E["Capability filter and action composition"] E --> F["SGLang gRPC request"] F --> G["SGLang session cache"]State and ordering
session_final=trueproduceskv.deref/CURRENT_SUCCESS. A compaction request produceskv.deref/NEXT_SUCCESS, which asks SGLang to retain the compacted summary and release the old session references only after the next ordinary request succeeds. If both signals are present,session_finalwins because the session is ending. The environment switch disables only compaction-derived actions; final-session cleanup remains enabled.Request lifecycle
The frontend parses agent headers into
AgentContext, and preprocessing derives aKvHintsenvelope containingkv.deref@1.0. Worker selection remains unchanged. After selection, the push router combines request-originated and router-originated actions, independently removes actions unsupported by the selected worker, and forwards the remaining envelope. The SGLang sidecar maps only the dereference action and session ID into its protobuf; SGLang enforces success and failure timing fromapply_on.Boundaries and limitations
This PR does not alter placement, scoring, or session affinity. Dynamo's SGLang sidecar does not claim or lower source-location actions. Unadvertised actions are dropped rather than sent optimistically, and disabling compaction dereference does not remove compaction metadata from request tracing.
Validation
cargo check -p dynamo-kv-router -p dynamo-llm -p dynamo-backend-common -p dynamo-sglang-sidecarcargo test -p dynamo-kv-router -p dynamo-llm -p dynamo-sglang-sidecar --lib --quiet(2,990 passed, 7 ignored)cargo fmt --allgit diff --checkSummary
Validation