Skip to content

feat(kv): hints! - #7

Closed
ishandhanani wants to merge 6 commits into
idhanani/dyn-3877-define-hint-contract-between-dynamo-enginefrom
idhanani/dyn-13010-session-cache-priority-control
Closed

ishandhanani wants to merge 6 commits into
idhanani/dyn-3877-define-hint-contract-between-dynamo-enginefrom
idhanani/dyn-13010-session-cache-priority-control

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Aug 12, 2026 •

Copy link
Copy Markdown
Owner

This PR adds kv.deref@1.0 to the versioned KvHints.actions contract from ai-dynamo/dynamo#13134 and lowers it through Dynamo's SGLang sidecar. This lets agent compaction and session-final requests control cache lifetime without changing worker selection or sending unsupported actions.

CLOSES: DYN-13010

How This Was Implemented

  • Derive CURRENT_SUCCESS for a final session request and NEXT_SUCCESS for a compaction request; DYN_DISABLE_AGENT_COMPACTION_DEREF=1 suppresses only the compaction action.
  • Add the dereference action to the generic action list owned by refactor(router): define typed KV hint contract ai-dynamo/dynamo#13134, compose request and router actions after worker selection, and retain only actions supported by the selected worker.
  • Lower session_id and the dereference payload into the native SGLang gRPC request, while registering only the SGLang capabilities reported by discovery.
Walkthrough

Mental model

Agent lifecycle metadata describes why a request exists. Dynamo converts that metadata into a backend action, performs normal worker selection, and attaches the action only after it knows that the selected worker supports the matching contract version.

flowchart LR
    A["Agent headers"] --> B["AgentContext"]
    B --> C["KvHints action list"]
    C --> D["Normal worker selection"]
    D --> E["Capability filter and action composition"]
    E --> F["SGLang gRPC request"]
    F --> G["SGLang session cache"]
Loading

State and ordering

session_final=true produces kv.deref/CURRENT_SUCCESS. A compaction request produces kv.deref/NEXT_SUCCESS, which asks SGLang to retain the compacted summary and release the old session references only after the next ordinary request succeeds. If both signals are present, session_final wins because the session is ending. The environment switch disables only compaction-derived actions; final-session cleanup remains enabled.

Request lifecycle

The frontend parses agent headers into AgentContext, and preprocessing derives a KvHints envelope containing kv.deref@1.0. Worker selection remains unchanged. After selection, the push router combines request-originated and router-originated actions, independently removes actions unsupported by the selected worker, and forwards the remaining envelope. The SGLang sidecar maps only the dereference action and session ID into its protobuf; SGLang enforces success and failure timing from apply_on.

Boundaries and limitations

This PR does not alter placement, scoring, or session affinity. Dynamo's SGLang sidecar does not claim or lower source-location actions. Unadvertised actions are dropped rather than sent optimistically, and disabling compaction dereference does not remove compaction metadata from request tracing.

Validation

  • cargo check -p dynamo-kv-router -p dynamo-llm -p dynamo-backend-common -p dynamo-sglang-sidecar
  • cargo test -p dynamo-kv-router -p dynamo-llm -p dynamo-sglang-sidecar --lib --quiet (2,990 passed, 7 ignored)
  • cargo fmt --all
  • git diff --check

Summary

  • Let custom worker pickers attach session priority, storage demotion, and prefetch actions to the selected request.
  • Carry the selected router hint through Dynamo preprocessing and the native SGLang sidecar protobuf.
  • Keep malformed hints fail-open and preserve the default routing path.
  • Update the current custom-policy examples for the WorkerPick result type.

Validation

  • 860 dynamo-kv-router tests passed; 2 ignored.
  • KV-router benches and three custom-policy examples compiled.
  • 29 SGLang sidecar tests, executable and proto contract tests passed.
  • 4 SGLang mocker sidecar integration tests passed.
  • Exact-head selector and router-hint tests passed after the final main rebase.

@ishandhanani
ishandhanani deployed to external_collaborator August 12, 2026 00:31 — with GitHub Actions Active
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 12, 2026 00:57 — with GitHub Actions Failure
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 12, 2026 21:11 — with GitHub Actions Failure
@github-actions github-actions Bot added documentation Improvements or additions to documentation backend::vllm backend::sglang labels Aug 12, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-13010-session-cache-priority-control branch from 8629042 to e5135af Compare August 13, 2026 07:05
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 13, 2026 07:05 — with GitHub Actions Failure
@ishandhanani
ishandhanani changed the base branch from main to idhanani/dyn-3877-define-hint-contract-between-dynamo-engine August 13, 2026 07:06
@ishandhanani ishandhanani changed the title feat(router): emit SGLang session cache control hints feat(kv): emit SGLang session dereference hints Aug 13, 2026
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-13010-session-cache-priority-control branch from e5135af to 962faf4 Compare August 13, 2026 07:08
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 13, 2026 07:08 — with GitHub Actions Failure
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyn-13010-session-cache-priority-control branch from 962faf4 to 95a3ff0 Compare August 14, 2026 00:15
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 14, 2026 00:15 — with GitHub Actions Failure
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 15, 2026 01:11 — with GitHub Actions Failure
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 15, 2026 19:36 — with GitHub Actions Failure
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 16, 2026 07:32 — with GitHub Actions Failure
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani deployed to external_collaborator August 17, 2026 17:34 — with GitHub Actions Active
@ishandhanani

Copy link
Copy Markdown
Owner Author

This consolidates the router-to-engine KV interface around a single selected-worker action envelope, rather than cache-manager-specific request fields.

The envelope is KvHints { protocol_version, message_id, actions[] }. message_id identifies the request envelope. Every action has { action_id, action_type, action_version, payload }; the action id is the operation identity and is intentionally not repeated in a payload. A worker advertises kv_hint.deref.v1, kv_hint.demote.v1, and kv_hint.prefetch.v1 independently, and Dynamo lowers only actions the selected worker says it supports. Existing P2P source-location metadata remains a separate transfer action; these three do not carry a worker plan or expose a session-cache implementation.

kv.deref@1.0 has {} payload. SGLang records it on a session-native request and, only after that request succeeds, releases the logical references owned by that session generation. Shared radix leaves remain protected by other references, and physical device eviction remains entirely the engine allocator decision.

kv.demote@1.0 has { session_id, session_generation? }. The SGLang cache manager snapshots that generation, writes the owned prefix through its configured storage tier, waits for all write acknowledgements, revalidates the snapshot, then uses the same session-release machinery as DEREF and demotes only backed device leaves with no remaining session reference. A write failure, a stale generation, a busy node, or a concurrent mutation retains the device KV. This is the useful primitive for an admission/controller policy that needs to make capacity deliberately.

kv.prefetch@1.0 has {} payload because the target can derive the exact prefix from the request it has already received. In SGLang it is a request-scoped override that lets the normal storage prefetch path consider a page-aligned prefix even when native size and rate gates would otherwise skip it. It does not create a router-initiated background request or transfer plan. The scheduler and cache still own allocation, lookup, and abort behavior.

What we observed in the compaction replay is important for policy: DEREF was correct but did not improve performance. On the 4xH100 concurrency-4 run, backend KV hit rate fell 2.60 percentage points, physical evict/input rose 14.78 percent, TTFT p50 rose 5.98 percent, and TTFT p95 rose 23.72 percent. DEREF only changes logical ownership; under pressure that changed the candidates and caused shared system/tool prefix KV to be refilled later. It is therefore a supported lifecycle mechanism, but not enabled as an optimization policy today.

DEMOTE and PREFETCH are intentionally mechanisms, not claims of a current policy win. The previous Mooncake write-back POC established the storage path; this implementation makes the safety boundary explicit. PREFETCH has no claimed benchmark win yet and should be driven by a controller only when it has a concrete upcoming request and target worker. The next work is controller/admission policy and telemetry, not more one-off cache mutations.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani had a problem deploying to external_collaborator August 17, 2026 19:52 — with GitHub Actions Failure
@ishandhanani ishandhanani changed the title feat(kv): emit SGLang session dereference hints feat(kv): hints! Aug 18, 2026

This branch had an error being deployed

1 failed deployment
external_collaborator — a761248b Deployed Aug 17, 2026 by ishandhanani via ok-to-test #20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant