Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/fern/pages/use-cases/agents/agent-harnesses.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,11 @@ An agent harness can compact older conversation history into a summary when its

For Codex, Dynamo preserves the optional `trigger`, `reason`, `implementation`, `phase`, and `strategy` values. The object appears only on the inference request that creates the summary. Invalid JSON and other request kinds leave it unset.

Workers receive the field in `AgentContext` but do not act on it.
Workers receive the field in `AgentContext`. Worker selection does not use the compaction details.

Compaction metadata is passive. It does not rotate `session_id`, change request routing, or evict KV cache blocks. See [Agent Tracing](agent-tracing.md#compaction-metadata) for the request-trace format.
Compaction keeps `session_id` and request routing stable. For a selected SGLang worker that advertises `kv_hint.deref.v1`, Dynamo sends `kv_hints.deref.apply_on=next_success`. SGLang registers the summary KV and defers the action. After the next request finishes successfully, SGLang releases the old session references and rotates the session generation before it protects the new leaf. An aborted request does not apply the action. SGLang does not delete KV blocks immediately. Unreferenced blocks remain available until normal memory pressure evicts them. Other backends do not receive the action.

Set `DYN_DISABLE_AGENT_COMPACTION_DEREF=1` on the frontend to keep compaction metadata but suppress the `deref` action. Use this switch for controlled comparisons. See [Agent Tracing](agent-tracing.md#compaction-metadata) for the request-trace format.

## See Also

Expand Down
2 changes: 1 addition & 1 deletion docs/fern/pages/use-cases/agents/agent-tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ For example, a Codex compaction request can produce:
}
```

Use `session_id` to group requests before, during, and after compaction. Compaction does not create a new session ID or change request placement. See [Agent Harnesses](agent-harnesses.mdx#compaction-signals) for current harness support.
Use `session_id` to group requests before, during, and after compaction. Dynamo derives the worker-facing `kv_hints.deref.apply_on=next_success` action from compaction metadata, but the action is not part of `agent_context` in the trace. SGLang releases the old session references after the next successful request. Compaction does not create a new session ID or change request placement. When `DYN_DISABLE_AGENT_COMPACTION_DEREF=1`, the compaction object remains in the trace but Dynamo suppresses the action. See [Agent Harnesses](agent-harnesses.mdx#compaction-signals) for current harness support and KV behavior.

For chat streams, Dynamo records finish metadata after parser and jail rewrites. Completion streams record the final OpenAI-compatible completion finish reason.

Expand Down
8 changes: 5 additions & 3 deletions lib/backend-common/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@ pub use dynamo_llm::protocols::common::llm_backend::{
LLMEngineOutput, LogProbs, TopLogprob, TopLogprobs,
};
pub use dynamo_llm::protocols::common::preprocessor::{
BootstrapInfo, KV_HINT_TRANSFER_CAPABILITY_KEY, KvHintAction, KvHints,
KvSourceLocationsPayload, MultimodalData, MultimodalDataMap, PrefillResult,
PreprocessedRequest, RoutingHints,
BootstrapInfo, KV_HINT_DEMOTE_CAPABILITY_KEY, KV_HINT_DEREF_CAPABILITY_KEY,
KV_HINT_PREFETCH_CAPABILITY_KEY, KV_HINT_TRANSFER_CAPABILITY_KEY, KvDemoteActionVersion,
KvDemotePayload, KvDerefActionVersion, KvDerefPayload, KvHintAction, KvHintProtocolVersion,
KvHints, KvPrefetchActionVersion, KvPrefetchPayload, KvSourceLocationsPayload, MultimodalData,
MultimodalDataMap, PrefillResult, PreprocessedRequest, RoutingHints,
};
pub use dynamo_llm::protocols::common::{
FinishReason, GuidedDecodingOptions, OutputOptions, SamplingOptions, StopConditions,
Expand Down
13 changes: 8 additions & 5 deletions lib/backend-common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,14 @@ pub use dynamo_llm::model_type::ModelInput;
pub use engine::{
AsyncEngineContext, BootstrapInfo, CompletionUsage, ComponentSnapshot, EngineConfig,
FinishReason, GenerateContext, GuidedDecodingOptions, HEALTH_CHECK_KEY,
KV_HINT_TRANSFER_CAPABILITY_KEY, KvEventPublisher, KvEventSource, KvHintAction, KvHints,
KvSourceLocationsPayload, LLMEngine, LLMEngineOutput, LLMEngineOutputExt, LlmRegistration,
LogProbs, Metrics, MetricsBindings, MetricsCtx, MultimodalData, OnPublisherReady,
OnSnapshotPublisherReady, OutputOptions, PrefillResult, PreprocessedRequest, RawEngine,
SamplingOptions, StopConditions, StopReason, TopLogprob, TopLogprobs, chunk, usage,
KV_HINT_DEMOTE_CAPABILITY_KEY, KV_HINT_DEREF_CAPABILITY_KEY, KV_HINT_PREFETCH_CAPABILITY_KEY,
KV_HINT_TRANSFER_CAPABILITY_KEY, KvDemoteActionVersion, KvDemotePayload, KvDerefActionVersion,
KvDerefPayload, KvEventPublisher, KvEventSource, KvHintAction, KvHintProtocolVersion, KvHints,
KvPrefetchActionVersion, KvPrefetchPayload, KvSourceLocationsPayload, LLMEngine,
LLMEngineOutput, LLMEngineOutputExt, LlmRegistration, LogProbs, Metrics, MetricsBindings,
MetricsCtx, MultimodalData, OnPublisherReady, OnSnapshotPublisherReady, OutputOptions,
PrefillResult, PreprocessedRequest, RawEngine, SamplingOptions, StopConditions, StopReason,
TopLogprob, TopLogprobs, chunk, usage,
};
pub use error::{BackendError, DynamoError, ErrorType};
pub use metrics::{ComponentGauges, EngineMetrics, LifecycleGauges};
Expand Down
167 changes: 167 additions & 0 deletions lib/kv-router/src/kv_hints.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ use crate::protocols::{ExternalSequenceBlockHash, WorkerWithDpRank};

/// The selected worker can consume a `TRANSFER` hint with the v1 payload.
pub const KV_HINT_TRANSFER_CAPABILITY_KEY: &str = "kv_hint.transfer.v1";
/// The selected worker can consume a `DEREF` hint with the v1 payload.
pub const KV_HINT_DEREF_CAPABILITY_KEY: &str = "kv_hint.deref.v1";
/// The selected worker can persist a session-owned KV prefix to its configured storage tier.
pub const KV_HINT_DEMOTE_CAPABILITY_KEY: &str = "kv_hint.demote.v1";
/// The selected worker can prepare the current request's prefix from its configured storage tier.
pub const KV_HINT_PREFETCH_CAPABILITY_KEY: &str = "kv_hint.prefetch.v1";

/// Worker runtime-data keys used to build transfer hints.
pub const KV_HINT_TRANSFER_WORKER_TYPE_RUNTIME_KEY: &str = "kv_hint_transfer_worker_type";
Expand All @@ -27,6 +33,24 @@ pub enum KvSourceLocationsActionVersion {
V1_0,
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub enum KvDerefActionVersion {
#[serde(rename = "1.0")]
V1_0,
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub enum KvDemoteActionVersion {
#[serde(rename = "1.0")]
V1_0,
}

#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub enum KvPrefetchActionVersion {
#[serde(rename = "1.0")]
V1_0,
}

/// Typed payload for the `kv.source_locations@1.0` point-to-point action.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct KvSourceLocationsPayload {
Expand All @@ -36,10 +60,55 @@ pub struct KvSourceLocationsPayload {
pub block_hashes: Vec<ExternalSequenceBlockHash>,
}

/// Typed payload for `kv.deref@1.0`.
///
/// The successful request releases only the session's logical references. Shared
/// leaves remain protected by their other owners; physical eviction remains an
/// engine-local allocator decision.
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct KvDerefPayload {}

/// Typed payload for `kv.demote@1.0`.
///
/// The target engine writes this session's owned prefix to its configured storage
/// tier, verifies the write, then releases only unshared device leaves. The
/// action id is the idempotency key; do not duplicate it in this payload.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct KvDemotePayload {
pub session_id: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub session_generation: Option<u64>,
}

/// Typed payload for `kv.prefetch@1.0`.
///
/// The target derives the exact prefix from the request it is already processing.
/// This is not an autonomous router-side fetch.
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct KvPrefetchPayload {}

/// One typed action in a [`KvHints`] envelope.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
#[serde(tag = "action_type")]
pub enum KvHintAction {
#[serde(rename = "kv.deref")]
Deref {
action_id: String,
action_version: KvDerefActionVersion,
payload: KvDerefPayload,
},
#[serde(rename = "kv.demote")]
Demote {
action_id: String,
action_version: KvDemoteActionVersion,
payload: KvDemotePayload,
},
#[serde(rename = "kv.prefetch")]
Prefetch {
action_id: String,
action_version: KvPrefetchActionVersion,
payload: KvPrefetchPayload,
},
#[serde(rename = "kv.source_locations")]
SourceLocations {
action_id: String,
Expand All @@ -49,6 +118,30 @@ pub enum KvHintAction {
}

impl KvHintAction {
pub fn deref(action_id: impl Into<String>) -> Self {
Self::Deref {
action_id: action_id.into(),
action_version: KvDerefActionVersion::V1_0,
payload: KvDerefPayload::default(),
}
}

pub fn demote(action_id: impl Into<String>, payload: KvDemotePayload) -> Self {
Self::Demote {
action_id: action_id.into(),
action_version: KvDemoteActionVersion::V1_0,
payload,
}
}

pub fn prefetch(action_id: impl Into<String>) -> Self {
Self::Prefetch {
action_id: action_id.into(),
action_version: KvPrefetchActionVersion::V1_0,
payload: KvPrefetchPayload::default(),
}
}

pub fn source_locations(
action_id: impl Into<String>,
payload: KvSourceLocationsPayload,
Expand All @@ -59,6 +152,15 @@ impl KvHintAction {
payload,
}
}

pub fn required_capability(&self) -> &'static str {
match self {
Self::Deref { .. } => KV_HINT_DEREF_CAPABILITY_KEY,
Self::Demote { .. } => KV_HINT_DEMOTE_CAPABILITY_KEY,
Self::Prefetch { .. } => KV_HINT_PREFETCH_CAPABILITY_KEY,
Self::SourceLocations { .. } => KV_HINT_TRANSFER_CAPABILITY_KEY,
}
}
}

/// Versioned actions for the selected backend request.
Expand All @@ -77,6 +179,10 @@ impl KvHints {
actions,
}
}

pub fn is_empty(&self) -> bool {
self.actions.is_empty()
}
}

#[derive(Debug, Clone, PartialEq, Eq)]
Expand Down Expand Up @@ -149,6 +255,67 @@ mod tests {
);
}

#[test]
fn serializes_versioned_deref_action() {
let hints = KvHints::new("msg-123", vec![KvHintAction::deref("deref")]);

assert_eq!(
serde_json::to_value(hints).unwrap(),
serde_json::json!({
"protocol_version": "0.1",
"message_id": "msg-123",
"actions": [{
"action_id": "deref",
"action_type": "kv.deref",
"action_version": "1.0",
"payload": {},
}],
})
);
}

#[test]
fn serializes_versioned_storage_actions_without_repeating_action_identity() {
let hints = KvHints::new(
"msg-123",
vec![
KvHintAction::demote(
"demote-1",
KvDemotePayload {
session_id: "session-a".to_string(),
session_generation: Some(7),
},
),
KvHintAction::prefetch("prefetch-1"),
],
);

assert_eq!(
serde_json::to_value(hints).unwrap(),
serde_json::json!({
"protocol_version": "0.1",
"message_id": "msg-123",
"actions": [
{
"action_id": "demote-1",
"action_type": "kv.demote",
"action_version": "1.0",
"payload": {
"session_id": "session-a",
"session_generation": 7,
},
},
{
"action_id": "prefetch-1",
"action_type": "kv.prefetch",
"action_version": "1.0",
"payload": {},
},
],
})
);
}

#[test]
fn best_source_selects_longest_eligible_prefix() {
let worker_a = WorkerWithDpRank::new(7, 0);
Expand Down
13 changes: 13 additions & 0 deletions lib/llm/src/kv_router.rs
Original file line number Diff line number Diff line change
Expand Up @@ -655,6 +655,19 @@ where
})
}

pub(crate) fn filter_kv_hints_for_worker(
&self,
worker_id: WorkerId,
mut hints: KvHints,
) -> Option<KvHints> {
let configs = self.workers_with_configs.borrow();
let config = configs.get(&worker_id)?;
hints
.actions
.retain(|action| config.supports_kv_hint(action.required_capability()));
(!hints.is_empty()).then_some(hints)
}

fn transfer_hint_for_selection(
&self,
target: WorkerWithDpRank,
Expand Down
16 changes: 15 additions & 1 deletion lib/llm/src/kv_router/push_router.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
use std::{sync::Arc, time::Duration};

use dynamo_kv_router::{
kv_hints::KvHints,
protocols::{TokensWithHashes, WorkerWithDpRank},
selector::WorkerSelector,
};
Expand Down Expand Up @@ -380,7 +381,20 @@ where

let (mut backend_input, context) = request.into_parts();
backend_input.routing_mut().dp_rank = Some(selection.worker.dp_rank);
backend_input.kv_hints = selection.kv_hints;
let mut actions = backend_input
.kv_hints
.take()
.map(|hints| hints.actions)
.unwrap_or_default();
if let Some(hints) = selection.kv_hints {
actions.extend(hints.actions);
}
backend_input.kv_hints = (!actions.is_empty())
.then(|| KvHints::new(&context_id, actions))
.and_then(|hints| {
self.chooser
.filter_kv_hints_for_worker(selection.worker.worker_id, hints)
});
let updated_request = context.map(|_| backend_input);
guard.record_prefill_start();

Expand Down
8 changes: 8 additions & 0 deletions lib/llm/src/preprocessor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,10 @@ fn attach_agent_context_from_context(
if let Ok(agent_context) = context.get::<crate::protocols::common::extensions::AgentContext>(
crate::protocols::common::extensions::AGENT_CONTEXT_CONTEXT_KEY,
) {
request.kv_hints = crate::protocols::common::extensions::kv_hints_from_agent_context(
&agent_context,
context.id(),
);
request.agent_context = Some(agent_context.as_ref().clone());
}
}
Expand Down Expand Up @@ -6665,6 +6669,10 @@ mod tests {
wire["agent_context"]["compaction"]["trigger"],
serde_json::json!("manual")
);
assert_eq!(
wire["kv_hints"]["actions"][0]["action_type"],
serde_json::json!("kv.deref")
);
}

#[test]
Expand Down
12 changes: 10 additions & 2 deletions lib/llm/src/protocols/agents.rs
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ pub(crate) struct AgentContextHeaderValues {
#[derive(Deserialize)]
struct CodexTurnMetadata {
request_kind: Option<String>,
turn_id: Option<String>,
window_id: Option<String>,
compaction: Option<AgentCompaction>,
}

Expand Down Expand Up @@ -118,8 +120,14 @@ pub(crate) fn agent_context_header_values(headers: &HeaderMap) -> Option<AgentCo
fn codex_compaction_header_value(headers: &HeaderMap) -> Option<AgentCompaction> {
let raw = borrowed_header_value(headers, HEADER_CODEX_TURN_METADATA)?;
let metadata: CodexTurnMetadata = serde_json::from_str(raw).ok()?;
(metadata.request_kind.as_deref() == Some("compaction"))
.then(|| metadata.compaction.unwrap_or_default())
(metadata.request_kind.as_deref() == Some("compaction")).then(|| {
let mut compaction = metadata.compaction.unwrap_or_default();
compaction.operation_id = metadata
.turn_id
.zip(metadata.window_id)
.map(|(turn_id, window_id)| format!("{turn_id}:{window_id}"));
compaction
})
}

pub(crate) fn session_affinity_header_value(headers: &HeaderMap) -> Option<String> {
Expand Down
Loading
Loading