Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions fern/versions/latest/pages/observability/rollout-evidence.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,113 @@ for completion in completions:
row, result = await completion # the record the shard holds, or the typed failure record
```

## Tool Timing v2

Set `trajectory_schema_version: "2.0"` alongside the collection settings above to retain
attempt-level tool timing. The default remains `"1.0"` for existing readers. Direct callers can use
`RolloutCollectionHelper.run_examples(..., enrich_results=True, trajectory_schema_version="2.0")`
or await `nemo_gym.rollouts.enrich_run_result(row, result, capture_dirs=[...],
observability_enabled=True, trajectory_schema_version="2.0")` after a run. Both paths use the
collector's shared capture merge and normalization. Manual enrichment accepts a collected
record; `run_examples` also handles materialized episode replies without changing their shape.
Raw `run_examples` behavior stays unchanged.

V2 names the capture namespace `capture_key`. Each tool execution has a stable `span_id`, its
native request ID, optional agent instance and requesting model-call ID, and `timings` with
explicit boundary, producer, clock domain, endpoints and duration method. OpenCode reads its
persisted tool parts as lifecycle measurements. Repeated native IDs remain separate attempts;
a started but unfinished tool stays `incomplete`, including when its enclosing harness times out.

The timing `boundary` describes the interval measured, independently of how its evidence is
stored or delivered:

| Boundary | Measured interval |
| --- | --- |
| `lifecycle` | Harness dispatch/start through result/completion handling; may include queuing. |
| `executor` | Instrumented executor entry through return or observed termination. |
| `transcript` | Native call-record timestamp through native result-record timestamp. |
| `event_receipt` | Observer receipt of start and end events. |

`timing_source` identifies the instrumentation. Reading native lifecycle timestamps from a
file or database preserves their `lifecycle` boundary; it does not make them `transcript`
timing. Likewise, receiving those timestamps later does not make them `event_receipt` timing.
A transcript timestamp must not be substituted for an executor endpoint.

Current producer mappings and migration constraints:

| Producer evidence | Boundary or migration status |
| --- | --- |
| OpenCode persisted `state.time.start/end` | `lifecycle`; database delivery is independent of this meaning. V1 retains its `artifact` source label. |
| Pi stdout start/end event receipt | Semantically `event_receipt`. The generic legacy upgrader currently maps `harness` to `lifecycle`; this needs explicit producer qualification. Producer-side timestamps are needed for lifecycle timing. |
| Simple Agent, compaction agent and agentic turn simulation dispatch wrappers | Legacy `executor` labels are coarse client-side measurements. Qualify their actual dispatch/response boundary before migrating them. |
| Hermes tool wrapper / hooks | Executor-wrapper / lifecycle scope respectively; preserve the instrumentation's actual endpoints. |
| NOOA before/after hooks | Lifecycle scope. |
| Stirrup native tool timestamps | Verify the pinned runtime's measured endpoints before assigning a boundary. |

New adapters should qualify the measured endpoints explicitly rather than copy a legacy
`harness`, `executor`, or `artifact` label. Cross-harness migration aims to use the same semantic
boundaries wherever the instrumentation supports them, leaving unsupported measurements unknown.

OpenCode runtime collection assigns one wall-clock domain per local launch or retained sandbox
session. Repeated snapshots in that session share it. Standalone database imports require explicit
clock-owner evidence and otherwise retain an unknown domain. Gym's model-call endpoints come from
its model-server ASGI middleware, on a separate observer clock; they cannot share OpenCode's domain.
Model capture does not yet emit its own domain, so v2 unions across multiple model calls also
remain unknown until that provenance is populated.
Tool-only busy-time unions are therefore available with runtime collection, while unions across
model and tool clocks still require evidence of clock comparability.

`TrajectoryRecord.model_validate(...)` accepts v1 and v2 and rejects unknown versions. Use
`nemo_gym.trajectory.project_trajectory_v1(...)` explicitly for a legacy reader. Where each
attempt has an explicit, unique `native_tool_call_id` and the destination does not collide with
retained IDs, v1 restores that ID throughout the invocation's tools, conversation and turn
references, including matching tool-item IDs. Already-native turn references remain unchanged
when exact captured model-call evidence proves their request identity; unproven references
remain collision barriers. Enrichment makes one projection decision for the retained trajectory
and observation bundle before stripping metadata, so repeated export cannot split an attempt.
Observation bundles also restore spawn and compaction references. Reused or
ambiguous native IDs retain their normalized IDs with a `tool_native_identity_unrepresentable`
gap; consumers needing those attempts should retain v2. Primary responses and captured model
payloads are unchanged. The projection
loses attempt, instance, parent, failure and detailed timing provenance. It projects one measured
boundary into legacy flat fields; multiple boundaries require an explicit consumer choice and
produce no flat interval. Legacy measurements with unknown boundaries remain diagnostic evidence
when upgraded. In particular, the coarse `artifact` source does not identify a timing boundary.
Raw observation spans and parent references are qualified by producer and capture namespace on
import together with their diagnostics; canonical trajectory spans and gaps retain those
identities on replay. An adapter gap that cannot identify one imported attempt keeps its code
and original detail, with `unresolved_span_reference` in its diagnostic detail and no canonical
`span_id`. It is never copied onto multiple attempts.

Tool names are span attributes by default. Metrics should use a bounded allowlist with an
`other` bucket, and omit missing durations instead of recording them as zero.

Missing agent identity, clock provenance, completion, failure details or exact requesting-call
links produce scoped gaps. Tool IDs alone never select a model call. Payload omission happens
after joins and leaves a verification gap. Durations from different clock domains can be summed
as work, but their endpoints cannot establish overlap. The wall-time reader selects lifecycle
measurements by default and leaves incompatible-clock unions unknown.

Invocation IDs must remain unique across the entire trajectory, including agent and verifier
invocations. Reusing an invocation ID across named agent instances is unsupported and rejected;
complete support depends on the capture-identity integration. Optional instance evidence is
preserved without changing the invocation namespace.

The shared OpenCode observation parser defaults to legacy v1 IDs and flat artifact timings for
existing callers. OpenCode explicitly requests `observation_schema_version="2.0"` before any
collector export, including when the exported trajectory uses v1. `run_examples` applies its
explicit schema choice, or the configured/default version, even with `enrich_results=False`;
this projection reads no captures and does not add enrichment. Direct HTTP `/run` responses
remain raw producer evidence. Caller-facing schema selection happens at `run_examples` or
export, rather than through a separate HTTP version-negotiation protocol.

OpenCode observation conversation IDs use the persisted part identity from the first snapshot;
`native_tool_call_id` retains the model-facing request ID. Runtime execution and primary response
output retain their native IDs. Interrupted calls with persisted parts retain usable partial
timing and an explicit completion gap. Missing or unreadable artifacts produce coverage gaps.
There is no guaranteed recovery of interrupted activity that was never persisted, and no
streaming recorder is enabled by this change.

## Which Artifact to Read

| Artifact | Produced by | Use |
Expand Down
31 changes: 18 additions & 13 deletions responses_api_agents/opencode_agent/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@
AgentObservationBundle,
ObservationGap,
SandboxObservation,
ToolCallObservation,
TrajectoryRecord,
)
from nemo_gym.utils import is_nemo_gym_fastapi_entrypoint
Expand Down Expand Up @@ -639,17 +638,25 @@ async def _run_opencode(
if timed_out
else parse_opencode_session(db_path, gym_reasoning=self.config.model_server is not None)
)
observations = AgentObservationBundle(source="opencode")
observations = AgentObservationBundle(schema_version="2.0", source="opencode")
if collect_observations:
try:
observations = _parse_opencode_session(
db_path, invocation_id, trajectory, model_ref=self.config.model_server
db_path,
invocation_id,
trajectory,
model_ref=self.config.model_server,
# This fresh local database is written by one OpenCode launch.
clock_domain=f"opencode-wall:{uuid4().hex}",
agent_instance=f"opencode-agent:{uuid4().hex}",
observation_schema_version="2.0",
)
except Exception:
LOG.exception("failed to read OpenCode session artifact")
if trajectory is not None:
trajectory.gaps.append(ObservationGap(code="turns_unavailable"))
observations = AgentObservationBundle(
schema_version="2.0",
source="opencode",
records=[AgentInvocation(invocation_id=invocation_id)],
gaps=[
Expand Down Expand Up @@ -1286,7 +1293,7 @@ async def _close_agent_session_state(self, state: AgentSessionState) -> AgentClo
assert isinstance(state, OpenCodeSandboxSession)
await state.close(self.config.session_close_timeout_seconds)
observations = state.observations or AgentObservationBundle(
source="opencode", gaps=[ObservationGap(code="agent_activation_interrupted")]
schema_version="2.0", source="opencode", gaps=[ObservationGap(code="agent_activation_interrupted")]
)
# A capture that close finished after the activation returned reports its export failure here.
observations.gaps.extend(state.take_export_gaps())
Expand Down Expand Up @@ -1493,7 +1500,9 @@ async def _activate_agent_session_state(
stop_reason="timeout",
duration_ms=(monotonic() - started) * 1000,
)
state.activation_observations = AgentObservationBundle(source="opencode", records=[invocation])
state.activation_observations = AgentObservationBundle(
schema_version="2.0", source="opencode", records=[invocation]
)
state.activation_trajectory = TrajectoryRecord(
task_id=state.request.task_id.task_id,
rollout_id=state.request.episode_id.capture_key,
Expand All @@ -1513,7 +1522,7 @@ async def _activate_agent_session_state(
if failure is not None:
observations = state.activation_observations
if observations is None:
observations = AgentObservationBundle(source="opencode")
observations = AgentObservationBundle(schema_version="2.0", source="opencode")
state.activation_observations = observations
if not any(
isinstance(r, AgentInvocation) and r.parent_invocation_id is None for r in observations.records
Expand Down Expand Up @@ -1721,6 +1730,7 @@ async def _session_response(
if state.is_continuation and state.harness_session_id is not None:
invocation_id += f":{state.harness_session_id}"
observations = AgentObservationBundle(
schema_version="2.0",
source="opencode",
records=(
[
Expand Down Expand Up @@ -1830,13 +1840,8 @@ async def _session_response(
record.error_type = "server_error"
cancelled = isinstance(failure, asyncio.CancelledError)
timed_out = result is not None and result["timed_out"]
if cancelled or timed_out:
# Gym stopped the process, so a tool OpenCode left running ended with that stop.
termination = "timeout" if timed_out else "cancelled"
for record in observations.records:
if isinstance(record, ToolCallObservation) and record.status == "incomplete":
record.status = termination
record.error_type = f"invocation_{termination}"
# The enclosing process deadline does not establish a tool's own terminal outcome.
Comment thread
lafrancef marked this conversation as resolved.
Resolved
# Persisted starts without native completion remain incomplete through normalization.
if result is not None:
sandbox_outcome = (
"timeout" if timed_out else "cancelled" if cancelled else "failed" if error or failure else "completed"
Expand Down
Loading
Loading