Proposal: observability semantics for durable agentic runtime execution

Summary

OpenTelemetry GenAI semantic conventions provide a strong foundation for model inference, agent invocation, tool execution, MCP, and provider-specific telemetry.

However, production agentic systems often behave as durable runtimes rather than short-lived request-response flows. They can persist workflow state, pause, resume from checkpoints, delegate work, invoke mutable tools, retry operations, and compensate for partial external effects.

This issue proposes a discussion on whether additional semantic guidance is needed for observing these runtime behaviors in a consistent, framework-agnostic way.

Problem

Today, an observability backend can often show:

an LLM inference call;
a tool call;
an agent invocation; and
an MCP request.

It is harder to answer the following questions consistently across runtimes:

Which durable execution does this work belong to?
Was the workflow resumed from persisted state?
Which state transition led to an action?
Which model decision triggered a tool call when they are not in a direct parent-child relationship?
Did a mutable external action use idempotency or deduplication?
Was a failed external effect compensated, partially compensated, or left unresolved?

These questions matter when operating long-running, stateful, or multi-agent workflows.

Proposed Direction

This is not a proposal to replace existing gen_ai.* conventions.

The proposed direction is to retain existing GenAI spans for inference, agent invocation, and tool execution, while defining guidance for runtime-specific concepts where needed:

Durable execution identity
- A logical execution identifier distinct from the trace identifier.
- A continuity identifier for workflows that span multiple Runs or resumptions.
Checkpoint and resume semantics
- Events or spans for checkpoint persistence and recovery.
- Guidance for associating a resumed execution with prior causal context.
State transitions
- A safe, low-cardinality representation of workflow state changes.
- Recording changed keys, state version, transition reason, and actor without requiring full state capture.
Non-linear causality
- Guidance on using span links for resume, delegation, retries, fan-out, aggregation, and compensation.
- In particular, linking an execute_tool operation to the inference or decision that triggered it.
External effects and compensation
- A way to distinguish read-only operations from mutable external effects.
- Guidance for idempotency keys, retries, deduplication, and Saga-style compensating operations.

Scope Boundaries

This proposal does not suggest:

recording private chain-of-thought;
requiring prompt, response, tool-argument, or full-state capture;
defining a new workflow engine;
replacing framework-specific state models; or
creating a single generic ontology for every agent framework.

The goal is operational interoperability: making durable agent behavior traceable while allowing individual runtimes to preserve their own execution models.

Initial Reference Material

Carniatto Labs has published an open architectural proposal and adoption guide:

The proposal is intentionally organized as an extension of existing OpenTelemetry semantics, not as a competing standard.

Request for Feedback

I would appreciate feedback on:

Whether these concerns belong in the GenAI semantic conventions repository, core semantic conventions, or framework-specific instrumentation guidance.
Which concepts are broadly observable across agent runtimes today.
Whether checkpoint/resume and compensation should be modeled as spans, events, links, attributes, or a combination.
Existing work or proposals that already address these runtime concerns.
Suitable runtimes and reference scenarios for validating any potential convention.

I am especially interested in perspectives from maintainers and users of agent frameworks, workflow engines, MCP implementations, and observability backends.

Items 1 and 2, and the resume half of item 4, are in flight: #159 defined the gap and PR #445 implements it, with gen_ai.agent.execution.id as the logical run id across suspend/resume and process boundaries, gen_ai.agent.paused/checkpointed/resumed events, and a flat resumed_from.type/.id pair linking a resumed segment to the boundary it continues from. Items 3 and 5, state transitions and external effects with compensation, are not covered there and read as the new ground worth scoping this issue to.

Items 1 and 2, and the resume half of item 4, are in flight: #159 defined the gap and PR #445 implements it, with gen_ai.agent.execution.id as the logical run id across suspend/resume and process boundaries, gen_ai.agent.paused/checkpointed/resumed events, and a flat resumed_from.type/.id pair linking a resumed segment to the boundary it continues from. Items 3 and 5, state transitions and external effects with compensation, are not covered there and read as the new ground worth scoping this issue to.

Thanks, this is really helpful context. It looks like state transitions and external effects/compensation are the more interesting gaps to explore now.

I’d love to hear your perspective on these and have you contribute to the discussion as the work evolves.

On item 5, the distinction with governance value is mutating versus read-only effects, declared before execution rather than inferred after; it is what a policy gate needs in order to require idempotency or approval for one class and not the other. A compensating action should carry a span link to the effect it compensates, since parent-child rarely survives a retry or replay boundary. We consume this telemetry on the monitoring side and will follow as it firms up.

One semantic boundary may be worth making explicit around item 4 (“which model decision triggered a tool call”), especially if span links are used when parent-child structure does not survive the runtime boundary.

OpenTelemetry Links are appropriate for representing causal association across non-tree relationships. But for agent observability there is a stronger claim that can easily get conflated with that: that the content of a particular plan/decision operation actually participated in producing or selecting the later tool action.

Those are not always the same thing.

A concrete counterexample:

An action-bearing command C88 is generated from source S55 and committed.
A later plan/decision span R123 contains the same operative outcome.
The eventual execute_tool span A92 is linked to R123 because they belong to the same execution path or because R123 is the nearest relevant decision record.
The trace and link are valid, but retained runtime state shows that C88 was not modified, re-derived, reselected, or replaced after R123 appeared.

In that case the span relation is still useful observability data, but reading the link as “R123 produced/selected A92” would be too strong.

The converse also matters: wording or representation may change between a decision and a tool action while a retained mapping or selection record still supports that the earlier decision was action-bearing.

So I would suggest keeping two layers distinct in any guidance that emerges from this issue:

execution relation: parent/child, span link, execution id, checkpoint/resume relation, etc.;
action-formation relation: appraisable evidence that an earlier plan/decision produced, transformed, ranked, or selected the later action.

The first can exist without the second. If the second is not observable, the telemetry should support “relation unresolved” rather than encouraging consumers to infer action formation from correlation, temporal order, or semantic similarity alone.

I do not think this requires a new span type or mandatory attribute at this stage. Even a short semantic note around non-linear causality could prevent observability backends from over-interpreting span links as content-level derivation evidence.

A closely related audit discussion is here:
mirjak/draft-audit-architecture#34

And an implementation-neutral case set exercising supported descent, independent reconstruction, and unresolved relation is here:
FG-TIDA/use-cases#12

I am referencing those only as concrete examples of the distinction, not proposing their labels or encoding for OpenTelemetry.

Given the scope narrowing in this thread, I think items 3 and 5 can be made more concrete with a small case matrix before choosing any final attribute/event names.

I would distinguish effect intent before execution from effect outcome after execution, and keep logical-effect identity separate from attempt/retry identity.

Six useful conformance-style cases:

READ_ONLY_EFFECT — read public metadata; no external mutation; ordinary retry is safe.
MUTATING_EFFECT / idempotent — one logical effect has a stable effect ID + idempotency key; retries get new attempt IDs but must not become a second intentional mutation.
APPROVAL_REQUIRED — the intended mutation is observable while approval remains pending; telemetry must not imply authorization or execution.
UNRESOLVED_EFFECT — dispatch happened, then the connection died before acknowledgement. This should remain unresolved until authoritative reconciliation; transport failure alone must not be rewritten as effect failure.
COMPENSATED_EFFECT — compensation is a distinct operation related to the completed original effect (a span link seems more robust than assuming parent/child continuity across replay/resume).
UNRESOLVED_COMPENSATION — the original effect remains completed while the compensation attempt is unresolved; history should not be rewritten as if rollback succeeded.

A few semantics this matrix seems to require, independent of naming:

Can effect class / mutability be declared before execution rather than inferred later?
Can a logical effect ID survive retries while attempt IDs remain unique?
Can approval-required be represented without implying approval?
Can “unknown/unresolved after dispatch” remain first-class until reconciliation?
Can a compensation link to the original effect across retry/replay boundaries?
Can observability distinguish an execution relation from stronger evidence that a previous decision actually formed/selected the action?

None of these cases requires prompt content, private chain-of-thought, full tool arguments/results, or full runtime state. They can be exercised with sanitized fixtures.

If this framing matches the direction of #462, the next useful step could be turning these into one or two reference scenarios (for example an idempotent mutation with lost acknowledgement, and a compensation that crosses a resume boundary) before standardizing field names.

A physical-control case may help sharpen item 5 because an external effect is not always safely retryable or inferable from transport success.

In a ventilation-control runtime I maintain, the boundary is:

proposed action
→ zero-motion/read-only preflight
→ explicit authorization
→ command dispatch
→ command ACK
→ measured actuator feedback
→ mandatory safe closeout
→ fresh post-action sensor state
→ next controller origin

The implementation deliberately refuses to promote software/mock execution to physical evidence, and it does not treat a dispatched command or transport ACK as proof that the intended physical state occurred:
https://github.com/hippoley/AirTrajectory

That suggests a useful physical-effect reference scenario for this discussion:

an agent proposes a mutating actuator command;
policy/readiness gates pass, but no mutation has happened yet;
the command is dispatched and acknowledged;
authoritative measured feedback either confirms the physical effect, contradicts it, or remains unavailable;
the runtime performs a bounded safe-closeout/compensation;
only fresh measured state becomes the origin for the next decision.

The semantic distinction I would want observability to preserve is therefore not only read-only vs mutating, but also:

dispatch/acceptance evidence vs effect evidence;
intended target state vs authoritatively observed state;
transport failure vs physical-effect unresolved;
compensation requested vs compensation physically verified.

This matters for retries: after an ACK is lost, blindly replaying an actuator command can itself be unsafe. Reconciliation against authoritative state may need to precede any retry.

I don't think this necessarily implies physical-specific attributes in GenAI conventions. It may be more useful as a reference scenario that tests whether the generic external-effect semantics remain valid when the side effect is irreversible, delayed, or only independently observable.

If that boundary is useful to #462, I can reduce the existing runtime into a small sanitized trace fixture/reference scenario around “lost ACK → reconcile measured state → bounded compensation” rather than proposing field names first.

I followed through on the physical-effect reference scenario I offered earlier and reduced it to a small sanitized fixture rather than proposing field names.

Review artifact:
https://github.com/hippoley/AirTrajectory/pull/168

Machine-readable scenario:
https://github.com/hippoley/AirTrajectory/blob/interop/otel-physical-effect-reference-scenario/interop/opentelemetry/physical-effect-lost-ack-v0.1.json

The sequence is deliberately generic:

mutating effect intent
→ dispatch
→ ACK lost
→ effect remains unresolved
→ authoritative readback
→ bounded compensation
→ compensation readback
→ fresh measured post-state

It is meant to exercise the semantics in #462 without requiring physical-specific attributes. The fixture only asks candidate conventions to preserve:

effect intent vs effect outcome;
logical effect id vs attempt id;
transport evidence vs authoritative effect evidence;
transport failure vs unresolved effect;
compensation requested vs compensation verified;
fresh measured state as the next decision origin.

If the thread converges on one or two reference scenarios before naming attributes, this can serve as the physical/irreversible-effect case or be simplified further.

I turned the earlier physical-effect reference scenario into a small runnable benchmark so this thread has something concrete to test candidate semantics against rather than only prose.

Merged artifact:
https://github.com/hippoley/AirTrajectory/pull/177

Portable vectors:
https://github.com/hippoley/AirTrajectory/blob/main/test-vectors/physical-effect-reconciliation-v0.1.json

The current cases exercise the exact boundary discussed here:

fresh measured readback can CONFIRM an effect;
fresh measured readback can CONTRADICT it;
stale readback remains UNRESOLVED;
runtime-estimated state remains UNRESOLVED;
missing observation remains UNRESOLVED;
even measured data without a measurement source remains UNRESOLVED.

The vectors replay through the same runtime reconciliation function used by the physical-control path. I am not proposing OpenTelemetry field names; this is only a reference benchmark for checking whether any eventual generic external-effect semantics preserve the distinction between transport evidence and authoritative effect evidence.

I extracted the common authority boundary behind the earlier lost-ACK / readback scenario into a small public, implementation-neutral conformance artifact:

hippoley/SpatialRuntime#13

The contract is intentionally not an OpenTelemetry attribute proposal. Its purpose is to make the execution semantics testable before deciding telemetry names.

The central invariant is:

proposal != authorization != dispatch != acknowledgement != effect evidence

The current v0.1 requirements exercise nine boundaries that seem directly relevant to items 3 and 5 here:

model/planner proposal has no mutation authority;
logical effect identity is runtime-owned;
read-only vs mutating/consequential class is known before dispatch;
intent / approval / authorization / dispatch / ACK / outcome stay distinct;
transport ACK is not effect proof;
absent/stale/contradictory authoritative evidence remains UNRESOLVED;
retries preserve one logical effect while attempts remain unique;
compensation is a distinct linked effect, and requested != verified;
fresh authoritative post-state becomes the origin for the next dependent decision.

It also includes a machine-readable manifest, downstream claim envelope, verifier, and CI tests so another runtime can map the requirement IDs to its own evidence without importing SpatialRuntime.

I think this gives #462 a second useful reference layer alongside the physical-effect vectors: the vectors test one hard scenario; the authority manifest states the generic invariants candidate observability semantics should be able to represent without conflation.

I am deliberately not proposing that these AEA names become gen_ai.* fields. If the thread converges on reference scenarios first, the manifest can instead be used as a checklist against those scenarios.

A follow-up from the physical-effect cases: I found the stronger boundary is not just "read back before retry", but whether the read-back is actually authoritative enough to resolve an external effect.

I reduced that into a small verifier profile in SpatialRuntime:
hippoley/SpatialRuntime#20

The profile keeps an effect UNRESOLVED unless the observation is bound to the same logical effect, occurs after the attempt, is fresh, and comes from a source explicitly authorized by an external policy for that effect domain / observation kind. Model estimates, transport ACKs, cached state, and self-declared "authoritative" sources do not resolve the effect.

The practical observability distinction is therefore:

external-effect span / event
  !=
authoritative effect evidence

A telemetry backend may know that a retry, compensation, or read-back happened without being able to assert that the external outcome is known. If #462 moves toward reference guidance for external effects, it may be useful to preserve that unresolved state unless the instrumentation can identify the authority basis for the observation.

I am not proposing new gen_ai.* names here; this is a verifier boundary for testing candidate semantics.

I turned the narrowest part of this discussion into a small upstream docs patch:

#588

It does not add effect-outcome attributes or ask OpenTelemetry to perform reconciliation.

The patch only clarifies how to interpret the existing execute_tool span:

it describes tool execution as observed at the instrumentation boundary;
span success / transport response / gen_ai.tool.call.result do not by themselves prove a consequential external effect is durably committed or confirmed;
if final confirmation is only observable outside that boundary, reconciliation remains outside the span's scope.

This is intentionally aligned with the repo's capturability rule: don't standardize data generic instrumentation cannot observe. The goal is to prevent telemetry consumers from promoting an observed tool result into stronger external-effect truth than the signal can support.

A follow-up from the original author, with something runnable rather than another framing.

We built the runtime side of this discussion: a reference implementation on LangGraph + Postgres
checkpoints, exporting through a Collector to Tempo/Prometheus/Loki
(oaar-reference-impl, tag v0.4.1). It
covers the durable-execution parts that #159 / #445 already address, plus the two that were left
open here: the execution-vs-outcome denominators, and external-effect evidence.

On effects I took the review of #588 at face value: a semantic note is not enough, and nothing
here needs new gen_ai.* fields. So the rule is published as thirteen implementation-neutral JSON
vectors
(vectors/).
A runtime conforms by reaching the same (status, reason) pair — no import, no attribute names,
no shared library. They cover the cases that usually get skipped: lost ACK, stale readback, no
readback at all, model estimate, digest mismatch, evidence bound to the wrong attempt or effect,
retracted evidence, untrusted verifier, pre-dispatch observation.

The concrete difference for a consumer is small and specific: a backend can leave an effect
unresolved and say why, instead of promoting a tool result or an ACK into proof that something
happened in the world.

What this does not claim: that transport semantics belong in execute_tool, that any of these
names should be standardized, or that a particular external system is authoritative. If the
thread converges on reference scenarios before choosing names, this is one, and it runs today.

Navigation Menu

Repository navigation

Proposal: observability semantics for durable agentic runtime execution #462

Summary

Problem

Proposed Direction

Scope Boundaries

Initial Reference Material

Request for Feedback

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: observability semantics for durable agentic runtime execution #462

Description

Summary

Problem

Proposed Direction

Scope Boundaries

Initial Reference Material

Request for Feedback

Activity

meshailabs commented on Aug 19, 2026

carloshenriquecarniatto commented on Aug 20, 2026

meshailabs commented on Aug 20, 2026

danyka-icam commented on Sep 24, 2026

soyeladice-svg commented on Sep 26, 2026

hippoley commented on Oct 7, 2026

hippoley commented on Oct 8, 2026

hippoley commented on Oct 8, 2026

hippoley commented on Oct 8, 2026

hippoley commented on Oct 8, 2026

hippoley commented on Oct 8, 2026

carloshenriquecarniatto commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions