Repository navigation
Add async/long-running agent lifecycle events: gen_ai.agent.paused / .resumed / .checkpointed #159
Description
TL;DR
Propose adding standard span events for long-running async agent lifecycles: gen_ai.agent.paused, .resumed, .checkpointed, plus supporting .started, .completed, .failed. These complement #55's gen_ai.task.status attribute (which captures final state) by capturing timing of mid-execution state transitions — critical for HITL, scheduled, and durable agents.
Filed separately from #55 per @Cirilla-zmh's suggestion that this is beyond #55's scope. Full background in the original gist.
Why it's distinct from gen_ai.task.status
#55's gen_ai.task.status (values: pending / in_progress / completed / failed) is useful as a final-state marker on a task span. It doesn't capture when or why an agent transitioned mid-execution: when it paused for human input, when it persisted a checkpoint, when it resumed.
For short-lived workflow execution, the status attribute is enough. For long-running, human-in-the-loop, scheduled, or durable agents — increasingly common as agents move to async execution patterns (see Zak Knill, "All Your Agents Are Going Async", 2026) — timing matters too. An agent that paused for 3 hours waiting on human approval needs observability on when and why it paused, not just whether it ultimately completed.
Adjacent issues that don't cover this
- Generic Grouping Attributes for Agentic Workflow Spans #94 (Generic Grouping Attributes) — logical grouping across spans, not lifecycle state transitions
- Adding ReAct Iterations Spans in Reasoning-Acting Agents #81 (ReAct Iterations Spans) — iteration counting within a reasoning loop, not pause/resume
- Propose semantic conventions for agentic AI failure repair (gen_ai.repair.*) #89 (
gen_ai.repair.*) — post-detection failure remediation, not in-flight lifecycle
Proposed events
Span events that long-running agents MAY emit, nested inside a gen_ai.workflow, gen_ai.agent, or gen_ai.task span:
| Event | Description | Stability |
|---|---|---|
gen_ai.agent.started |
Agent execution has begun; resources loaded | Development |
gen_ai.agent.paused |
Agent has paused execution awaiting external input | Development |
gen_ai.agent.resumed |
Agent has resumed execution after a pause | Development |
gen_ai.agent.checkpointed |
Agent state has been persisted for durability/recovery | Development |
gen_ai.agent.completed |
Agent execution ended successfully | Development |
gen_ai.agent.failed |
Agent execution ended in failure requiring attention | Development |
Suggested event attributes
| Attribute | Description | Type |
|---|---|---|
gen_ai.agent.pause.reason |
Why execution paused (human_input, external_system, rate_limit, scheduled_delay) |
string |
gen_ai.agent.checkpoint.id |
Identifier for the persisted checkpoint, for recovery correlation | string |
gen_ai.agent.execution.id |
Identifier for a single end-to-end execution run (can differ from gen_ai.agent.id when the same agent runs repeatedly) |
string |
Example: nightly reconciliation agent
span: gen_ai.agent invoke_agent reconciliation-agent [4h duration]
event: gen_ai.agent.started (t+0s)
span: execute_tool fetch_invoices (t+5s)
span: chat claude-sonnet-4-7 (t+12s)
event: gen_ai.agent.checkpointed (checkpoint.id=ck_001) (t+900s)
event: gen_ai.agent.paused (reason=human_input) (t+1800s)
event: gen_ai.agent.resumed (t+14000s, 3h later)
span: execute_tool post_journal_entry (t+14010s)
event: gen_ai.agent.completed (t+14200s)
Without these events, reconstructing "what was the agent doing from t+900s to t+14200s" requires custom instrumentation.
Open question for the SIG
Should these be events on the existing invoke_agent span (simpler), or should long-running async agents get a new span kind entirely (more honest about the different execution model)?
Implementation status
MeshAI Labs has implemented these as custom attributes in our platform. Happy to:
- Refine into a formal spec PR once the SIG indicates direction
- Contribute a reference implementation showing the attributes across CrewAI / AutoGen / LangGraph
cc @Cirilla-zmh per #55 redirect
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsPost-stability
Activity
Rul1an commentedon May 18, 2026
This distinction from final
gen_ai.task.statusmakes sense to me. Final state and mid-execution lifecycle transitions answer different questions.One thing worth making explicit before the event shape settles: the pause -> checkpoint -> resume transitions need to be correlatable across process and time boundaries, not just visible inside one trace. A span event is attached to whatever span is open when it fires, so the correlation has to live in the attributes the events carry, not in span structure alone.
For short in-process runs, events on an existing
invoke_agentspan may be enough. But the cases this proposal targets are the hard ones: human approval after hours, scheduled delay, durable recovery, or a process restart. In those cases the original span may not still be open, and resumed work may land in a new span or even a new trace.So I’d treat
gen_ai.agent.execution.idas the logical run id, and make the transition edges explicit:checkpointedcarriescheckpoint.idpausedcarries a stable pause or interruption id when external resolution is expectedresumedcarries the checkpoint id or pause id it resumed fromThat gives downstream consumers enough to reconstruct: this execution paused here, persisted this state, and later resumed from that exact boundary.
Without that edge,
resumedis observable but weaker. A consumer can tell that something resumed, but not always which suspended state it continued from.This also affects the events-vs-span-kind question. I wouldn’t require one long span to cover the whole async lifecycle. A segment-per-active-execution model with stable lifecycle ids or span links may be more honest for durable agents, while the same event names still work for the simple in-process case.
meshailabs commentedon May 18, 2026
+1 on
gen_ai.agent.execution.idas the logical run id — agree the correlation has to live in attributes, not span structure. When we instrumented agentic tool hooks via OpenTelemetry, we hit the same gap. Thecheckpoint.id/pause.idcarry-through on the transition events is the right shape: a consumer can reconstruct the suspended → resumed boundary even when spans aren't continuous.The segment-per-execution model belongs in the spec text explicitly. Treating
invoke_agentas one continuous span breaks the moment you have a multi-hour pause.Happy to draft YAML for the attribute additions if useful.
Keesan12 commentedon May 23, 2026
The segment-per-active-execution model feels right for the durable cases this proposal targets. I would make the edge fields required whenever an event crosses a suspend/resume boundary.
A small invariant could be: a resumed segment MUST identify the suspended boundary it resumes from, either by
checkpoint.id,pause.id, or an equivalent lifecycle transition id. Then an observability backend can reconstruct the chain even if the original span is closed, sampled out, or exported in a different trace.That also gives implementations a clean test: pause -> checkpoint -> process restart -> resume should produce two active execution segments with a shared logical execution id and an explicit resumed-from edge. Without that, the event names are useful for timelines but weaker for audit and recovery.
Rul1an commentedon May 23, 2026
I agree with this invariant. One nuance I’d keep in the wording: the required edge should be a resumed-from relationship, not necessarily a checkpoint-specific relationship.
For durable recovery,
checkpoint.idis the natural edge. For human approval or scheduled delay, the suspended boundary may be apause.id,interruption.id, or more generic lifecycle transition id. The important rule is thatresumednames the boundary it continues from, and the same logicalgen_ai.agent.execution.idties the active segments together.That gives the spec a clean conformance test without forcing every implementation to manufacture a checkpoint when it only paused. Something like:
gen_ai.agent.resumedMUST identify the prior suspended boundary it resumes fromThat feels like the durable-agent minimum: enough structure to rebuild the chain, without requiring one long span or one storage model.
meshailabs commentedon May 28, 2026
I've drafted the full semconv model encoding the convergence from this thread:
gen_ai.agent.execution.idas the logical run id, plus @Keesan12's conformance invariant (a resumed segment must name the boundary it continues from) generalized per @Rul1an into agen_ai.agent.resumed_fromedge that takes acheckpoint.id, apause.id, or any equivalent transition id, so human-approval / scheduled-delay pauses that never checkpoint still satisfy the rule.The load-bearing decision is the requirement level:
resumed_fromis conditionally_required only when the resumed work starts a new active execution segment (process restart, durable recovery, different trace), and MAY be omitted when resumption continues in the same still-open span. That keeps the short-lived in-process case zero-overhead while making the durable case auditable.Rather than drop the whole model inline, two open questions for the SIG:
gen_ai.agent.execution.idvsgen_ai.agent.run.id? Want to align with any existing run/execution convention in gen_ai.resumed_fromas a single string ref, or a structured{type, id}pair?I can paste the full YAML here or open it as a draft PR, whichever the SIG prefers. cc @Cirilla-zmh
StevenMih commentedon Aug 7, 2026
+1 on this shape and on where the thread converged (gen_ai.agent.execution.id + a resumed_from edge — the correlation has to survive process boundaries, since durable interrupts routinely resume in a different worker, or not at all).
Two additions from implementing exactly this:
paused/resumed assumes every pause eventually resumes. The two other endings are the ones oversight consumers care about most: refused (the human said no) and expired (nobody answered before the deadline). Both are terminal resolutions of the pause, not absences of a resume, and both need first-class representation — a refusal that's indistinguishable from an error, or an expiry indistinguishable from a still-pending pause, is wrong in both directions for anyone consuming this for governance. Concretely: a resolution event (or final state) with approved / refused / expired, linked to the pause it resolves.
An interrupted execution is a dispatch record, not a failure. Whatever status vocabulary eventually lands (here, or in Add Workflows/Agents/Tasks to gen_ai #55's gen_ai.task.status, or Add semantic conventions for A2A (Agent-to-Agent) protocol #70's A2A states — they should agree), interrupted must be distinct from failed: the semantics are "dispatched, awaiting a resolution that arrives elsewhere," and vendor extensions are already shipping interrupted/cancelled values ahead of the spec, so a vocabulary that can't express this gets forked in practice on day one.
This is framework-neutral now — durable approval mechanics with exactly these outcomes (approve/refuse/expire) have shipped in LangGraph, the OpenAI Agents SDK, Google ADK, and Microsoft Agent Framework within the last year, so the conventions can bind to the common shape rather than any single framework's.
meshailabs commentedon Aug 7, 2026
+1 to both additions. From the oversight-consumer side, refused and expired are not edge cases, they are the outcomes a reviewer of the telemetry most needs to tell apart. The two failure modes you name are real and asymmetric: a refusal recorded as an error corrupts error-rate metrics and understates governance activity, while an expiry indistinguishable from a pending pause hides that the approval channel itself is broken. Those are different problems with different owners, and we compute oversight metrics from exactly this distinction in production.
The resolution event also closes the state machine this thread converged on: every pause ends in exactly one of resumed, refused, or expired.
resumed_fromthen only ever exists on the approved path, and refused/expired terminate the logical execution at the suspended boundary without starting a new segment. That keeps the segment model intact and gives conformance a clean invariant: a pause with no resolution is pending, full stop.Agree the status vocabulary should be settled once and shared with #55 and #70, with interrupted distinct from failed.
Rul1an commentedon Aug 7, 2026
+1 on @StevenMih's refused/expired split, and on @meshailabs' framing of the two as asymmetric in what they corrupt. Agreement, nothing to add there.
On the two open questions.
2, the
resumed_fromshape. Carry the type, but as two flat attributes rather than a nested pair:gen_ai.agent.resumed_from.typealongsidegen_ai.agent.resumed_from.id.Carrying the type at all is the same reason the edge stopped being checkpoint-specific. The suspended boundary can be a durable checkpoint, a pause awaiting human approval, or a scheduled delay, and those are not interchangeable to whoever reads the record later. With a bare id a consumer learns which kind it is holding only by resolving it against the event that emitted it, and this thread's whole premise is that the counterpart may be closed, sampled out, or in a different trace. Requiring resolution to learn the type gives back exactly what the edge was introduced to protect.
Flat rather than a complex value because the authoring guidance is to represent complex values as a set of flat attributes whenever possible, and to assume backends do not index their individual properties. The eight
any-typed attributes inmodel/gen-ai/registry.yamltoday are all bulk content payloads with a JSON schema behind each, which a two-field reference does not need.gen_ai.tool.typesitting next togen_ai.tool.call.idis already this shape. It also keeps "group resumptions by boundary type" an ordinary query.1, naming. I went looking for the convention you wanted to align with and there is no run or execution attribute in the gen_ai registry at all, so nothing constrains the choice. The nearest pattern is
<noun>.id:gen_ai.conversation.id,gen_ai.agent.id,gen_ai.memory.store.id. Both candidates fit it.execution.idreads more precisely to me, though only weakly, and matching whatever the SIG settles on matters more than the choice.A draft PR would be easier to review than inline YAML, and I am happy to review either. One case I would rather work through against concrete YAML than in the abstract:
expiredcovers the deadline passing, but a producer that dies while holding the pause emits nothing at all, so "no resolution recorded" ends up with more than one cause. Whether that belongs in the conventions or is left to the producer is the part worth arguing against a real requirement level.StevenMih commentedon Aug 9, 2026
@meshailabs , agreed on all of it, and "a pause with no resolution is pending, full stop" is the invariant I'd want in the conventions text verbatim. The asymmetry you name (refusal corrupts error-rate metrics; expiry hides a broken approval channel — different problems, different owners) is the best one-paragraph case for the split I've seen; better than mine.
@Rul1an , with you on the flat pair (gen_ai.agent.resumed_from.type + .id), for your the reasons given: the counterpart event may be closed, sampled out, or in another trace, and an edge you must resolve to interpret gives back what the edge exists to protect. No strong view on execution.id vs alternatives — matching whatever the SIG settles is worth more than the choice.
On the producer that dies holding the pause: I'd keep expired strictly as a deadline passed and a resolution event says so — a record of absence — and never let consumers infer it from silence, which is absence of record. They're different facts with different owners: an expiry is a governance outcome; silence is an availability problem, and collapsing them recreates the exact ambiguity the resolution event removes. Concretely: a producer SHOULD emit the expiry resolution when a deadline exists, MUST NOT emit expired without one, and what unresolved silence means stays consumer policy rather than vocabulary. Worth working through against real YAML, agreed.
On the draft PR: @meshailabs, it's your issue — if you'd like to author it, I'll review same-day. If it's more useful the other way, I'm happy to draft the YAML plus a reference scenario (the three events, the resolution event with approved / refused / expired, and the flat resumed_from pair as discussed) and you both review. Either way this looks ready for concrete YAML.
meshailabs commentedon Aug 9, 2026
Honored to take it, and thank you both for the design work that got this thread to convergence. The draft PR is up: #445.
It encodes what we settled on: the three lifecycle events,
gen_ai.agent.pause.resolvedwith approved / refused / expired, the flatresumed_from.type+resumed_from.idpair recorded together or not at all, and the deadline rule for expired exactly as stated above (a record of absence: SHOULD with a configured deadline, MUST NOT without one, and what silence means stays consumer policy). The pending invariant sits in thegen_ai.agent.pause.resolutiondefinition itself. A framework-neutral reference scenario models a durable approval gate and exercises all three outcomes, including an approved resume in a new execution segment after the original span closed.@StevenMih the same-day review offer is gratefully accepted. @Rul1an your flat-pair reasoning is close to verbatim in the attribute note, so please check I did it justice. Scope notes in the PR: started/completed/failed stay out (the invoke_agent span already expresses them), and the status vocabulary question stays with #55 and #70 as discussed.
meshailabs commentedon Aug 13, 2026
Cross-referencing a downstream consumer that has appeared since the PR went up: OpenSecureAIAlliance/wg-safe#13, part of the SAFE incident-evidence work, cites these lifecycle events as the natural carrier for the approval moment in machine-verifiable approval-to-execution binding. Its four-outcome problem statement (no authority existed, execution stayed in scope, execution left scope, evidence insufficient to tell) depends on being able to join an approval to the pause it resolved, which is exactly the pause.id linkage #445 proposes. Noting it here as concrete external demand for the convention while the PR is in review.
SpaceBum9 commentedon Aug 23, 2026
Implementation note from MCT-2600027: while drafting a W3C/OpenTelemetry adapter, we hit a distinction that may be useful for this lifecycle proposal.
A durable checkpoint has at least two identifiers with different semantics:
Neither should be reused as a W3C
trace-idorparent-id. In our current implementation the checkpoint digest is 64-hex SHA-256, while W3Ctrace-idis 32 hex andparent-idis a 16-hex span ID. Beyond the size mismatch, the semantics differ: physical span causality is not artifact lineage.The interop shape we plan to test is therefore:
gen_ai.agent.execution.idremains stable across active execution segments;A minimal negative fixture would cover a missing resume edge, a changed checkpoint digest, an unknown canonicalization profile, and a checkpoint that exists but has not been externally verified.
We documented the proposed MCT mapping and non-claims in MCT issue #2. PR #1 contains the deterministic offline checkpoint/evaluation foundation; it is not an OpenTelemetry conformance claim.
Would the SIG prefer a standard checkpoint content-hash attribute beside the correlation ID, or should integrity metadata remain implementation-specific with only the checkpoint ID standardized?
meshailabs commentedon Aug 24, 2026
Thank you, the two-identifier distinction is worth making explicit.
On your question: I would standardize the correlation id only, for now.
gen_ai.agent.checkpoint.idin #445 names the boundary a resume continued from, and it is deliberately not trace context, so your size and semantics point holds there. A resumed segment links to the prior span rather than reusing either value astrace-idorparent-id.A checkpoint content hash reads as a separate proposal rather than an addition to this one. The guidance here is that an attribute needs a signal that carries it and a reference scenario in a library that owns the value, and integrity metadata has neither today. Your three states, existence, signature, and verified verdict, also point to more than one attribute, which is easier to settle on its own thread than folded into this one.
If a library exposes a canonical digest at its checkpoint API, that would be the natural place to start.