Repository navigation
Span + metric definitions principles for agents, inference, control plane #249
Description
Capturing discussions and proposals inspired by #215 #203 #201
Current span/metric/operation name definitions
| Span Type | Duration metric | gen_ai.operation.name values |
|---|---|---|
| gen_ai.inference.client | gen_ai.client.operation.duration |
chat, generate_content, text_completion |
| gen_ai.embeddings.client | gen_ai.client.operation.duration |
embeddings |
| gen_ai.retrieval.client | gen_ai.client.operation.duration |
retrieval |
| gen_ai.memory.client | gen_ai.client.operation.duration |
create_memory_store, search_memory, create_memory, update_memory, upsert_memory, delete_memory, delete_memory_store |
| gen_ai.create_agent.client | gen_ai.client.operation.duration |
create_agent |
| gen_ai.invoke_agent.client | gen_ai.client.operation.duration |
invoke_agent |
| gen_ai.invoke_agent.internal | gen_ai.client.operation.duration |
invoke_agent |
| gen_ai.execute_tool.internal | gen_ai.client.operation.duration |
execute_tool |
| gen_ai.invoke_workflow.internal | gen_ai.workflow.duration |
invoke_workflow |
| gen_ai.plan.internal | gen_ai.client.operation.duration |
plan |
This table is highly controversial and the modeling is problematic:
- does
gen_ai.client.operation.durationcapture tool calling or internal agents? is it documented? - one metric definition for very different operations (not much in common between tool calling and memory updates)
Proposal
Naming
Span and metric names are built from the same segments so they stay aligned:
span = gen_ai.<operation-class>.<kind># partially enforced by semconv policies, this is not yet visible on OTLP, can be changedmetric = gen_ai.[client.]<operation-class>.<measured-thing>
Rules:
- First segment is always
gen_ai. - The kind is encoded by a single
clientfragment: present forclientspans, omitted forinternal. There is nointernal/serverfragment. <operation-class>is identical in the span and the metric, and names an action. It is either:- a nominalized verb when the action stands alone (
inference,embeddings,retrieval), or <verb>_<target>when the action must be qualified by what it acts on (invoke_agent,execute_tool,manage_agent,search_memory).
- a nominalized verb when the action stands alone (
- The last metric segment names the measured quantity (
duration, ...).
Grouping
- A metric/span per
gen_ai.operation.namevalue would not be usable, some grouping is needed - Span definitions and metrics are aligned (1:1 relationship between span definition and duration metric definition)
- Management operations are grouped per resource under a
manage_<resource>class (one metric/span definition per resource type, because resources have different attributes describing them). - Grouping is based on operations having a similar attribute set and a comparable duration range (otherwise the histogram is meaningless - even for exponential buckets).
- Important runtime operations get an individual metric and span definition.
| Span Type | Duration metric | gen_ai.operation.name values |
|---|---|---|
| gen_ai.inference.client | gen_ai.client.inference.duration |
chat, generate_content, text_completion, compaction |
| gen_ai.embeddings.client | gen_ai.client.embeddings.duration |
embeddings |
| gen_ai.retrieval.client | gen_ai.client.retrieval.duration |
retrieval |
| gen_ai.search_memory.client | gen_ai.client.search_memory.duration |
search_memory |
| gen_ai.manage_memory.client | gen_ai.client.manage_memory.duration |
create_memory, update_memory, upsert_memory, delete_memory |
| gen_ai.manage_memory_store.client | gen_ai.client.manage_memory_store.duration |
create_memory_store, delete_memory_store |
| gen_ai.manage_agent.client | gen_ai.client.manage_agent.duration |
create_agent, update_agent, delete_agent |
| gen_ai.invoke_agent.client | gen_ai.client.invoke_agent.duration |
invoke_agent |
| gen_ai.invoke_agent.internal | gen_ai.invoke_agent.duration |
invoke_agent |
| gen_ai.invoke_workflow.internal | gen_ai.invoke_workflow.duration |
invoke_workflow |
| gen_ai.execute_tool.internal | gen_ai.execute_tool.duration |
execute_tool |
Notes:
- Per offline discussion with @trask and @aabmass we had split memory into memory records operations (search, CRUD) and memory store CRUD, but I consulted with AI and it pointed out that memory search is different from memory CRUD. It aligns with retrieval being separate operations. /cc @nagkumar91
gen_ai.workflow.durationto be renamed togen_ai.invoke_workflow.durationto match the pattern, we didn't release the current one yet. /cc @wrisa
Future
Operations that are not yet defined, following the same naming and grouping rules they might look like this:
| Span Type | Duration metric | gen_ai.operation.name values |
|---|---|---|
| gen_ai.manage_data_store.client | gen_ai.client.manage_data_store.duration |
create_data_store, update_data_store, delete_data_store |
| gen_ai.reasoning.internal | gen_ai.reasoning.duration |
reason, think, plan |
| gen_ai.execute_step.internal | gen_ai.execute_step.duration |
execute_step, invoke_node, run_task |
| gen_ai.apply_guardrail.client | gen_ai.client.apply_guardrail.duration |
apply_guardrail |
| gen_ai.apply_guardrail.internal | gen_ai.apply_guardrail.duration |
apply_guardrail |
| gen_ai.load_skill.internal | gen_ai.load_skill.duration |
load_skill |
| gen_ai.manage_skill.client | gen_ai.manage_skill.duration |
create_skill, etc |
| gen_ai.evaluate.client | gen_ai.client.evaluate.duration |
evaluate |
| gen_ai.evaluate.internal | gen_ai.evaluate.duration |
evaluate |
Activity
AgentGymLeader commentedon Jun 4, 2026
This split looks directionally useful to me, especially separating inference, agent invocation, tool execution, and future control-plane operations such as guardrails/evaluation.
One small thing I would preserve in the principles is that control-plane operations should not collapse back into inference metrics just because they happen during the same agent run. They often have different attributes, duration ranges, and downstream consumers.
If future guardrail/evaluation spans need to connect to review, audit, or incident workflows, I would keep that as opaque producer-defined span attributes or span events rather than metric labels, and avoid standardizing any external evidence record format here. That seems consistent with keeping metrics aggregatable while still leaving a join point for systems that need one.
Nik-Reddy commentedon Jun 9, 2026
This direction makes sense to me.
I like the principle that span definitions and duration metrics should align by operation class, instead of using one broad
gen_ai.client.operation.durationmetric for unrelated work.The client vs internal split also seems important for agent telemetry. For example,
invoke_agent.clientandinvoke_agent.internalhave different operation boundaries, so separate aligned duration metrics would make dashboards clearer.This also lines up with the
gen_ai.agent.finish_reasonwork in #171 / #238: the finish reason should be attached to the agent invocation span whose lifecycle ended, and the duration metric should describe that same boundary.For #215, I think the current clarification is useful as a near-term fix. Longer term, this issue seems like the right place to define the more precise metric family and migration path.
Nik-Reddy commentedon Jun 9, 2026
One additional scope point: I would keep the first pass focused on operation classes that already have concrete SDK/reference-scenario boundaries, rather than trying to model every framework concept as a top-level span.
The initial stable set seems to be:
invoke_agentexecute_toolinvoke_workflowplanmanage_agentapply_guardrailThat keeps the model grounded in observable boundaries and avoids overfitting too early to framework-specific concepts like crews, teams, tasks, branches, checkpoints, or sessions.
#238 and #267 are not trying to solve the full span/metric taxonomy from #249. They are focused follow-ups for the
invoke_agentoperation: once an agent invocation span exists,gen_ai.agent.finish_reasonexplains why that specific invocation ended.I can help with small reference-backed follow-ups, especially around
invoke_agentfinish reasons (#171 / #238 / #267), guardrail boundaries, and how handoff should relate to the A2A work.trask commentedon Jun 10, 2026
@AgentGymLeader I noticed that some of your comments on other issues bring the discussion back to ideas from #239. Even when the connection is relevant, it can make those issue threads harder to follow because the discussion shifts away from the specific topic of the issue.
We appreciate your involvement. Please keep comments focused on the issue at hand, and use #239 for broader discussion of that proposal. Thanks!
Address review comments on PR open-telemetry#201 (round 2)
AgentGymLeader commentedon Jun 12, 2026
@trask thanks, that's fair. Going forward I'll keep my comments on the specific issue they're in, and take any broader discussion of #239 to the #239 thread itself. Sorry for making some threads harder to follow — I'll watch that.
Address review comments on PR open-telemetry#201 (round 2)
Add gen_ai.invoke_agent.duration and gen_ai.execute_tool.duration met…
lmolkova commentedon Jun 24, 2026
Feedback: we should have an idea about its connection to MCP metrics (
mcp.client.operation.durationhas the same problem but also conflicts with potentialgen_ai.client.execute_tool.duration)[Update] Possibly we should not add
gen_ai.client.execute_tool.duration- there is just one standard remote tool calling protocol and there is no need to generalize the metric