Skip to content

Add gen_ai.agent.steps metric #203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pvlsirotkin wants to merge 4 commits into from
Closed

Add gen_ai.agent.steps metric #203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7fe63e4
Add gen_ai.workflow.steps metric
pvlsirotkin May 27, 2026
f0a82a3
Address review comments on PR #203
pvlsirotkin Jun 2, 2026
f4f0a4c
Refocus on per-agent step counting, rename to gen_ai.agent.steps
pvlsirotkin Jun 10, 2026
26131bc
Remove duplicate ExplicitBucketBoundaries paragraph
pvlsirotkin Jun 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
([#126](https://github.com/open-telemetry/semantic-conventions-genai/pull/126))
- Add `moonshot_ai` to `gen_ai.provider.name` well-known values.
([#99](https://github.com/open-telemetry/semantic-conventions-genai/pull/99))
- Add `gen_ai.agent.steps` metric to track the number of steps a GenAI agent
takes during a single invocation.
([#203](https://github.com/open-telemetry/semantic-conventions-genai/pull/203))

### 🧰 Bug fixes 🧰

Expand Down
77 changes: 77 additions & 0 deletions docs/gen-ai/gen-ai-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ linkTitle: Metrics
- [Metric: `gen_ai.server.time_to_first_token`](#metric-gen_aiservertime_to_first_token)
- [Generative AI workflow metrics](#generative-ai-workflow-metrics)
- [Metric: `gen_ai.workflow.duration`](#metric-gen_aiworkflowduration)
- [Generative AI agent metrics](#generative-ai-agent-metrics)
- [Metric: `gen_ai.agent.steps`](#metric-gen_aiagentsteps)

<!-- tocstop -->

Expand Down Expand Up @@ -931,6 +933,81 @@ If there is no low-cardinality workflow name available for a given framework, th
<!-- END AUTOGENERATED TEXT -->
<!-- endweaver -->

## Generative AI agent metrics

Individual systems may include additional system-specific attributes.
It is recommended to check system-specific documentation, if available.

### Metric: `gen_ai.agent.steps`

This metric is [recommended][MetricRecommended] for instrumentations that
can attribute steps to a specific agent within a single invocation.

This metric SHOULD be specified with [ExplicitBucketBoundaries] of
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024].

<!-- weaver .registry.metrics[] | select(.name == "gen_ai.agent.steps") -->
<!-- NOTE: THIS TEXT IS AUTOGENERATED. DO NOT EDIT BY HAND. -->
<!-- see templates/registry/markdown/snippet.md.j2 -->
<!-- prettier-ignore-start -->

| Name | Instrument Type | Unit (UCUM) | Description | Stability | Entity Associations |
| -------- | --------------- | ----------- | -------------- | --------- | ------ |
| `gen_ai.agent.steps` | Histogram | `{step}` | Number of steps a GenAI agent takes during a single invocation. [1] | ![Development](https://img.shields.io/badge/-development-blue) | |

**[1]:** Recorded once per agent invocation. A "step" is a framework-defined
unit of work that the agent itself performs during the invocation;
instrumentations MUST count only steps the agent itself authored.

Common framework-specific definitions:

- **ADK**: each [event](https://adk.dev/runtime/event-loop/#event) in
the invocation context whose `event.author` equals the agent name
(typically a model response, a tool call the agent issued, or an
agent transfer the agent initiated). Tool results, which are
authored by the tool rather than the agent, are not counted.
- **LangGraph**: each node execution the agent itself performed in
the workflow graph.
- **LangChain agents**: each `AgentAction` or `AgentFinish` step the
agent itself emitted.
- **CrewAI**: each agent turn the agent itself took within a task.

@lmolkova lmolkova Jun 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a bit of a research and I think this generalization would be confusing, it does not make sense to merge these things into one metric unless they are comparable:

  1. For ADK it would count LLM and tool calls, handoffs
  2. Langgraph has a notion of super-step which is "considered a single iteration over the graph nodes" (active ones). So by definition it's more than one node and step used in the metric name would be misleading. Langgraph also has configurable recursion_limit for number of super-steps and remaining_steps in the step that provides number of remaining ones until limit is reached. This is a signal that these things could be meaningful to LangGraph users.
  3. I can't find a term 'turn' in crewai, my AI found iterations in agent executor - https://github.com/crewAIInc/crewAI/blob/b6e5d632c111e098d883274f0a26f02836a2172e/lib/crewai/src/crewai/agents/crew_agent_executor.py#L218 and there is a corresponding config for max_interations, but iteration is a full ReAct loop. It has something called step_callback but does not clearly document when it's called.

I think this metric tries to capture something that's a proxy to number of loops / iterations within one agent, but tries to express it through internal frameowrk implementation details.

Some alternative approaches:

  1. Stamp agent name on the underlying llm and tool calls and derive average number of calls per agent from gen_ai.inference | execute_tool.duration and invoke agent duration. This won't give outliers or percentiles, but arguably many loops / events also leads to long invoke_agent duration.
  2. Measure gen_ai.agent.inference_calls_per_invocation, gen_ai.agent.tool_calls_per_invocations, gen_ai.agent.transfers_per_invocation - these are understandable and generically applicable. The list can probably grow with retrievals and memory calls
  3. Have specialized metrics for frameworks. [gcp.]adk.event_loop.event.count with attributes for author and type or [gcp].adk.event_loop.events_per_invocation would be specific and understandable. Langgraph and crewai would have specialized metrics too

Approaches 1 and 2 can exist together: gen_ai.agent.inference_calls_per_invocation could be a common 'give me a sense of how long this agent invocation was' that shows up on common dashboards and [gcp].adk.event_loop.events_per_invocation can be a detailed breakdown. This would align with t-shaped signals approach.


Steps emitted by sub-agents or by the surrounding workflow SHOULD
NOT be counted; each step SHOULD be recorded exactly once across
the full call tree.

Steps are counted regardless of outcome: a step that errored or
partially completed still consumed the agent's execution and SHOULD
be counted.

Step counts MUST NOT be compared across different frameworks. They
are only meaningful within the context of a single framework's
instrumentation, which is why instrumentations MUST document what
counts as a step for their framework.

**Attributes:**

| Key | Stability | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values |
| --- | --- | --- | --- | --- | --- |
| [`error.type`](https://github.com/open-telemetry/semantic-conventions/blob/v1.41.1/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If the operation ended in an error. | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` |
| [`gen_ai.agent.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | string | Human-readable name of the GenAI agent provided by the application. | `Math Tutor`; `Fiction Writer` |

**[1] `error.type`:** The `error.type` SHOULD match the error code returned by the Generative AI provider or the client library,
the canonical name of exception that occurred, or another low-cardinality error identifier.
Instrumentations SHOULD document the list of errors they report.

---

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

| Value | Description | Stability |
| --- | --- | --- |
| `_OTHER` | A fallback error value to be used when the instrumentation doesn't define a custom value. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) |

<!-- prettier-ignore-end -->
<!-- END AUTOGENERATED TEXT -->
<!-- endweaver -->

[DocumentStatus]: https://opentelemetry.io/docs/specs/otel/document-status
[MetricRequired]: https://github.com/open-telemetry/semantic-conventions/blob/v1.40.0/docs/general/metric-requirement-level.md#required
[MetricRecommended]: https://github.com/open-telemetry/semantic-conventions/blob/v1.40.0/docs/general/metric-requirement-level.md#recommended
Expand Down
43 changes: 43 additions & 0 deletions model/gen-ai/metrics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,49 @@ metrics:
- ref: gen_ai.workflow.name
requirement_level:
conditionally_required: If available.
- name: gen_ai.agent.steps
annotations:
code_generation:
metric_value_type: int
brief: >
Number of steps a GenAI agent takes during a single invocation.
note: |

@MikeGoldsmith MikeGoldsmith Jun 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we only count completed steps, or also include partial / failed steps too?

@pvlsirotkin pvlsirotkin Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New answer: tbc
Old answer: Both. The note now says: 'Steps are counted regardless of outcome: a step that errored or partially completed still consumed the agent's execution and SHOULD be counted.'

Recorded once per agent invocation. A "step" is a framework-defined
unit of work that the agent itself performs during the invocation;
instrumentations MUST count only steps the agent itself authored.

Common framework-specific definitions:

- **ADK**: each [event](https://adk.dev/runtime/event-loop/#event) in
the invocation context whose `event.author` equals the agent name
(typically a model response, a tool call the agent issued, or an
agent transfer the agent initiated). Tool results, which are
authored by the tool rather than the agent, are not counted.
- **LangGraph**: each node execution the agent itself performed in
the workflow graph.
- **LangChain agents**: each `AgentAction` or `AgentFinish` step the
agent itself emitted.
- **CrewAI**: each agent turn the agent itself took within a task.

Steps emitted by sub-agents or by the surrounding workflow SHOULD
NOT be counted; each step SHOULD be recorded exactly once across
the full call tree.

Steps are counted regardless of outcome: a step that errored or
partially completed still consumed the agent's execution and SHOULD
be counted.

Step counts MUST NOT be compared across different frameworks. They
are only meaningful within the context of a single framework's
instrumentation, which is why instrumentations MUST document what
counts as a step for their framework.
instrument: histogram
Comment thread
pvlsirotkin marked this conversation as resolved.
unit: "{step}"
stability: development
attributes:
Comment thread
pvlsirotkin marked this conversation as resolved.
- ref_group: attributes.gen_ai.error
- ref: gen_ai.agent.name
requirement_level: recommended

metric_refinements:
- id: openai.client.token.usage
Expand Down
149 changes: 149 additions & 0 deletions schema-snapshot/registry.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading