Skip to content

Commit cb5a8d8

Browse files
RKestRKest
committed
Replace gen_ai.agent.steps with per-invocation call-count metrics
Following review feedback on open-telemetry#203, the single gen_ai.agent.steps metric conflated things that are not comparable across frameworks (LLM calls, tool calls, handoffs, framework-specific 'super-steps'/'iterations'), which made the 'step' unit misleading and framework-dependent. This replaces it with two understandable, generically-applicable per-invocation metrics: * gen_ai.invoke_agent.inference_calls (histogram, {inference_call}): number of inference (model) calls an agent makes during a single invocation. * gen_ai.invoke_agent.tool_calls (histogram, {tool_call}): number of tool calls an agent makes during a single invocation. Both are dimensioned by gen_ai.agent.name (recommended) and live in the gen_ai.invoke_agent.* namespace alongside gen_ai.invoke_agent.duration. Counts are scoped to a single invocation and counted exactly once across the call tree (sub-agent/transferred calls belong to those agents' invocations). Failed and partial calls still count. Both SHOULD be emitted together with the gen_ai.invoke_agent.internal span. A transfers/handoffs count (also suggested in the review) is intentionally omitted: it is not a good measure of an agent's workflow complexity. The list can still grow later (e.g. retrievals, memory calls).
1 parent 4831c65 commit cb5a8d8

3 files changed

Lines changed: 163 additions & 2 deletions

File tree

‎changelog.d/336.enhancement.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add `gen_ai.invoke_agent.inference_calls` and `gen_ai.invoke_agent.tool_calls` metrics to track how many inference (model) and tool calls a GenAI agent makes during a single invocation.

‎docs/gen-ai/gen-ai-metrics.md‎

Lines changed: 98 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ linkTitle: Metrics
2121
- [Metric: `gen_ai.workflow.duration`](#metric-gen_aiworkflowduration)
2222
- [Generative AI agent metrics](#generative-ai-agent-metrics)
2323
- [Metric: `gen_ai.invoke_agent.duration`](#metric-gen_aiinvoke_agentduration)
24+
- [Metric: `gen_ai.invoke_agent.inference_calls`](#metric-gen_aiinvoke_agentinference_calls)
25+
- [Metric: `gen_ai.invoke_agent.tool_calls`](#metric-gen_aiinvoke_agenttool_calls)
2426
- [Generative AI tool metrics](#generative-ai-tool-metrics)
2527
- [Metric: `gen_ai.execute_tool.duration`](#metric-gen_aiexecute_toolduration)
2628

@@ -977,7 +979,7 @@ the metric value SHOULD be the same as the span duration.
977979

978980
| Key | Stability | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values |
979981
| --- | --- | --- | --- | --- | --- |
980-
| [`error.type`](https://github.com/open-telemetry/semantic-conventions/blob/v1.41.1/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If the operation ended in an error. | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` |
982+
| [`error.type`](https://github.com/open-telemetry/semantic-conventions/blob/v1.42.0/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If the operation ended in an error. | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` |
981983
| [`gen_ai.agent.id`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` When available. | string | The unique and stable identifier of the GenAI hosted agent resource. [2] | `asst_5j66UpCpwteGg4YSxUnt7lPY`; `arn:aws:bedrock:us-east-1:123:agent/42`; `urn:agent:projects-123:projects:123:locations:us-east1:aiplatform:reasoningEngines:456` |
982984
| [`gen_ai.agent.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` When available. | string | Human-readable name of the GenAI agent provided by the application. | `Math Tutor`; `Fiction Writer` |
983985
| [`gen_ai.agent.version`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` When available. | string | The version of the GenAI agent. | `1.0.0`; `2025-05-01` |
@@ -1001,6 +1003,100 @@ It's NOT RECOMMENDED to record in-memory agent instance ids on this attribute du
10011003
<!-- END AUTOGENERATED TEXT -->
10021004
<!-- endweaver -->
10031005

1006+
### Metric: `gen_ai.invoke_agent.inference_calls`
1007+
1008+
This metric is [recommended][MetricRecommended] for instrumentations of agent
1009+
frameworks that can reliably attribute inference (model) calls to a single
1010+
agent invocation.
1011+
1012+
This metric SHOULD be specified with [ExplicitBucketBoundaries] of
1013+
[1, 2, 4, 8, 16, 32, 64, 128].
1014+
1015+
<!-- weaver .registry.metrics[] | select(.name == "gen_ai.invoke_agent.inference_calls") -->
1016+
<!-- NOTE: THIS TEXT IS AUTOGENERATED. DO NOT EDIT BY HAND. -->
1017+
<!-- see templates/registry/markdown/snippet.md.j2 -->
1018+
<!-- prettier-ignore-start -->
1019+
1020+
| Name | Instrument Type | Unit (UCUM) | Description | Stability | Entity Associations |
1021+
| -------- | --------------- | ----------- | -------------- | --------- | ------ |
1022+
| `gen_ai.invoke_agent.inference_calls` | Histogram | `{inference_call}` | The number of inference (model) calls a GenAI agent makes during a single invocation. [1] | ![Development](https://img.shields.io/badge/-development-blue) | |
1023+
1024+
**[1]:** This metric counts the inference calls (calls to a language or other
1025+
generative model) that an agent issues while handling a single
1026+
invocation. It gives operators a generic, framework-independent sense
1027+
of how "chatty" an agent invocation is with the underlying model,
1028+
which is a common proxy for cost and latency and is useful for
1029+
capacity planning and regression detection across agent versions.
1030+
1031+
The count is scoped to a single agent invocation and SHOULD include
1032+
only the inference calls the agent itself issues; calls made by
1033+
sub-agents or transferred-to agents are recorded against those agents'
1034+
own invocations so that each inference call is counted exactly once
1035+
across the call tree.
1036+
1037+
Inference calls that fail or return partial results are still counted,
1038+
because they consumed the agent's execution.
1039+
1040+
This metric SHOULD be emitted together with the
1041+
`gen_ai.invoke_agent.internal` span for the same invocation.
1042+
1043+
**Attributes:**
1044+
1045+
| Key | Stability | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values |
1046+
| --- | --- | --- | --- | --- | --- |
1047+
| [`gen_ai.agent.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | string | Human-readable name of the GenAI agent provided by the application. | `Math Tutor`; `Fiction Writer` |
1048+
1049+
<!-- prettier-ignore-end -->
1050+
<!-- END AUTOGENERATED TEXT -->
1051+
<!-- endweaver -->
1052+
1053+
### Metric: `gen_ai.invoke_agent.tool_calls`
1054+
1055+
This metric is [recommended][MetricRecommended] for instrumentations of agent
1056+
frameworks that can reliably attribute tool calls to a single agent
1057+
invocation.
1058+
1059+
This metric SHOULD be specified with [ExplicitBucketBoundaries] of
1060+
[1, 2, 4, 8, 16, 32, 64, 128].
1061+
1062+
<!-- weaver .registry.metrics[] | select(.name == "gen_ai.invoke_agent.tool_calls") -->
1063+
<!-- NOTE: THIS TEXT IS AUTOGENERATED. DO NOT EDIT BY HAND. -->
1064+
<!-- see templates/registry/markdown/snippet.md.j2 -->
1065+
<!-- prettier-ignore-start -->
1066+
1067+
| Name | Instrument Type | Unit (UCUM) | Description | Stability | Entity Associations |
1068+
| -------- | --------------- | ----------- | -------------- | --------- | ------ |
1069+
| `gen_ai.invoke_agent.tool_calls` | Histogram | `{tool_call}` | The number of tool calls a GenAI agent makes during a single invocation. [1] | ![Development](https://img.shields.io/badge/-development-blue) | |
1070+
1071+
**[1]:** This metric counts the tool executions an agent triggers while
1072+
handling a single invocation. It gives operators a generic,
1073+
framework-independent sense of how tool-heavy an agent invocation is,
1074+
which is useful for capacity planning, regression detection across
1075+
agent versions, and diagnosing pathological tool loops (an agent stuck
1076+
calling tools without making progress).
1077+
1078+
The count is scoped to a single agent invocation and SHOULD include
1079+
only the tool calls the agent itself triggers; calls made by
1080+
sub-agents or transferred-to agents are recorded against those agents'
1081+
own invocations so that each tool call is counted exactly once across
1082+
the call tree.
1083+
1084+
Tool calls that fail or return partial results are still counted,
1085+
because they consumed the agent's execution.
1086+
1087+
This metric SHOULD be emitted together with the
1088+
`gen_ai.invoke_agent.internal` span for the same invocation.
1089+
1090+
**Attributes:**
1091+
1092+
| Key | Stability | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values |
1093+
| --- | --- | --- | --- | --- | --- |
1094+
| [`gen_ai.agent.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | string | Human-readable name of the GenAI agent provided by the application. | `Math Tutor`; `Fiction Writer` |
1095+
1096+
<!-- prettier-ignore-end -->
1097+
<!-- END AUTOGENERATED TEXT -->
1098+
<!-- endweaver -->
1099+
10041100
## Generative AI tool metrics
10051101

10061102
Individual systems may include additional system-specific attributes.
@@ -1034,7 +1130,7 @@ the metric value SHOULD be the same as the span duration.
10341130
| Key | Stability | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Value Type | Description | Example Values |
10351131
| --- | --- | --- | --- | --- | --- |
10361132
| [`gen_ai.tool.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Required` | string | Name of the tool utilized by the agent. | `Flights` |
1037-
| [`error.type`](https://github.com/open-telemetry/semantic-conventions/blob/v1.41.1/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If the operation ended in an error. | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` |
1133+
| [`error.type`](https://github.com/open-telemetry/semantic-conventions/blob/v1.42.0/docs/registry/attributes/error.md) | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | `Conditionally Required` If the operation ended in an error. | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` |
10381134
| [`gen_ai.agent.name`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` When available. | string | Human-readable name of the GenAI agent provided by the application. | `Math Tutor`; `Fiction Writer` |
10391135
| [`gen_ai.tool.type`](/docs/registry/attributes/gen-ai.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Recommended` | string | Type of the tool utilized by the agent [2] | `function`; `extension`; `datastore` |
10401136

‎model/gen-ai/metrics.yaml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,70 @@ metrics:
154154
- ref: gen_ai.agent.version
155155
requirement_level:
156156
conditionally_required: When available.
157+
- name: gen_ai.invoke_agent.inference_calls
158+
annotations:
159+
code_generation:
160+
metric_value_type: int
161+
brief: >
162+
The number of inference (model) calls a GenAI agent makes during a
163+
single invocation.
164+
note: |
165+
This metric counts the inference calls (calls to a language or other
166+
generative model) that an agent issues while handling a single
167+
invocation. It gives operators a generic, framework-independent sense
168+
of how "chatty" an agent invocation is with the underlying model,
169+
which is a common proxy for cost and latency and is useful for
170+
capacity planning and regression detection across agent versions.
171+
172+
The count is scoped to a single agent invocation and SHOULD include
173+
only the inference calls the agent itself issues; calls made by
174+
sub-agents or transferred-to agents are recorded against those agents'
175+
own invocations so that each inference call is counted exactly once
176+
across the call tree.
177+
178+
Inference calls that fail or return partial results are still counted,
179+
because they consumed the agent's execution.
180+
181+
This metric SHOULD be emitted together with the
182+
`gen_ai.invoke_agent.internal` span for the same invocation.
183+
instrument: histogram
184+
unit: "{inference_call}"
185+
stability: development
186+
attributes:
187+
- ref: gen_ai.agent.name
188+
requirement_level: recommended
189+
- name: gen_ai.invoke_agent.tool_calls
190+
annotations:
191+
code_generation:
192+
metric_value_type: int
193+
brief: >
194+
The number of tool calls a GenAI agent makes during a single
195+
invocation.
196+
note: |
197+
This metric counts the tool executions an agent triggers while
198+
handling a single invocation. It gives operators a generic,
199+
framework-independent sense of how tool-heavy an agent invocation is,
200+
which is useful for capacity planning, regression detection across
201+
agent versions, and diagnosing pathological tool loops (an agent stuck
202+
calling tools without making progress).
203+
204+
The count is scoped to a single agent invocation and SHOULD include
205+
only the tool calls the agent itself triggers; calls made by
206+
sub-agents or transferred-to agents are recorded against those agents'
207+
own invocations so that each tool call is counted exactly once across
208+
the call tree.
209+
210+
Tool calls that fail or return partial results are still counted,
211+
because they consumed the agent's execution.
212+
213+
This metric SHOULD be emitted together with the
214+
`gen_ai.invoke_agent.internal` span for the same invocation.
215+
instrument: histogram
216+
unit: "{tool_call}"
217+
stability: development
218+
attributes:
219+
- ref: gen_ai.agent.name
220+
requirement_level: recommended
157221
- name: gen_ai.execute_tool.duration
158222
annotations:
159223
code_generation:

0 commit comments

Comments
 (0)