You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Expose captured model calls excluded from agent performance totals #4722
A resources server can make model calls during verification that are captured under the rollout but are not referenced by an AgentInvocation.
_build_ng_perf() intentionally sums calls referenced by agent invocations. Its token_observability_coverage measures coverage of agent reasoning turns. A value of 1.0 therefore does not mean that every captured call contributes to the reported token totals.
This distinction matters for integrations such as Auto-Ontology Eval PR #44, where the resources server runs the agent during verification and separately reports its usage.
Illustrative case
A rollout has three captured calls with 100, 200, and 300 input tokens. Only the first call belongs to the single observed agent turn. The existing agent-scoped total can correctly be 100 with coverage 1.0, while 500 input tokens remain outside that total.
This is a synthetic acceptance scenario, not a claim about a measured benchmark run.
Requested change
Make captured calls outside agent invocations visible through explicit counts and token totals, and document the scope of the existing coverage metric. Preserve the meaning of existing agent-scoped metrics.
If a captured-call coverage ratio is added, give it a distinct name and document its denominator.
Acceptance criteria
The synthetic case exposes three captured calls, one agent-owned call, and two calls outside agent invocations.
Activity