Skip to content

Expose captured model calls excluded from agent performance totals #4722

Open

Description

@anwithk

Parent: Address Auto-Ontology integration gaps in NeMo Gym #4720

Problem

A resources server can make model calls during verification that are captured under the rollout but are not referenced by an AgentInvocation.

_build_ng_perf() intentionally sums calls referenced by agent invocations. Its token_observability_coverage measures coverage of agent reasoning turns. A value of 1.0 therefore does not mean that every captured call contributes to the reported token totals.

This distinction matters for integrations such as Auto-Ontology Eval PR #44, where the resources server runs the agent during verification and separately reports its usage.

Illustrative case

A rollout has three captured calls with 100, 200, and 300 input tokens. Only the first call belongs to the single observed agent turn. The existing agent-scoped total can correctly be 100 with coverage 1.0, while 500 input tokens remain outside that total.

This is a synthetic acceptance scenario, not a claim about a measured benchmark run.

Requested change

Make captured calls outside agent invocations visible through explicit counts and token totals, and document the scope of the existing coverage metric. Preserve the meaning of existing agent-scoped metrics.

If a captured-call coverage ratio is added, give it a distinct name and document its denominator.

Acceptance criteria

  • The synthetic case exposes three captured calls, one agent-owned call, and two calls outside agent invocations.
  • Input-token reporting distinguishes 100 agent-owned tokens from 500 tokens outside agent invocations.
  • Any captured-call coverage metric reports 1/3 for this case; agent-turn coverage may remain 1.0.
  • Zero captured calls and missing usage have explicit semantics.
  • Shared references to the same physical call do not double count it.
  • Documentation explains which totals include verification or other non-agent calls.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions