Skip to content

Add run-level totals for model-call capture metrics #4723

Open

Description

@anwithk

Parent: Address Auto-Ontology integration gaps in NeMo Gym #4720

Problem

Applications need to summarize captured model usage across an evaluation run. In the inspected aggregate-metrics request path, ng_model_call_capture is removed before verification results are sent to /aggregate_metrics.

Auto-Ontology Eval PR #44 works around this by reading captures inside verify() and copying usage into numeric response fields. Each integration should not have to implement that accounting separately.

Requested change

Provide a standard run summary of model-call capture metrics, either through a lightweight aggregation payload or a dedicated summary path. Keep full prompts, responses, and capture records out of aggregate requests.

Include call counts, input/output/total tokens when available, and clearly defined call latency totals. Distinguish summed call latency from elapsed run time because calls may overlap.

Acceptance criteria

  • A two-rollout fixture with 2 and 3 captured calls reports 5 calls in the run summary.
  • Token totals reconcile with the available per-rollout capture metrics without custom verification fields.
  • Missing usage and incomplete captures are surfaced rather than silently presented as complete totals.
  • Agent-owned and other captured calls have documented inclusion rules, consistent with the excluded-calls child issue #4722.
  • Summaries do not add agent metrics to capture metrics in a way that counts the same call twice.
  • Capture-disabled runs remain supported without claiming complete zero usage.
  • A public example documents the output and aggregation semantics.

Relationship to other children

The excluded-calls issue #4722 defines accounting scope within a rollout. This issue makes capture metrics available across a run. The cache-counting fix #4721 establishes correctness of the normalized inputs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions