You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add run-level totals for model-call capture metrics #4723
Applications need to summarize captured model usage across an evaluation run. In the inspected aggregate-metrics request path, ng_model_call_capture is removed before verification results are sent to /aggregate_metrics.
Auto-Ontology Eval PR #44 works around this by reading captures inside verify() and copying usage into numeric response fields. Each integration should not have to implement that accounting separately.
Requested change
Provide a standard run summary of model-call capture metrics, either through a lightweight aggregation payload or a dedicated summary path. Keep full prompts, responses, and capture records out of aggregate requests.
Include call counts, input/output/total tokens when available, and clearly defined call latency totals. Distinguish summed call latency from elapsed run time because calls may overlap.
Acceptance criteria
A two-rollout fixture with 2 and 3 captured calls reports 5 calls in the run summary.
Token totals reconcile with the available per-rollout capture metrics without custom verification fields.
Missing usage and incomplete captures are surfaced rather than silently presented as complete totals.
Activity