Skip to content

AgentX: report E2E Output Token Throughput (avg/P50/P90/P99) as the headline metric - #5571

Open
dwadhwaniTT wants to merge 2 commits into
mainfrom
dwadhwani/agentx-speed-metric
Open

dwadhwaniTT wants to merge 2 commits into
mainfrom
dwadhwani/agentx-speed-metric

Conversation

@dwadhwaniTT

@dwadhwaniTT dwadhwaniTT commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #5570

Summary

Makes AIPerf's E2E Output Token Throughput the headline metric of AgentX reports, under AIPerf's own name. It is output tokens divided by total request time, with TTFT included, and it is the number Artificial Analysis ("speed", average) and OpenRouter ("throughput", P50) publish for a deployment.

  • Parser (llm_module/drivers/aiperf_agentic_traces.py): exports mean_e2e_output_token_throughput, median_e2e_output_token_throughput, p90_e2e_output_token_throughput and p99_e2e_output_token_throughput from AIPerf's e2e_output_token_throughput. P90/P99 use the worse-tail convention that OpenRouter now uses (P99 is the rate only 1% of requests fell below), so they read AIPerf's p10/p1. The end-of-run log prints all four.
  • Report (report_module/agentic_traces_renderer.py): a new "Per-run E2E Output Token Throughput (tok/s per request)" table is the first metric table, with Avg (AA), P50 (OR), P90 and P99 columns. A definition explains the metric. The "E2E Norm Intvty" columns, which showed the same values, are removed from the throughput table.
  • Unchanged: the e2e_norm_intvty fields stay in the payload and in the agenticSweep export, so grading against requirements documents is unaffected. swo-bench rows have no such metric, so the table does not appear for them.
  • Docs and tests: docs/workflow_development.md documents the fields. New tests cover the parser mapping and the table's position, columns and formatting.

Testing

  • Unit tests: the AgentX parser, report and sweep-export tests pass (223).
  • Existing real result: a Kimi-K2.7 AgentX export (32 concurrent users, full mode, 2,989 requests) gives Avg/P50/P90/P99 of 39.34 / 33.79 / 7.34 / 0.97. Recomputing each request's output tokens ÷ total request time from profile_export.jsonl gives exactly the same values.
  • End-to-end local run: the AgentX workflow ran via launchers/run_agentic_traces.py (Kimi-K2.7 config, CI mode, 8 concurrent users, 900 s) against a local OpenAI-compatible stand-in for vLLM, not real hardware. The run passed acceptance. The report's E2E Output Token Throughput table showed 45.62 / 46.55 / 36.59 / 12.85, which matches the per-request recomputation over 339 requests. agentic_sweep.json still carries the e2eNormIntvty* fields.
  • Not yet tested on hardware: this hasn't run against a real Tenstorrent deployment.

🤖 Generated with Claude Code

Export avg/P50/P90/P99 of AIPerf's e2e_output_token_throughput as
mean/median/p90/p99_speed_tps and render them as the first table of the
agentic-traces report. This is the metric Artificial Analysis (avg) and
OpenRouter (P50) publish. P90/P99 use the worse-tail convention, so they
read AIPerf's p10/p1.

The duplicate "E2E Norm Intvty" columns are dropped from the throughput
table; the e2e_norm_intvty fields stay in the payload and agenticSweep
export, so requirements grading is unchanged.

Closes #5570

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dwadhwaniTT
dwadhwaniTT marked this pull request as ready for review November 13, 2026 19:49

@bgoelTT bgoelTT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look great, but can you please preserve the aiperf official terminology of "E2E Output Token Throughput"?

Rename the speed fields to mean/median/p90/p99_e2e_output_token_throughput
and label the report table, columns, definition and log line with AIPerf's
official term. "speed" remains only where the text maps the metric to what
Artificial Analysis publishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dwadhwaniTT dwadhwaniTT changed the title AgentX: report per-request E2E output throughput as headline "speed" (avg/P50/P90/P99) AgentX: report E2E Output Token Throughput (avg/P50/P90/P99) as the headline metric Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AgentX: make per-request E2E output throughput ("speed") a headline release metric

2 participants