Skip to content

Update display metric for E2E-QNA - #2653

Merged
hanyunfan merged 1 commit into
masterfrom
e2e_metric_update
Aug 18, 2026
Merged

Update display metric for E2E-QNA#2653
hanyunfan merged 1 commit into
masterfrom
e2e_metric_update

Conversation

@pgmpablo157321

Copy link
Copy Markdown
Contributor

No description provided.

@pgmpablo157321
pgmpablo157321 requested review from a team as code owners August 17, 2026 20:13
@github-actions

Copy link
Copy Markdown
Contributor

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@hans-intel

Copy link
Copy Markdown
Contributor

Queries per sec (qps) is commonly used in single LLM call benchmarks, creating confusion why E2E-RAG has such a low perf number, especially we are using MLPerf GPT-OSS-120B model. In E2E-RAG,

  • a single query can invoke more than a dozen LLM calls
  • The pipeline involves 4 different inference models
  • The pipeline involves non-model workload

Therefore, we propose to use tasks/sec, where each query represents a QnA task.

@hanyunfan hanyunfan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hanyunfan
hanyunfan merged commit d2e600b into master Aug 18, 2026
35 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants