Skip to content

Report avg_inference_latency from Android LLM benchmark app #8578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
guangy10 opened this issue Feb 19, 2025 · 3 comments · Fixed by #10417
Closed

Report avg_inference_latency from Android LLM benchmark app #8578

guangy10 opened this issue Feb 19, 2025 · 3 comments · Fixed by #10417
Assignees
@kirklandsign
Labels
module: benchmark Issues related to the benchmark infrastructure triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone
0.6.0

Comments

@guangy10
Copy link
Contributor

guangy10 commented Feb 19, 2025 •
edited by pytorch-bot bot
Loading

🚀 The feature, motivation and pitch

Missing from Android. See details here: #8576 (comment)

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @huydhn @kirklandsign @shoumikhin
@guangy10 guangy10 added the module: benchmark Issues related to the benchmark infrastructure label Feb 19, 2025
@guangy10 guangy10 added this to the 0.6.0 milestone Feb 19, 2025
@guangy10 guangy10 moved this to Ready in ExecuTorch Benchmark Feb 19, 2025
@guangy10 guangy10 mentioned this issue Feb 19, 2025
Open
@swolchok swolchok added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 21, 2025
@kirklandsign kirklandsign changed the title Report avg_inference_latency from Android benchmark app Report avg_inference_latency from Android LLM benchmark app Mar 13, 2025
@kirklandsign kirklandsign moved this from Ready to In Progress in ExecuTorch Benchmark Mar 13, 2025
@kirklandsign
Copy link
Contributor

Sorry just realize for LLM it's missing

@kirklandsign
Copy link
Contributor

Numbers are reported now https://hud.pytorch.org/benchmark/llms?startTime=Wed,%2016%20Apr%202025%2017:26:16%20GMT&stopTime=Wed,%2023%20Apr%202025%2017:26:16%20GMT&granularity=day&repoName=pytorch%2Fexecutorch&benchmarkName=&modelName=llama&backendName=xnnpack_q8&modeName=inference&dtypeName=&deviceName=Samsung%20Galaxy%20S22%205G%20(Android%2013)&archName=All%20Platforms

@github-project-automation github-project-automation bot moved this from In Progress to Done in ExecuTorch Benchmark Apr 23, 2025
@kirklandsign
Copy link
Contributor

#10376

@kirklandsign kirklandsign mentioned this issue Apr 24, 2025
Merged
kirklandsign added a commit that referenced this issue Apr 24, 2025
@kirklandsign
Reland minibench refactor (#10417)
3e38b85 
Reland ebab7bb

Resolved: #8578
Resolved: #10417
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kirklandsign kirklandsign

Labels
module: benchmark Issues related to the benchmark infrastructure triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
ExecuTorch Benchmark
Status: Done
Milestone
0.6.0
Development

Successfully merging a pull request may close this issue.

Reland minibench refactor pytorch/executorch
3 participants
@swolchok @guangy10 @kirklandsign