Skip to content

Prototype benchmarking with private/custom devices that are not leased via AWS #8251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
guangy10 opened this issue Feb 6, 2025 · 2 comments
Open

Prototype benchmarking with private/custom devices that are not leased via AWS #8251

guangy10 opened this issue Feb 6, 2025 · 2 comments
Assignees
@huydhn
Labels
module: benchmark Issues related to the benchmark infrastructure module: user experience Issues related to reducing friction for users triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@guangy10
Copy link
Contributor

guangy10 commented Feb 6, 2025 •
edited by pytorch-bot bot
Loading

🚀 The feature, motivation and pitch

In some scenarios, users may want to use private/custom devices for benchmarking.

  • The device is not available via AWS, e.g. old phones, embedded systems, etc.
  • For confidential reason, can't run in the public device pool, e.g. FoA/RL models, etc.

This task is to experiment whether we can support this feature within OSS benchmark infra.

CC: @kimishpatel @digantdesai @cbilgin

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @huydhn @kirklandsign @shoumikhin @mergennachin @byjlw
@guangy10 guangy10 added feature module: benchmark Issues related to the benchmark infrastructure module: user experience Issues related to reducing friction for users labels Feb 6, 2025
@github-project-automation github-project-automation bot moved this to To triage in ExecuTorch DevX Feb 6, 2025
@digantdesai digantdesai added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 6, 2025
@mergennachin mergennachin moved this from To triage to Backlog in ExecuTorch DevX Feb 6, 2025
@guangy10
Copy link
Contributor Author

guangy10 commented Feb 11, 2025 •
edited
Loading

@huydhn Given that the only PoC of AIBench is no longer in the team, I think this work is more crucial for us to further scaling to support FoA/RL users in the long term. Bump the priority to P1 temporarily, but would continue the discussion with the team. Size wise I think this is XL?

fyi, @cbilgin @kimishpatel @digantdesai

@huydhn
Copy link
Contributor

huydhn commented Feb 11, 2025

Yeah, this issue is a full size project in itself

This was referenced Feb 11, 2025
Open
Open
@huydhn huydhn moved this to Cold Storage in PyTorch OSS Dev Infra Feb 17, 2025
@jackzhxng jackzhxng removed the feature label Feb 21, 2025
This was referenced Feb 24, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@huydhn huydhn

Labels
module: benchmark Issues related to the benchmark infrastructure module: user experience Issues related to reducing friction for users triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
ExecuTorch Benchmark
Status: No status
ExecuTorch DevX
Status: Backlog
PyTorch OSS Dev Infra
Status: Cold Storage
Milestone
No milestone
Development

No branches or pull requests
4 participants
@digantdesai @huydhn @jackzhxng @guangy10