[API BASE]Setup cli & python lib for benchmark toolings

[yangw-dev]

Overview

Set up base for cli tools & python lib that can be imported and called for benchmark use cases.

Details

• Added bash script to auto-generate datamodel for both ts and python, it includes validations

  • generate.sh
  • zod_generate.sh

• set up base for cli tool using cement, so user can installl and call the cli tool anywhere in their machine

• ExecuTorch specific (for now):

  • Add shortcut cli & python method to fetch execuBenchmark result from hud api

How to use the cli

See readme

Third Party License Check

All confirmed pre-approved in oss_project_use(meta)
- zod [MIT]
- pydantic[MIT]
- datamodel-code-generator[MIT]
- cement [BSD 3-Clause License]

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
torchci	⬜️ Ignored (Inspect)	Visit Preview	Jun 13, 2025 11:18am

[seemethere]

Maybe I'm missing some context on this, what is this supposed to be doing?

[yangw-dev]

Maybe I'm missing some context on this, what is this supposed to be doing?

This basically provide benchmark tools for xfn teams to fetch data, trigger lambda and bisect etc.
I want to have a central place to put all scripts and api client for benchmark users, & use cases.

Wrapped in as cli-tool so they can call it everywhere once it's installed.

In this pr, one use case is, execuTorch currently ask to fetch all data grouped by device so they can analyze how reliable the benchmark result is

[huydhn]

Let's sync on this as I don't think we want to implement this CLI tool without a proper design. IMO, I don't think we need a cli tool like this for now because of several reasons:

• There is already an experimental query API for them to get the data https://github.com/pytorch/pytorch/wiki/PyTorch-OSS-benchmark-infra#experimental-query-api, I create this few weeks back to allows people from compiler and vLLM to query their benchmark results. They can do whatever they want with the data. I think we should do the same for ExecuTorch
• I think we want to avoid having customer-specific logic on benchmark infra side, so having a CLI tool to support ExecuTorch use case seems wrong. It should be a script on ExecuTorch side instead and the grouped by device part should be moved there. Otherwise, this might become another behemoth like TorchBench
• In addition, other features like triggering lambda or bisecting is too early to implement now because we don't even have them on CI yet

[huydhn]

Let's hold on to this cli tool for further discussion #6756 (comment)

[yangw-dev]

kk, sounds good!
This is def just a proposal