Support benchmark using prebuilt artifacts

[guangy10]

🚀 The feature, motivation and pitch

Enable this for on-demand workflow only, to offer developers additional flexibility and efficiency.

Scenarios where benchmarking w/ prebuilt artifacts are needed:

• Sometimes the pte model may come from outside, for example the model may come from external partners. Or the model is downloaded from the executorch community from Hugging Face, https://huggingface.co/executorch-community/DeepSeek-R1-Distill-Llama-8B/tree/main.
• Developers who work on the runtime may not necessarily to re-export the same model all the time.
• Developers who work on exporting may not need to build the banchmark app all the time

UX:

• via GitHub UI
• via script

Source of the artifacts to be used in the benchmark workflow:

• pte models from Hugging Face, e.g. https://huggingface.co/executorch-community/Llama-3.2-1B-Instruct-ET/tree/main
• From S3 (uploaded by developers/users)

We will need to define the UX to support this feature. For example, allow users to upload prebuilt artifacts via script. The script will return with links to the artifacts. Then users can schedule an on-demand workflow via UI, or users can do everything via the script.

Policy and TTL to keep the uploaded artifacts.

CC: @digantdesai @kimishpatel @cccclai

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @huydhn @kirklandsign @shoumikhin @mergennachin @byjlw

[guangy10]

IMO this infra work can potentially benefit both #8249 and #8250

[huydhn]

AI: Circle back with @kirklandsign and @shoumikhin on what pre-built artifacts that need to be there on the device besides the app and the export model

[shoumikhin]

@huydhn if you're asking about the benchmarking app, we need to put all .xcframework under Frameworks, and all .pte under Resources, then build and run the app.

[guangy10]

@shoumikhin @kirklandsign I believe @huydhn is referring to the new artifacts generated by the profilers, e.g. ETdump, chrome trace, simpleperf, etc. specifically, generated from these work:

• @kirklandsign Integrate mem profiler to benchmark infra #8241
• @kirklandsign Integrate ETdump to benchmark infra #8240
• @kirklandsign Enable framegraphs for benchmarking on Android #8242
• @shoumikhin Enable framegraphs for benchmarking on iOS #8243

For example, for users be able to accessible the artifacts and visualize in framegraphs, we will need to store the artifacts somewhere, e.g. S3. Same for Etdump.

[guangy10]

@shoumikhin @kirklandsign fyi, Huy will be on leave from Feb 25, if you guys need help to store the artifacts in DB, or access the artifacts from the dashboard UI or showing the URL in the job log, Huy can help with it or give you a pointer before he in on leave.

[guangy10]

Follow up on the discussion around supporting artifacts generated from profilers. I think if we can provide a generic script allow uploading arbitrary blobs (typically zipped) to S3, it should provide the flexibility to store any artifact from profiling. wdyt @shoumikhin @kirklandsign @huydhn?

@huydhn is there a size limit for uploading to S3? If yes, what it is?

Put together with the work in #8245, the UX would like what I described here: #8402 (comment)

[guangy10]

Uploading from linux/macos runner should be ok per Huy.
Experiment to trigger an upload from device farm to see if there is a size limits