Test MV3 model in CI wheel builds

[jathu]

Summary

As part of #9019, we want to ensure we can export models on P0 hosts targeting P0 backends. So, let's test this in our wheel creation. For now, we introduce macOS/Linux XNNPACK export.

Given the wheel building GitHub workflow doesn't support using jobs as a test step, we must go use a python script. This runs a similar test to what we do in PRs:

executorch/.github/workflows/pull.yml

Lines 85 to 136 in 1011fdc

	test-models-linux:
	name: test-models-linux
	uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
	permissions:
	id-token: write
	contents: read
	strategy:
	matrix:
	model: [linear, add, add_mul, ic3, mv2, resnet18, resnet50, mobilebert, emformer_transcribe]
	backend: [portable, xnnpack-quantization-delegation]
	runner: [linux.2xlarge]
	include:
	- model: ic4
	backend: portable
	runner: linux.4xlarge.memory
	- model: ic4
	backend: xnnpack-quantization-delegation
	runner: linux.4xlarge.memory
	- model: emformer_join
	backend: portable
	runner: linux.4xlarge.memory
	- model: emformer_join
	backend: xnnpack-quantization-delegation
	runner: linux.4xlarge.memory
	- model: phi-4-mini
	backend: portable
	runner: linux.4xlarge.memory
	- model: llama3_2_vision_encoder
	backend: portable
	runner: linux.4xlarge.memory
	- model: w2l
	backend: portable
	runner: linux.4xlarge.memory
	fail-fast: false
	with:
	runner: ${{ matrix.runner }}
	docker-image: executorch-ubuntu-22.04-clang12
	submodules: 'true'
	ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
	timeout: 90
	script: |
	# The generic Linux job chooses to use base env, not the one setup by the image
	CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
	conda activate "${CONDA_ENV}"
	MODEL_NAME=${{ matrix.model }}
	BUILD_TOOL=cmake
	BACKEND=${{ matrix.backend }}
	PYTHON_EXECUTABLE=python bash .ci/scripts/setup-linux.sh --build-tool "${BUILD_TOOL}"
	# Build and test ExecuTorch
	PYTHON_EXECUTABLE=python bash .ci/scripts/test_model.sh "${MODEL_NAME}" "${BUILD_TOOL}" "${BACKEND}"

Test plan

CI → Build {Linux, macOS} Wheels

cc @larryliu0820 @lucylq

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9192

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ 1 Pending, 2 Unrelated Failures

As of commit 61d2afe with merge base a05c4da ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest-arm / linux-job (gh) (trunk failure)
  The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[larryliu0820]

Thanks! This is way better than before and thanks for adding the test after building the wheel.

[mergennachin]

why do we need so many redirections?

Can we just call this command directly in the .yml file?

[mergennachin]

Instead of having this

smoke-test-script: ${{ matrix.smoke-test-script }}

can we just run the script directly, similar to how we do in pull.yml and trunk.yml

[jathu]

Instead of having this

smoke-test-script: ${{ matrix.smoke-test-script }}

can we just run the script directly, similar to how we do in pull.yml and trunk.yml

I don't think this is possible with the current job template, it only support running python scripts:

https://github.com/pytorch/test-infra/blob/dd355a4aa131ba56a81bee1ff408ff14b1bcbf70/.github/workflows/build_wheels_macos.yml#L220-L251