[CI] Run nightly PyTorch model hub scopes on related PRs and on dispatch by mvafin · Pull Request #38457 · openvinotoolkit/openvino

mvafin · 2026-09-28T16:48:54Z

Details:

The PyTorch Models Nightly Scope1/Scope2 tests jobs in ubuntu_22.yml run only on schedule. Changes to their model lists or tests are validated only after merge (e.g. the Scope1 runner OOM and the removed HF model fixed in [PT FE] Fix timm model filter selecting huge models, update timm to 1.0.30, support aten.meshgrid in FX #38443).
Both jobs now also run:
- in pre-commit, for PRs that change tests/model_hub_tests/pytorch, tests/model_hub_tests/models_hub_common or job_pytorch_models_tests.yml. A step in the Smart_CI job checks the PR's changed files through the GitHub API, so no new label or Smart CI component is needed;
- on manual dispatch with the new pytorch-models-nightly input: gh workflow run ubuntu_22.yml --ref <branch> -f pytorch-models-nightly=true.
They use the same reusable job, runner, image and settings as the nightly. Push and merge-queue runs don't start them, and they aren't part of Overall_Status.

Tickets:

N/A

AI Assistance:

AI assistance used: yes
AI implemented the change and checked it locally:
- actionlint (with shellcheck): no findings in the new code.
- Ran the file-check script against existing PRs: it returns true for [PT FE] Fix timm model filter selecting huge models, update timm to 1.0.30, support aten.meshgrid in FX #38443 and [PT FE] Keep tqdm below 4.70 for LLM model hub tests and re-enable the opt_gptq export test #37890, which change model hub tests, and nothing for this PR or a CPU-only PR.

Copilot

Copilot review overview

🟡 Changes recommended

Missing repository labels prevent activation, and Scope2 currently triggers expensive runs for unrelated files.

Review effort: Balanced
Findings: 1

Open (1)

Missing repository labels prevent triage and nightly jobs · New

What changed in this PR

Adds selective pre-commit and manual execution for PyTorch nightly model scopes.

Changes:

Adds manual scope selection and PR-aware job conditions.
Registers Scope1/Scope2 Smart CI components and labels.
Documents the nightly-job pattern.

File	Description
`.github/workflows/ubuntu_22.yml`	Adds dispatch input and conditional nightly jobs.
`.github/labeler.yml`	Maps changed files to nightly scopes.
`.github/components.yml`	Registers isolated Smart CI components.
`docs/dev/ci/github_actions/adding_tests.md`	Documents selective nightly execution.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Nightly PyTorch Models Scope1/Scope2 jobs ran only on schedule, so changes to their model lists, test code or requirements were validated only after merge. Run each scope in pre-commit when a PR changes its files (new Smart CI labels with no dependents), and allow starting them via workflow_dispatch with the pytorch-models-nightly-scope input. Same runner, image and settings.

Replace the two scope labels with one PyTorch_Models_Nightly component, use a boolean workflow_dispatch input, and drop the docs change.

mvafin · 2026-09-29T09:54:41Z

Moved into #38443, so that PR can run the nightly PyTorch model hub scopes on its own model list changes before merge. @mryzhov, your comments are answered above; the same ubuntu_22.yml change is now in #38443.

Check the PR's changed files in the Smart_CI job instead of adding a Smart CI label and component.

…-nightly-triggers

….0.30, support aten.meshgrid in FX (openvinotoolkit#38443) ### Details: - Nightly Scope1 has been crashing with runner OOM since openvinotoolkit#33706. `filter_timm` sorted size tokens alphabetically (`enormous` < `giant` < ... < `small`) and didn't recognize `7b`/`so400m`, so it picked the *largest* variants (e.g. `vit_7b_dinov3`, 26.9 GB). Several of these were loaded at the same time under `-n 4`. - `filter_timm` now keeps the smallest variant of each architecture. Size tokens are ordered by size, parameter counts and bare `t`/`h` sizes are recognized, and width-multiplier variants (`resnetv2_50x3`, `resnext101_32x8d`) are folded into their base architecture (they use the same ops). - timm updated to 1.0.30. `timm_models` is regenerated: 517 models, down from 532. The largest model is now 4.05 GB, and total weights drop from ~248 GB to ~108 GB. New failing models are marked xfail: `iformer_*` (numpy<2 load error) and `gemma4_vit_167m*` in export mode. - `aten::meshgrid` is now converted during translation for both TorchScript and FX. `PrimListUnpackReplacer` is still the fallback when the input list is unknown at that point. This fixes export mode for 9 new timm models and meshgrid outputs used directly by `stack`/`cat`/`[i]`. - Nightly Scope2 fails because `DunnBC22/efficientformer-l3-300-Brain_Tumors_Image_Classification` was removed from HF Hub. That failure also skips the LLM, Detectron2 and Torchbench steps. The model is replaced with `snap-research/efficientformer-l1-300`. - LLM nightly test `qwen3_compressed_tensors` (`cyankiwi/Qwen3.5-4B-AWQ-4bit`, a multimodal checkpoint) is marked xfail. With the pinned transformers 5.5.3, `AutoModelForCausalLM` drops its `quantization_config` (fixed in 5.8, huggingface/transformers#45494). Newer versions still don't match the checkpoint's ignore list to the text-only module names, so `linear_attn.in_proj_a/b` are initialized randomly. - Torchbench: all models failed in `install.py`, which requires torchaudio. torchaudio was dropped from the requirements in openvinotoolkit#36098 and doesn't support torch 2.14. The test removes torchaudio from torchbench's `TORCH_DEPS`, pins `setuptools<81` for sdist builds (`visdom` needs `pkg_resources`), and sets `PIP_USE_PEP517=1`. With pip 24, legacy `setup.py` packages are built in the test venv, where the `egg_info` hook of `kernels` (installed for LLM tests) crashes against the older `huggingface_hub` installed by torchbench. The maml models' inputs, pickled with numpy<2, now load with numpy 2. - Scope1 failures that existed before this PR: - `convit_tiny` (trace): the model fills a cache on its first forward, so the test runs one warm-up forward first. - torchvision segmentation models (export): FX model outputs are now named after their dict keys, which for `ExportedProgram` are taken from `out_spec`. A key is skipped if it's ambiguous (one tensor under several keys, or the same key in several nested dicts) or matches the name of another graph node or input. - `efficientvit_b0` (export): fixed by openvinotoolkit#38396, now passes. - Documented how to run model hub tests with a model cache. - The nightly `PyTorch Models Nightly Scope1/Scope2 tests` jobs in `ubuntu_22.yml` now also run for PRs that change `tests/model_hub_tests/pytorch`, `tests/model_hub_tests/models_hub_common` or `job_pytorch_models_tests.yml`. The `Smart_CI` job checks the PR's changed files with `git diff` on the merge commit, without extra API calls. The jobs also run on manual dispatch with the new `pytorch-models-nightly` input. They use the same runner and settings as the nightly, and aren't part of `Overall_Status`, so a failure doesn't block merge. This PR runs them itself. Moved from openvinotoolkit#38457. ### Tickets: - N/A ### AI Assistance: - AI assistance used: yes - AI analyzed the CI logs, implemented the changes and ran local validation (torch 2.14.0+cpu, timm 1.0.30): - New timm models and the timm models that use meshgrid: pass or xfail in trace and export mode. - `test_models_list_complete`: passes. - Full layer suites: 0 failures, with TorchScript (16533 passed) and with export (12969 passed, including the new dict output name tests). - Torchbench, convit, efficientvit and torchvision segmentation models: pass. - New efficientformer HF test: passes.

mvafin requested review from a team as code owners September 28, 2026 16:48

mvafin requested review from ValentinaKats and removed request for a team September 28, 2026 16:48

github-actions Bot added category: CI OpenVINO public CI category: docs OpenVINO documentation github_actions Pull requests that update GitHub Actions code labels Sep 28, 2026

mvafin requested a balanced review from Copilot September 28, 2026 16:52

Copilot started reviewing on behalf of mvafin September 28, 2026 16:52

View session

github-actions Bot removed the category: docs OpenVINO documentation label Sep 28, 2026

Copilot AI reviewed Sep 28, 2026

View reviewed changes

Comment thread .github/labeler.yml Resolved

mvafin added 2 commits September 28, 2026 19:09

[CI] Use a single component for nightly PyTorch model hub triggers

0d918a7

Replace the two scope labels with one PyTorch_Models_Nightly component, use a boolean workflow_dispatch input, and drop the docs change.

mryzhov reviewed Sep 29, 2026

View reviewed changes

Comment thread .github/components.yml Resolved

mryzhov reviewed Sep 29, 2026

View reviewed changes

Comment thread .github/workflows/ubuntu_22.yml Resolved

mryzhov approved these changes Sep 29, 2026

View reviewed changes

mvafin enabled auto-merge September 29, 2026 09:40

mvafin disabled auto-merge September 29, 2026 09:48

mvafin mentioned this pull request Sep 29, 2026

[PT FE] Fix timm model filter selecting huge models, update timm to 1.0.30, support aten.meshgrid in FX #38443

Merged

mvafin closed this Sep 29, 2026

mvafin added 2 commits September 29, 2026 11:57

[CI] Detect nightly PyTorch model test changes without a new label

17d298d

Check the PR's changed files in the Smart_CI job instead of adding a Smart CI label and component.

Merge remote-tracking branch 'upstream/master' into ci-pytorch-models…

82e0823

…-nightly-triggers

Navigation Menu

Repository navigation

[CI] Run nightly PyTorch model hub scopes on related PRs and on dispatch - #38457

[CI] Run nightly PyTorch model hub scopes on related PRs and on dispatch#38457
mvafin wants to merge 4 commits into
openvinotoolkit:masterfrom
mvafin:ci-pytorch-models-nightly-triggers

mvafin commented Sep 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvafin commented Sep 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mvafin commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvafin commented Sep 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mvafin commented Sep 28, 2026 •

edited

Loading