Skip to content

ci: improve test CI speed and efficiency#5904

Open
jezdez wants to merge 12 commits intomainfrom
ci-speed-improvements
Open

ci: improve test CI speed and efficiency#5904
jezdez wants to merge 12 commits intomainfrom
ci-speed-improvements

Conversation

@jezdez
Copy link
Copy Markdown
Member

@jezdez jezdez commented Jan 27, 2026

Summary

This PR improves CI test speed through several optimizations:

CI Workflow Improvements

  • Reduced test matrix: Skip Python 3.11 (3.10 and 3.12 provide sufficient coverage), use reduced matrix for PRs (only 3.10 and 3.13)
  • Exclude benchmark tests: 5 benchmark tests (including the 22-minute test_render_recipe) now only run in the dedicated linux-benchmarks job
  • Improved caching: Add restore-keys fallback patterns for better cache hit rates across jobs
  • Reduced reruns: Decrease flaky test reruns from 3 to 1 on Linux
  • Faster macOS runners: Switch from macos-15-intel to macos-latest (ARM), which are 2-3x faster

Template Environment Cloning (New Feature)

  • New test_env_template config option: Allows create_env to clone from a pre-existing template environment instead of creating from scratch
  • Uses conda create --clone: Properly handles prefix replacement in scripts and metadata files
  • Uses MatchSpec for spec parsing: Correctly handles all spec formats including ~=, >=, and bracket syntax like numpy[version='>1.0']
  • New warm_package_cache pytest fixture: Creates a session-scoped template environment with Python, pip, setuptools, and expat

Test Fixes

  • Updated test_transitive_subpackage_dependency assertion to match openssl 3.0 (config was updated from 1.0.2)
  • Updated test_top_level_finalized to use xz 5.6 (5.4.2 no longer available in channels)

Estimated CI Impact

Platform Before (jobs) After (PRs) After (Main)
Linux 8 4 6
Windows 4 4 4
macOS 4 (~70min) 4 (~30min) 4 (~30min)

Why test_render_recipe was slow

The benchmark test uses a complex conda-forge recipe (petsc4py) with ~950 lines of conda_build_config.yaml generating 48 variants on Linux. This is intentional for benchmarking real-world rendering performance.

Test plan

  • CI passes on all platforms
  • Verify macOS tests work with ARM runners and 11.3 SDK
  • Verify benchmark tests only run in the linux-benchmarks job
  • Monitor cache hit rates in subsequent runs
  • Verify template cloning works on all platforms (macOS, Linux, Windows)

- Reduce test matrix: skip Python 3.11, use reduced matrix for PRs
- Exclude benchmark tests from regular CI (run in dedicated job)
- Add cache restore-keys for better cache hit rates
- Reduce flaky test reruns from 3 to 1
- Switch macOS to faster ARM runners (macos-latest)
- Update SDK download for ARM architecture support
@jezdez jezdez requested a review from a team as a code owner January 27, 2026 11:09
@github-project-automation github-project-automation bot moved this to 🆕 New in 🔎 Review Jan 27, 2026
@conda-bot conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Jan 27, 2026
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Jan 27, 2026

Merging this PR will not alter performance

✅ 5 untouched benchmarks


Comparing ci-speed-improvements (60137c9) with main (aed61ed)

Open in CodSpeed

Add test_env_template config option that allows create_env to clone
from a pre-existing template environment instead of creating from
scratch. This provides ~38% speedup for test builds where the template
contains all required packages.

Cross-platform support:
- macOS (APFS): uses cp -c for copy-on-write (~1.8s)
- Linux: uses cp --reflink=auto for reflinks on btrfs/xfs
- Windows: uses shutil.copytree (~2.5s)

Also adds warm_package_cache pytest fixture that creates a session-scoped
template environment with Python, pip, setuptools, and expat.
- Update package versions in test recipes to versions available on osx-arm64:
  - xz: 5.2.3 → 5.4.2 (30_top_level_finalized)
  - openssl: 1.0.2 → 3.0 (transitive_subpackage)
  - python: 3.6 → 3.10 (transitive_subpackage)
  - libpng: 1.6.34 → 1.6.37 (always_include_files_glob)

- Update test_top_level_finalized assertion to match new xz version

- Fix test_add_pip_as_python_dependency_from_condarc_file[False] by
  disabling template environment cloning for that test (template includes
  pip which defeats the test's purpose)

- Skip R-related tests on osx-arm64 since r-base is not available in
  defaults channel for that platform
@jezdez jezdez mentioned this pull request Feb 2, 2026
3 tasks
Copy link
Copy Markdown
Contributor

@beckermr beckermr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few questions and confusions on my part.

jezdez and others added 3 commits February 4, 2026 20:00
- Use `conda create --clone` instead of platform-specific copy methods
  for proper prefix replacement in scripts and metadata files
- Use MatchSpec for proper spec parsing instead of manual string splitting
- Fix test_transitive_subpackage_dependency: update assertion for openssl 3.0
- Fix test_top_level_finalized: update xz version to 5.6 (available in channels)
- Update news file and docstrings to reflect the conda clone approach
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed [bot] added once the contributor has signed the CLA

Projects

Status: 🏗️ In Progress

Development

Successfully merging this pull request may close these issues.

3 participants