Skip to content

CI: Optimize benchmarks by cloning template environments#236

Merged
jezdez merged 7 commits intomainfrom
optimize-benchmarks-clone-env
Feb 6, 2026
Merged

CI: Optimize benchmarks by cloning template environments#236
jezdez merged 7 commits intomainfrom
optimize-benchmarks-clone-env

Conversation

@jezdez
Copy link
Copy Markdown
Member

@jezdez jezdez commented Feb 4, 2026

Summary

This PR optimizes benchmark runtime by cloning a pre-created template environment instead of running conda create for each test.

Problem

The benchmark tests call conda create 5 times (3x in test_convert_tree, 2x in test_build_conda). Each conda create invokes the solver and extracts packages, taking ~10 seconds each.

Solution

  1. Session-scoped template: Create a Python environment once at the start of the test session
  2. Fast cloning: Clone the template using conda create --clone instead of raw file copying

Why conda create --clone instead of raw file copy?

Raw file copying (cp, shutil.copytree, etc.) doesn't properly handle conda's prefix relocation:

  • conda-meta/*.json files contain hardcoded prefix paths
  • Scripts have shebang lines pointing to the original environment
  • Some packages store paths in config files

conda create --clone properly handles all of this while still being faster than a full create because it:

  • Skips the solver (no SAT solving needed)
  • Skips downloading (packages already cached)
  • Only needs to relocate and link packages

Measured Performance Improvement

Branch Benchmark Duration Job Link
main (baseline) 17m 39s benchmarks py312
This PR 10m 47s benchmarks py312
Improvement 6m 52s faster (39%)

Changes

  • tests/conftest.py:
    • Add python_template_env session-scoped fixture using session_conda_cli
    • Template creation failures propagate immediately (fail fast)
  • tests/test_benchmarks.py:
    • Update tests to use python_template_env fixture
    • Clone from template using conda create --clone
    • Use idiomatic nonlocal for setup counters instead of list workaround

Test plan

  • All benchmark tests pass locally
  • CI passes on all platforms (Linux, macOS, Windows)
  • Benchmarks complete successfully with CodSpeed
  • Verified ~39% benchmark runtime improvement

Instead of running `conda create` for each benchmark test (5 times total),
this PR creates a session-scoped template environment once and clones it
using platform-specific fast copy methods:

- macOS (APFS): cp -c for copy-on-write (~1.8s)
- Linux: cp --reflink=auto for reflinks on btrfs/xfs, else regular copy
- Windows: shutil.copytree (~2.5s)

This avoids the conda solver and package extraction overhead (~10s) for
each environment creation, potentially saving 40-50 seconds per benchmark
run.

Changes:
- Add `clone_env()` helper with cross-platform fast copy support
- Add `python_template_env` session-scoped fixture
- Update benchmark tests to clone from template instead of conda create
- Graceful fallback to conda create if cloning fails
@jezdez jezdez requested a review from a team as a code owner February 4, 2026 13:33
Instead of manually calling subprocess to create the template environment,
use conda's built-in session_tmp_env fixture from conda.testing.fixtures.

This leverages the official conda testing infrastructure and simplifies
the code while maintaining the same functionality.
@jezdez jezdez marked this pull request as draft February 4, 2026 13:36
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Feb 4, 2026

CodSpeed Performance Report

Merging this PR will not alter performance

Comparing optimize-benchmarks-clone-env (de83b98) with main (3ff94a4)

Summary

✅ 5 untouched benchmarks

On Windows, use robocopy instead of shutil.copytree for environment
cloning. Robocopy provides:
- Multi-threaded copying (/mt:8) for parallel file operations
- Native Windows optimization for file operations
- Better performance than Python's shutil.copytree

This should reduce the Windows test runtime gap compared to Linux/macOS.
Replace raw file copying with `conda create --clone` to properly handle
conda's prefix relocation. Raw file copying doesn't update conda-meta
JSON files or fix shebang lines in scripts, causing broken environments.

Using `conda create --clone` is slightly slower than raw copy but still
faster than a full `conda create` because it skips solver and downloads.
@jezdez
Copy link
Copy Markdown
Member Author

jezdez commented Feb 5, 2026

Validation Results ✅

The conda create --clone optimization has been validated. Here's the comparison:

Benchmark Job Runtime

Branch Duration Job Link
main (baseline) 17m 56s benchmarks py312
This PR 10m 20s benchmarks py312
Improvement 7m 36s faster (42%)

Full Workflow Comparison

Workflow main This PR
Test 18m 8s 15m 43s

Why this works

The original approach used raw file copying (cp, shutil.copytree) which doesn't handle conda's prefix relocation. This was fixed to use conda create --clone which:

  1. Skips the solver - No SAT solving needed since we're copying an existing solution
  2. Skips downloading - Packages are already cached from template creation
  3. Properly relocates prefixes - Updates conda-meta/*.json and fixes shebang lines in scripts

The improvement is larger than initially estimated because:

  • Template creation cost is amortized across all benchmark iterations
  • Solver avoidance saves more time than package extraction alone
  • Each benchmark runs multiple setup/teardown cycles

All tests pass on all platforms (Linux, macOS, Windows).

@jezdez jezdez marked this pull request as ready for review February 5, 2026 11:06
@jezdez jezdez added this to the 0.4.0 milestone Feb 5, 2026
@jezdez
Copy link
Copy Markdown
Member Author

jezdez commented Feb 5, 2026

This PR is more of a nice-to-have, than a hard requirement for 0.4.0

Copy link
Copy Markdown
Contributor

@soapy1 soapy1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice speed up ⏩

@jezdez jezdez force-pushed the optimize-benchmarks-clone-env branch from 53e1efa to 0f279aa Compare February 6, 2026 08:33
- Remove try/except fallback from python_template_env fixture; let
  failures propagate immediately instead of silently yielding None
- Replace setup_counter list hack with idiomatic nonlocal int
- Remove conditional clone-or-create logic since the fixture now
  always provides a valid template path
@jezdez jezdez force-pushed the optimize-benchmarks-clone-env branch from 0f279aa to de83b98 Compare February 6, 2026 08:37
@jezdez
Copy link
Copy Markdown
Member Author

jezdez commented Feb 6, 2026

Addressed all review feedback:

  • Removed try/except fallback from python_template_env fixture — if template creation fails, the test session now fails fast with a clear error instead of silently yielding None and deferring the failure to each individual test.
  • Replaced setup_counter = [0] with idiomatic nonlocal setup_counter in both test_convert_tree and test_build_conda.
  • Removed conditional clone-or-create logic — since the fixture now always provides a valid path (or raises), the tests always use conda create --clone directly.

Also merged latest main and updated the PR description with fresh benchmark numbers (17m 39s → 10m 47s, ~39% improvement).

@jezdez jezdez requested a review from soapy1 February 6, 2026 08:50
@jezdez jezdez removed this from the 0.4.0 milestone Feb 6, 2026
@jezdez jezdez enabled auto-merge (squash) February 6, 2026 09:03
Copy link
Copy Markdown
Contributor

@soapy1 soapy1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jezdez jezdez merged commit 9b246be into main Feb 6, 2026
34 checks passed
@jezdez jezdez deleted the optimize-benchmarks-clone-env branch February 6, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants