CI: Optimize benchmarks by cloning template environments#236
Merged
Conversation
Instead of running `conda create` for each benchmark test (5 times total), this PR creates a session-scoped template environment once and clones it using platform-specific fast copy methods: - macOS (APFS): cp -c for copy-on-write (~1.8s) - Linux: cp --reflink=auto for reflinks on btrfs/xfs, else regular copy - Windows: shutil.copytree (~2.5s) This avoids the conda solver and package extraction overhead (~10s) for each environment creation, potentially saving 40-50 seconds per benchmark run. Changes: - Add `clone_env()` helper with cross-platform fast copy support - Add `python_template_env` session-scoped fixture - Update benchmark tests to clone from template instead of conda create - Graceful fallback to conda create if cloning fails
Instead of manually calling subprocess to create the template environment, use conda's built-in session_tmp_env fixture from conda.testing.fixtures. This leverages the official conda testing infrastructure and simplifies the code while maintaining the same functionality.
CodSpeed Performance ReportMerging this PR will not alter performanceComparing Summary
|
On Windows, use robocopy instead of shutil.copytree for environment cloning. Robocopy provides: - Multi-threaded copying (/mt:8) for parallel file operations - Native Windows optimization for file operations - Better performance than Python's shutil.copytree This should reduce the Windows test runtime gap compared to Linux/macOS.
4 tasks
Replace raw file copying with `conda create --clone` to properly handle conda's prefix relocation. Raw file copying doesn't update conda-meta JSON files or fix shebang lines in scripts, causing broken environments. Using `conda create --clone` is slightly slower than raw copy but still faster than a full `conda create` because it skips solver and downloads.
Member
Author
Validation Results ✅The Benchmark Job Runtime
Full Workflow Comparison
Why this worksThe original approach used raw file copying (
The improvement is larger than initially estimated because:
All tests pass on all platforms (Linux, macOS, Windows). |
Member
Author
|
This PR is more of a nice-to-have, than a hard requirement for 0.4.0 |
53e1efa to
0f279aa
Compare
- Remove try/except fallback from python_template_env fixture; let failures propagate immediately instead of silently yielding None - Replace setup_counter list hack with idiomatic nonlocal int - Remove conditional clone-or-create logic since the fixture now always provides a valid template path
0f279aa to
de83b98
Compare
Member
Author
|
Addressed all review feedback:
Also merged latest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR optimizes benchmark runtime by cloning a pre-created template environment instead of running
conda createfor each test.Problem
The benchmark tests call
conda create5 times (3x intest_convert_tree, 2x intest_build_conda). Eachconda createinvokes the solver and extracts packages, taking ~10 seconds each.Solution
conda create --cloneinstead of raw file copyingWhy
conda create --cloneinstead of raw file copy?Raw file copying (cp, shutil.copytree, etc.) doesn't properly handle conda's prefix relocation:
conda-meta/*.jsonfiles contain hardcoded prefix pathsconda create --cloneproperly handles all of this while still being faster than a full create because it:Measured Performance Improvement
Changes
tests/conftest.py:python_template_envsession-scoped fixture usingsession_conda_clitests/test_benchmarks.py:python_template_envfixtureconda create --clonenonlocalfor setup counters instead of list workaroundTest plan