Skip to content

ci(tests[benchmarks]) Add CodSpeed performance benchmarks#525

Open
tony wants to merge 1 commit intomasterfrom
feat/codspeed-benchmarks
Open

ci(tests[benchmarks]) Add CodSpeed performance benchmarks#525
tony wants to merge 1 commit intomasterfrom
feat/codspeed-benchmarks

Conversation

@tony
Copy link
Copy Markdown
Member

@tony tony commented Apr 26, 2026

Summary

  • Adds a CodSpeed-backed CI benchmark job for tracking GitSync.obtain (and future hot paths) performance over time
  • Functional-style benchmark test in tests/benchmarks/ (no class TestX: patterns)
  • Benchmarks are deselected by default via the benchmark marker; opt-in with pytest tests/benchmarks --codspeed -m benchmark
  • Replaces the long-stalled poetry-era Codspeed - performance benchmarks #471 with a uv-native implementation against current master

Changes

  • pyproject.toml
    • Add pytest-codspeed to the testing dependency group
    • Register the benchmark marker
    • Default addopts adds -m 'not benchmark' so regular uv run py.test skips benchmarks
    • mypy override to ignore missing pytest_codspeed.* stubs
  • tests/benchmarks/test_sync_git.py — single representative benchmark for GitSync.obtain against an initial-commit remote (~49ms locally)
  • .github/workflows/tests.yml — new benchmarks job running CodSpeedHQ/action@v3 on pull requests and master pushes
  • CHANGES — entry under 0.41.x (unreleased)

Test plan

  • uv run ruff check . — clean
  • uv run ruff format . — clean
  • uv run mypy — clean
  • uv run py.test --reruns 0 -vvv — 615 passed, 1 skipped, 1 deselected (the new benchmark, as designed)
  • just build-docs — succeeds
  • Local benchmark run (uv run --group testing py.test tests/benchmarks --codspeed -m benchmark) — test_git_obtain_initial_commit_repo 48.8ms median
  • CI `benchmarks` job runs on PR (requires `CODSPEED_TOKEN` repo secret)

Supersedes #471 (which targeted the poetry-era pyproject.toml; closed).

why: Track GitSync.obtain (and future hot paths) for performance regressions in
CI. Replaces the long-stalled poetry-era #471 with a uv-native, functional-tests
implementation against current master.

what:
- pyproject.toml: add `pytest-codspeed` to `testing` group; register a
  `benchmark` marker; deselect benchmarks by default via `addopts`; let mypy
  ignore missing `pytest_codspeed.*` stubs
- tests/benchmarks/test_sync_git.py: add functional-style benchmark for
  `GitSync.obtain` against an initial-commit remote
- .github/workflows/tests.yml: add `benchmarks` job using
  `CodSpeedHQ/action@v3` on pull requests and master pushes
- CHANGES: note the benchmark addition under 0.41.x
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 26, 2026

Codecov Report

❌ Patch coverage is 60.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.23%. Comparing base (d9391c3) to head (90af158).

Files with missing lines Patch % Lines
tests/benchmarks/test_sync_git.py 60.00% 4 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master     #525   +/-   ##
=======================================
  Coverage   58.23%   58.23%           
=======================================
  Files          40       41    +1     
  Lines        6489     6499   +10     
  Branches     1098     1098           
=======================================
+ Hits         3779     3785    +6     
- Misses       2177     2181    +4     
  Partials      533      533           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant