Skip to content

fix(ci): pass --cov-fail-under=0 in parallel-tests buckets - #270

Merged
williaby merged 1 commit into
mainfrom
fix/parallel-tests-per-bucket-cov-fail-under
Jul 12, 2026
Merged

williaby merged 1 commit into
mainfrom
fix/parallel-tests-per-bucket-cov-fail-under

Conversation

@williaby

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #269. The opt-in parallel-tests jobs (test-unit, test-integration, test-security) each run pytest on their own marker slice on a separate runner, inheriting the caller's pyproject.toml addopts. When those addopts include a whole-suite --cov-fail-under, a bucket that only exercises part of the tree false-fails the threshold, even though the combined coverage clears it.

This passes --cov-fail-under=0 in all three bucket pytest invocations so per-bucket runs never apply a whole-suite gate. The real threshold is unchanged: coverage-combine enforces coverage-threshold once, after merging the three .coverage.* files.

Why

cyo-adventure is the first real caller of the parallel-tests opt-in. Its test-security bucket failed:

ERROR: Coverage failure: total of 52.02 is less than fail-under=80.00
FAIL Required test coverage of 80% not reached. Total coverage: 52.02%

The security marker covers ~52% of the tree on its own; the repo's pyproject.toml sets --cov-fail-under=80 in [tool.pytest.ini_options] addopts, which the bucket inherited.

Scope

  • Only the three parallel bucket "Run ... tests with coverage" steps change (15 lines: a flag plus an explanatory comment each).
  • The sequential quality-checks path is deliberately untouched: it accumulates coverage across unit/integration/security steps in one job, so the whole-suite threshold is correct there.
  • No input or interface change; existing callers unaffected.

Testing

CI on this PR exercises the sequential path (this repo does not set parallel-tests). The parallel path is validated by re-pinning cyo-adventure's caller to this PR's merge SHA and confirming test-security/coverage-combine go green with the same combined coverage percentage as a sequential run.

Generated with Claude Code

The opt-in parallel-tests jobs (test-unit/test-integration/test-security,
added in #269) each run pytest on their own marker slice on a separate
runner. They inherit the caller's pyproject addopts, which commonly include a
whole-suite --cov-fail-under. A bucket that exercises only part of the tree
(e.g. security tests covering ~50%) then false-fails the whole-suite gate,
even though combined coverage clears it.

Pass --cov-fail-under=0 in all three bucket pytest invocations so the
per-bucket runs never apply a whole-suite threshold. The real gate is
unchanged: coverage-combine enforces coverage-threshold once, after merging
the three .coverage.* files. The sequential quality-checks path is untouched
(it accumulates coverage across steps in one job, so the threshold is correct
there).

Surfaced by cyo-adventure, the first real caller of the parallel-tests
opt-in: its test-security bucket failed at 52.02% against the repo's 80% gate.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 12, 2026 16:12
@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@williaby, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 778950ba-4b70-4875-9639-9f3e2b2809df

📥 Commits

Reviewing files that changed from the base of the PR and between 24c0593 and dae3584.

📒 Files selected for processing (3)
  • .github/workflows/python-ci.yml
  • CHANGELOG.md
  • docs/workflows/python-ci.md
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/parallel-tests-per-bucket-cov-fail-under

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@williaby
williaby added this pull request to the merge queue Jul 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates the reusable python-ci.yml workflow’s opt-in parallel-tests path so that each individual pytest bucket run (unit, integration, security) explicitly disables per-bucket coverage gating by passing --cov-fail-under=0. This prevents false failures when a caller configures a whole-suite --cov-fail-under in pyproject.toml addopts, while keeping the real coverage threshold enforcement unchanged in coverage-combine after merging coverage data.

Changes:

  • Add --cov-fail-under=0 to the test-unit, test-integration, and test-security pytest invocations, with explanatory inline comments.
  • Update docs/workflows/python-ci.md to document the per-bucket override and the post-merge enforcement behavior.
  • Add a changelog entry describing the fix and why it is needed for real-world callers using parallel-tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
.github/workflows/python-ci.yml Ensures per-bucket pytest runs never enforce whole-suite coverage thresholds, leaving enforcement to coverage-combine.
docs/workflows/python-ci.md Documents the coverage gating behavior for parallel-tests callers and explains the --cov-fail-under=0 override.
CHANGELOG.md Records the behavioral fix for parallel per-bucket jobs and clarifies the unchanged post-merge threshold enforcement.

Merged via the queue into main with commit c4ba2d1 Jul 12, 2026
35 checks passed
@williaby
williaby deleted the fix/parallel-tests-per-bucket-cov-fail-under branch July 12, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants