Skip to content

feat(ci): add opt-in parallel test-bucket jobs to python-ci.yml - #269

Merged
williaby merged 4 commits into
mainfrom
feat/parallel-test-buckets-opt-in
Jul 12, 2026
Merged

williaby merged 4 commits into
mainfrom
feat/parallel-test-buckets-opt-in

Conversation

@williaby

Copy link
Copy Markdown
Collaborator

Description

quality-checks currently runs unit, integration, and security pytest buckets as sequential steps inside a single job. For a repo with a sizable integration suite (e.g. cyo-adventure: ~5.5 min of integration tests alone), this puts several minutes of serial pytest on the CI critical path even though the three buckets are independent and could run concurrently.

This adds an opt-in parallel-tests input (default false) that splits those buckets into three parallel jobs — test-unit, test-integration, test-security — plus a coverage-combine job that merges their coverage data and republishes the same coverage-reports artifact (same name, same file layout) that quality-checks produces today, so callers that already download that artifact need no changes.

Related Issue

N/A — internal CI performance investigation for cyo-adventure.

Type of Change

  • Bug fix
  • New feature (non-breaking, opt-in)
  • Breaking change
  • Documentation update
  • Refactoring
  • Performance improvement
  • Test addition/update
  • CI/CD update
  • Dependencies update

Changes Made

  • New parallel-tests boolean input (default false).
  • New jobs test-unit, test-integration, test-security: each does its own checkout/install and runs one pytest bucket, writing coverage to a job-scoped .coverage.<type> data file.
  • New coverage-combine job: downloads all three coverage-data artifacts, runs coverage combine + coverage report --fail-under, and uploads the merged result as the coverage-reports artifact (matching the non-parallel path's name and file set exactly).
  • quality-checks's own unit/integration/security/coverage-combine/upload steps are now gated && !inputs.parallel-tests, so it becomes lint/format/typecheck/Vulture/Bandit/pip-audit only when the flag is on.
  • ci-gate now also depends on the four new jobs, using the same success or skipped acceptance already used for matrix-testing — when parallel-tests is unset, they report skipped and the gate is unaffected.

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have performed manual testing

Test Details

  • Validated the full file parses as YAML and that every job/if: expression decodes to the intended string (python3 -c "yaml.safe_load(...)").
  • Diffed against the pre-change file to confirm the non-parallel path is touched in only 6 lines (5 step-level if: gates gaining && !inputs.parallel-tests, plus ci-gate's needs: list) — everything else is additive, so existing callers with the input unset get byte-identical behavior.
  • Not yet validated against a real workflow run — actionlint wasn't available in the sandbox this was authored in. cyo-adventure is opting in as the first real-world test; see companion PR there.
  • Known open risk (documented inline in the workflow): the cross-job coverage combine step depends on the caller's pyproject.toml setting [tool.coverage.run] relative_files = true so coverage data recorded on three separate runners merges correctly. cyo-adventure already sets this. Callers that opt in without it may see an inaccurate combined coverage percentage.

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings or errors (YAML validated)
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published
  • I have checked my code and corrected any misspellings
  • I have signed off my commits with git commit --signoff (DCO)

Reviewer Notes

Recommend merging this once cyo-adventure's opt-in run confirms the coverage-combine step produces an accurate merged coverage number and the wall-clock improvement materializes as expected (~11 min → ~6-7 min was the estimate). Happy to iterate on the job split if review turns up a cleaner structure.


Generated by Claude Code

quality-checks currently runs unit, integration, and security pytest
buckets as sequential steps in a single job; for repos with a large
integration suite this puts several minutes of serial pytest on the
critical path. Add an opt-in `parallel-tests` input (default false) that
splits those buckets into three independent jobs (test-unit,
test-integration, test-security) plus a coverage-combine job that merges
their coverage data and republishes the same coverage-reports artifact
downstream callers already consume, so no caller needs to change to pick
up the artifact.

Default-off: when parallel-tests is unset, quality-checks behaves
byte-identically to before (5 step-level `if:` conditions gain `&&
!inputs.parallel-tests`, which is a no-op when the input is false), and
the new jobs report skipped, which ci-gate already treats as acceptable
for optional jobs (matching the existing enable-matrix-testing pattern).

Signed-off-by: Claude <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 12, 2026 14:36
@coderabbitai

coderabbitai Bot commented Jul 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@williaby, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: da2ab335-3e73-4297-95de-884c9dee40ac

📥 Commits

Reviewing files that changed from the base of the PR and between ea23b47 and 0241ecf.

📒 Files selected for processing (3)
  • .github/workflows/python-ci.yml
  • CHANGELOG.md
  • docs/workflows/python-ci.md
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/parallel-test-buckets-opt-in

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an opt-in parallel-tests mode to the reusable python-ci.yml workflow to reduce CI wall clock time by splitting unit, integration, and security pytest buckets into parallel jobs, then recombining coverage into the same coverage-reports artifact consumers already expect.

Changes:

  • Added a new parallel-tests boolean input (default false) to enable parallelized test execution.
  • Introduced test-unit, test-integration, test-security jobs that run pytest buckets independently and publish per-bucket coverage data artifacts.
  • Added a coverage-combine job to download per-bucket artifacts, run coverage combine, and re-upload the merged coverage-reports artifact, plus updated ci-gate to include the new jobs.

Comment on lines +750 to +755
if: |
always() && inputs.parallel-tests &&
needs.test-unit.result == 'success' &&
(needs.test-integration.result == 'success' || needs.test-integration.result == 'skipped') &&
(needs.test-security.result == 'success' || needs.test-security.result == 'skipped')
permissions:
Comment thread .github/workflows/python-ci.yml Outdated
Comment on lines +779 to +781
- name: Install dependencies
run: uv sync --all-extras --frozen $NO_BUILD_FLAG
shell: bash
Comment on lines +114 to +122
parallel-tests:
description: >-
Opt-in: run the unit/integration/security test buckets as separate
parallel jobs (test-unit, test-integration, test-security) instead
of sequential steps inside quality-checks, with a coverage-combine
job producing the same coverage-reports artifact downstream callers
already expect. Default false preserves the existing single-job
behavior for all other callers.
required: false
claude added 2 commits July 12, 2026 14:44
Qlty/actionlint (shellcheck SC2086) and SonarCloud (S8541) both flagged
the new-code $NO_BUILD_FLAG/$GITHUB_OUTPUT usages introduced in
test-unit/test-integration/test-security/coverage-combine: these are
verbatim copies of quality-checks' own patterns, but the accompanying
suppression comments only carried over for the branches that already had
them there, not for every new occurrence. Quote $GITHUB_OUTPUT (always
safe) and add the shellcheck disable + NOSONAR(S8541) comments the
un-quoted $NO_BUILD_FLAG word-splitting requires, verified against a real
shellcheck binary rather than guessed.

Also fixes two real bugs Copilot's review caught in the new jobs:

- coverage-combine could run even when the caller repo has no
  pyproject.toml: test-unit's own steps all no-op in that case (state=
  skip), so it reports success with no artifact uploaded, and
  coverage-combine's download-artifact step would fail trying to fetch
  it. test-unit now exposes its detected repo-state as a job output;
  coverage-combine's `if:` gates on it not being 'skip', matching
  quality-checks' existing graceful-skip behavior.
- coverage-combine always ran `uv sync --frozen`, which requires a
  committed uv.lock; the workflow explicitly supports uv-managed callers
  without one (uv-no-lock). coverage-combine now runs its own lockfile
  detection and branches the install the same way every other job here
  does.

Also documents the new `parallel-tests` input and its
`relative_files = true` caller requirement in docs/workflows/python-ci.md
(previously undocumented per review).

Signed-off-by: Claude <noreply@anthropic.com>
The previous commit put NOSONAR on its own line preceding the flagged
command (to sit alongside the shellcheck disable directive, which must
precede). SonarCloud's githubactions plugin evidently only recognizes
NOSONAR as a trailing same-line comment on the flagged line itself: the
one pre-existing pattern in this file that is genuinely clean on main
(quality-checks' "Install dependencies (uv without lockfile)" one-liner)
uses a trailing comment, and every occurrence I added with a preceding
comment was still flagged as a fresh vulnerability on this PR's diff
despite the suppression text being present.

Keep `# shellcheck disable=SC2086` on its own preceding line (shellcheck
requires this) and move `# NOSONAR(...)` back onto the same line as the
command it annotates, matching the codebase's only confirmed-working
example. Verified against a real shellcheck binary that this ordering
still satisfies SC2086 with zero warnings across all 15 affected lines.

Signed-off-by: Claude <noreply@anthropic.com>
@williaby

williaby commented Jul 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

PR Review

PREMISE QUESTION: Feature is justified and opt-in, but triplicates ~480 lines of job boilerplate that this file's own matrix-testing job pattern (strategy.matrix) would collapse into one job definition. Worth deciding before consumers opt in; refactoring a shared workflow afterward is much harder.

Critical (must fix before merge)

  1. NOSONAR placement is the documented-broken pattern. The latest commit moved NOSONAR comments to inline-trailing position inside run: | block scalars, which docs/sonarcloud-nosonar-patterns.md documents as NOT honored by SonarCloud (proven in PR fix(python-sonarcloud): detect repo state #166, commit bb31ec1). Confirmed empirically: SonarCloud re-analyzed this head and still reports 16 open issues (12x S8541, 4x S8544), quality gate ERROR. Affected lines include python-ci.yml#L523-L540 and python-ci.yml#L862-L868. Fix with Pattern A (single-line run: form) or Pattern B (preceding line, single rule) from the doc.

  2. Em-dash in docs. docs/workflows/python-ci.md#L85 ships a literal U+2014, violating the hard no-em-dash rule. It passed CI because scripts/check-no-em-dash.sh scans git diff --cached and runs vacuously under pre-commit run --all-files (pre-existing hook gap, follow-up issue recommended separately).

Important (should fix)

  1. CHANGELOG.md not updated for a feat(ci) change.
  2. relative_files = true prerequisite is documented but unenforced. A caller opting in without it gets a green gate with silently under-counted merged coverage. Add a preflight check in coverage-combine.
  3. coverage combine invocation diverges from the sequential path. python-ci.yml#L862 drops the --keep flag and || true fallback used at python-ci.yml#L390.
  4. Missing fetch-depth: 0 in all four new jobs' checkouts (L490, L581, L675, L794) while quality-checks uses it; setuptools-scm/hatch-vcs callers behave differently only under parallel-tests: true, contradicting the "mirrors quality-checks" comment.
  5. Missing "Validate source layout" fail-fast step in the new parallel jobs; bad source config fails deep inside pytest only in parallel mode.
  6. Asymmetric artifact upload on failure: coverage-combine uploads coverage-reports with if: always() but the sequential path only on success. Confirm intent and align.

SonarQube: 16 issues queued for auto-fix (all resolved by Critical item 1).

Copilot review received (3 comments); all were already addressed at 0d02c92. CodeRabbit check is green but no review ran (rate-limited).

🤖 Generated with Claude Code

The org's own docs/sonarcloud-nosonar-patterns.md (surfaced via review)
documents this exact failure precisely: SonarCloud's githubactions
ruleset only reliably honors NOSONAR as a trailing comment on a genuine
single-line YAML `run:` scalar (Pattern A), not a preceding-line comment
inside a `run: |` block (Pattern B is reliable only for multi-line
backslash-continued commands with a literal --frozen, e.g. the pytest
steps, which is why those were never flagged across any of this
feature's prior commits). The previous commit used a preceding-line
comment for the single-line install/coverage commands, an anti-pattern
the doc traces back to PR #166 commit bb31ec1.

Rebuild every affected `uv sync`/`uv run` line as a bash array
(NO_BUILD_ARGS=(); [ -n "$NO_BUILD_FLAG" ] && NO_BUILD_ARGS+=(...); ...
"${NO_BUILD_ARGS[@]}") on a single physical `run:` line with a trailing
NOSONAR comment. This satisfies Pattern A for SonarCloud while staying
shellcheck-clean without needing a `# shellcheck disable=SC2086` comment
(which itself requires its own preceding line and so cannot coexist with
a single-line run:). Verified against a real shellcheck binary, and that
the array construction survives `set -e` when the flag is empty (GitHub
Actions bash steps run with -e by default).

Also addresses the review's other findings:
- Fix the literal em-dash in docs/workflows/python-ci.md (hard project
  rule).
- Add a CHANGELOG.md entry under [Unreleased] for the new input.
- Add `fetch-depth: 0` and a "Validate source layout" step to the three
  new test-* jobs, matching quality-checks (previously only
  coverage-combine and quality-checks had these).
- coverage-combine's coverage-combine step now uses --keep and || true,
  matching quality-checks' own combine step.
- coverage-combine now runs its own "Verify relative_files is configured"
  preflight (parses pyproject.toml with tomllib) and fails fast with an
  actionable error instead of silently under-counting coverage when a
  caller opts in without relative_files = true.
- Drop coverage-combine's final upload step's `if: always()` to match
  quality-checks' upload step (no always(); only runs on success).

Not addressed here (raised as a design question, not a bug): the
review's premise question about collapsing test-unit/test-integration/
test-security into one `strategy.matrix` job. That's a real
simplification but changes the job/check-name surface this feature
exposes to callers; deferring to a follow-up discussion rather than
folding into this already-large diff.

Signed-off-by: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Collaborator Author

Pushed 0241ecf addressing all Critical/Important items:

  1. NOSONAR placement: rebuilt every affected uv sync/uv run line as a single-line run: with a bash-array flag (NO_BUILD_ARGS=(); [ -n "$NO_BUILD_FLAG" ] && NO_BUILD_ARGS+=(...); ... "${NO_BUILD_ARGS[@]}") and trailing NOSONAR — Pattern A per docs/sonarcloud-nosonar-patterns.md. The array avoids needing a shellcheck disable comment (which requires its own preceding line and can't coexist with a single-line run:), so it satisfies both SonarCloud and qlty/shellcheck at once. Verified against a real shellcheck binary, including that the array construction survives set -e when the flag is empty.
  2. Em-dash: fixed.
  3. CHANGELOG.md: added under [Unreleased].
  4. relative_files unenforced: coverage-combine now has a preflight step that parses pyproject.toml with tomllib and fails fast with an actionable error if it's missing, instead of silently under-counting.
  5. coverage combine divergence: added --keep and || true to match quality-checks.
  6. Missing fetch-depth: 0: added to all three test-* jobs' checkouts, plus a "Validate source layout" step matching quality-checks.
  7. Folded into fix: resolve CI workflow startup and configuration failures #6.
  8. Asymmetric artifact upload: dropped if: always() from coverage-combine's upload step to match quality-checks.

On the premise question (collapsing test-unit/test-integration/test-security into one strategy.matrix job): agreed it's a real simplification, but I deliberately didn't fold it into this diff — it changes the job/check-name surface (Unit Tests/Integration Tests/Security Tests as distinct required-check names vs. one matrixed Test (unit)/Test (integration)/Test (security) job), which is a caller-facing decision, not just an internal refactor. Want me to do that refactor now before this merges, or land this first and revisit as a follow-up once the coverage-combine mechanics have proven out on a live caller?


Generated by Claude Code

@sonarqubecloud

Copy link
Copy Markdown

@williaby
williaby added this pull request to the merge queue Jul 12, 2026
Merged via the queue into main with commit 24c0593 Jul 12, 2026
34 checks passed
@williaby
williaby deleted the feat/parallel-test-buckets-opt-in branch July 12, 2026 15:40
williaby pushed a commit to ByronWilliamsCPA/cyo-adventure that referenced this pull request Jul 12, 2026
Was pinned to a commit on the feature branch to trial parallel-tests
before ByronWilliamsCPA/.github#269 merged (per the #VERIFY note it
carried). Now points at 24c0593, the squash-merge commit on main, so CI
no longer depends on the feature branch continuing to exist.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants