CI: Consolidate the two release workflows

[leofang]

At certain point in time (#1626) the release workflow was rage-forked into two

• https://github.com/NVIDIA/cuda-python/blob/main/.github/workflows/release-cuda-pathfinder.yml (new, for pathfinder only)
• https://github.com/NVIDIA/cuda-python/blob/main/.github/workflows/release.yml (old, for all components)

with the understanding that eventually we want to join them. But it never happened after the team moved on. It's time to get it cleaned up and organized.

[jpascucci-nv]

Proposed consolidation plan

release.yml already exposes cuda-pathfinder as a component, so consolidation should move the dedicated workflow's additional release policy into the shared graph. Simply deleting release-cuda-pathfinder.yml would lose verification, concurrency, release metadata, and automatic finalization.

Target flow

tag/input validation --+-- determine exact tag-triggered CI run
                       +-- check release notes
                       +-- create/reuse draft
                                   |
                                 docs
                          +--------+--------+
                     upload assets      TestPyPI
                          |                  |
                          |       [Pathfinder verify]
                          |                  |
                          |                PyPI
                          |                  |
                          |       [Pathfinder verify]
                          +--------------+---+
                               [Pathfinder finalize]

For cuda-core, cuda-bindings, and cuda-python, the bracketed jobs remain skipped and the GitHub release remains a draft, preserving today's behavior.

1. Keep the shared workflow interface

A full Pathfinder release should use:

• component: cuda-pathfinder
• release-action: full-release
• git-tag: cuda-pathfinder-vX.Y.Z
• empty run-id, backport-git-tag, and dry-run-docs-branch

For full Pathfinder releases only:

• Require dispatch from the default branch.
• Reject a supplied run-id, or validate that it is a successful tag-push run for the exact tag SHA with the expected artifacts. Rejecting it is simpler and matches the dedicated workflow.
• Serialize runs under release-cuda-pathfinder with cancel-in-progress: false.

Do not impose those restrictions on the other components: maintenance/backport releases may intentionally use other workflow branches or explicit run IDs.

2. Preserve Pathfinder draft behavior

For Pathfinder, draft preparation should:

• Resolve an exact refs/tags/<tag> ref.
• Use gh release view "$TAG" rather than searching a potentially limited release list.
• Reject an already-published release and reuse an existing draft.
• Create a missing draft with --verify-tag, --latest=false, title cuda-pathfinder vX.Y.Z, and the existing release-notes, versioned-docs, PyPI, and Conda links.

Other components should retain the current generic body and manual publication. Tag, notes, and CI-run validation should finish before draft creation so invalid requests do not leave orphan drafts.

3. Reuse the generalized machinery

The shared implementation already supports Pathfinder:

• ci/tools/check_release_notes.py knows the Pathfinder tag namespace and notes path.
• build-docs.yml builds Pathfinder docs from the selected tag/run.
• release-upload.yml creates the source archive/checksum, downloads wheels, supports dry-run artifacts, and uploads release assets.
• ci/tools/download-wheels collects Pathfinder artifacts.
• ci/tools/validate-release-wheels checks the exact version and rejects development/local wheels.
• release.yml already owns the TestPyPI and PyPI OIDC publishing jobs.

These should replace the dedicated workflow's duplicated artifact-download/upload blocks. Keep the actual pypa/gh-action-pypi-publish steps directly in top-level release.yml; see the OIDC note below.

4. Preserve Pathfinder's docs-failure policy

upload-archive and publish-testpypi may continue after doc == failure only when all tag, notes, and run-ID prerequisites succeeded and the run is a full Pathfinder release. Cancellation must remain blocking. Every other component must continue to require doc == success.

This requires explicit !cancelled() and needs.<job>.result checks; otherwise GitHub propagates a failed/skipped dependency before the product exception is evaluated.

5. Add reusable registry verification

Add one small helper such as ci/tools/verify-published-package, called by separate verify-testpypi and verify-pypi jobs. Preserve:

• A fresh virtual environment.
• Six installation attempts with 30-second delays.
• Exact cuda-pathfinder==X.Y.Z installation.
• Successful cuda.pathfinder import.
• Exact __version__ comparison.
• No package cache while checking registry propagation.

Keep verification separate from publishing. If registry propagation is slow, rerunning verification must not retry an immutable package upload.

publish-pypi must explicitly tolerate the Pathfinder-only TestPyPI verifier being skipped for other products:

publish-pypi:
  needs:
    - determine-run-id
    - publish-testpypi
    - verify-testpypi
  if: >-
    ${{
      !cancelled() &&
      inputs.release-action == 'full-release' &&
      needs.determine-run-id.result == 'success' &&
      needs.publish-testpypi.result == 'success' &&
      (
        inputs.component != 'cuda-pathfinder' ||
        needs.verify-testpypi.result == 'success'
      )
    }}

Without this explicit condition, adding a skipped verifier to needs would also skip PyPI publication for every non-Pathfinder component.

6. Add Pathfinder-only finalization

After both verify-pypi and upload-archive succeed:

• Check the versioned docs URL; HTTP and transport failures should be warnings.
• Confirm the GitHub release is still a draft.
• Run gh release edit --draft=false --latest=false.
• Write the docs URL to the job summary.

Do not generalize finalization to other components; doing so would materially change their release process.

7. Harden shared input, permissions, and concurrency handling

• Pass every free-form dispatch value through env and use quoted shell variables. Do not inject ${{ inputs.git-tag }}, ${{ inputs.run-id }}, etc. directly into run: source.
• Qualify release source checkouts as refs/tags/....
• Grant actions: read where cross-run artifacts are read, contents: read for checkout, contents: write only for draft/assets/finalization, and id-token: write only for registry publishing.
• Validate wheels before uploading any GitHub assets, avoiding partially populated drafts after an artifact failure.

release-upload.yml currently groups concurrency by workflow and branch and cancels in progress. Since releases are normally dispatched from main, unrelated products/tags can cancel one another's uploader. Key it by component and tag instead, for example:

group: ${{ github.workflow }}-release-upload-${{ inputs.component }}-${{ inputs.git-tag }}

Prefer cancel-in-progress: false for release mutations. At minimum, the new key removes unintended cross-product cancellation.

8. Migrate trusted-publisher configuration before cutover

PyPI and TestPyPI Trusted Publishers bind authorization to the exact workflow filename and optionally the GitHub environment. Before deleting the dedicated workflow:

1. Register .github/workflows/release.yml for cuda-pathfinder on TestPyPI and PyPI.
2. Keep the existing testpypi and pypi environment names and protections.
3. Retain the old publisher records during the transition.
4. Remove the old release-cuda-pathfinder.yml publishers only after a successful unified release.

PyPI documents the workflow-filename requirement and says reusable workflows cannot currently be registered as Trusted Publishers. Therefore, moving the actual publishing action into another reusable workflow would be the wrong kind of deduplication.

9. Resolve the Pathfinder post-release version edge case

The shared notes checker skips new notes for .postN releases, while the dedicated workflow requires a literal .postN-notes.rst. Pathfinder docs intentionally publish post releases under the base X.Y.Z directory, while the existing release body uses the full version. The integration should explicitly maintain separate package and docs versions so generated docs URLs are correct. Alpha/beta policy was deferred in #1626 and should remain outside this issue.

10. Validation and rollout

1. Run actionlint via pre-commit and unit-test the new verification helper.
2. Exercise the existing release.yml dry-run matrix for mainline bindings, bindings/Python backports, and mainline Python where relevant.
3. Add a Pathfinder dry run using a real successful tag-triggered CI run, leaving run-id empty.
4. Test wrong-prefix/nonexistent tags, non-default-branch full release, supplied Pathfinder run ID, existing published release, and manually undrafted release.
5. Verify that TestPyPI failure blocks PyPI, and that PyPI verification or asset failure leaves the GitHub release as a draft.
6. Verify that docs failure remains nonblocking only for Pathfinder.
7. Verify that concurrent Pathfinder releases serialize and that another product cannot cancel the Pathfinder uploader.
8. Prove OIDC, propagation checks, and finalization during the next scheduled Pathfinder release.

After successful cutover, delete .github/workflows/release-cuda-pathfinder.yml and update the stale reference in .github/RELEASE-core.md.