Skip to content

ci: retry artifact downloads once on transient infra failures - #3037

Open
juenglin wants to merge 2 commits into
NVIDIA:mainfrom
juenglin:artifact-download-retry
Open

juenglin wants to merge 2 commits into
NVIDIA:mainfrom
juenglin:artifact-download-retry

Conversation

@juenglin

@juenglin juenglin commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

This is the first of two PRs. It adds a local composite action (.github/actions/download-artifact-retry) that wraps the pinned actions/download-artifact@v8.0.1 with a single retry. When the first download attempt fails due to a transient infrastructure error (e.g. "Unable to download artifact(s): other side closed", as seen in run 37529061189 job 112495868687 on the linux-aarch64 3.14t lane), the action waits 15 s and tries again. The job fails only if the second attempt also fails.

In this PR the action is adopted by the 4 artifact-download steps in the sdist test workflows:

  • test-sdist-linux.yml (2 steps)
  • test-sdist-windows.yml (2 steps)

The sdist jobs check out the workflow commit, so a local ./.github/actions/... reference resolves correctly.

Follow-up PR

The 14 download steps in test-wheel-linux.yml and test-wheel-windows.yml are intentionally left unchanged here. The nightly workflow passes inputs.sha (the head of the last successful main CI run), and the wheel jobs check out that commit, which can predate the new action. A local ./ reference would therefore fail with a missing action.yml until a main build contains the action. Once this PR is merged, a second PR will point the wheel steps at NVIDIA/cuda-python/.github/actions/download-artifact-retry@<merged-sha>, which GitHub fetches directly from that commit.

The "prior branch" download steps that use gh run download are unchanged. The remaining steps in build-wheel.yml, build-docs.yml, and coverage.yml can adopt the action later.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Add a local composite action (.github/actions/download-artifact-retry)
that wraps actions/download-artifact@v8.0.1 with a single retry.  When
the first download attempt fails (e.g. "other side closed" network
flakes), the action waits 15 s and tries again; the job fails only if
the second attempt also fails.

Apply the composite action to the 18 artifact-download steps in the
four test workflows:
- test-wheel-linux.yml  (7 steps)
- test-wheel-windows.yml (7 steps)
- test-sdist-linux.yml  (2 steps)
- test-sdist-windows.yml (2 steps)

The "prior branch" download steps use gh and are unchanged.
The remaining 16 steps in build-wheel.yml, build-docs.yml, and
coverage.yml can adopt the action in a follow-up.
@juenglin juenglin added this to the cuda.core 2.0.0 milestone Oct 6, 2026
@juenglin juenglin added the CI/CD CI/CD infrastructure label Oct 6, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Oct 6, 2026
@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • Bug Fixes
    • Improved reliability of build artifact downloads in Linux and Windows test workflows. Failed downloads are automatically retried after a short delay.

Walkthrough

A composite GitHub Action now retries failed artifact downloads once after a configurable delay. Linux and Windows source-distribution and wheel test workflows use the action in place of direct artifact-download steps.

Changes

Artifact download retry

Layer / File(s) Summary
Retry action and download flow
.github/actions/download-artifact-retry/action.yml
The composite action accepts artifact name and path inputs, plus optional run ID, token, and retry delay. It warns and waits after an initial failure, then retries once.
Test workflow integration
.github/workflows/test-sdist-linux.yml, .github/workflows/test-sdist-windows.yml, .github/workflows/test-wheel-linux.yml, .github/workflows/test-wheel-windows.yml
The workflows use the local action for the existing artifact downloads. Their artifact configuration and conditions remain unchanged.

Suggested reviewers: leofang

Priority: ⬇️ Low

Change: Bug fix

Merge Risk: 🟡 Moderate · up to 8deb1

Nightly wheel tests that target an older commit can fail at the first artifact download because the new retry action does not exist at that commit. Resolve the action from the workflow commit before merging.

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 484d389c-e5ff-469a-9e96-8fef91bdd900
📥 Commits

Reviewing files that changed from the base of the PR and between 4c893de and ba0f43e.

📒 Files selected for processing (5)
  • .github/actions/download-artifact-retry/action.yml
  • .github/workflows/test-sdist-linux.yml
  • .github/workflows/test-sdist-windows.yml
  • .github/workflows/test-wheel-linux.yml
  • .github/workflows/test-wheel-windows.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Comment thread .github/workflows/test-wheel-linux.yml Outdated
The nightly workflow checks out an older wheel commit (inputs.sha), so a
local ./.github/actions reference would not resolve until a main build
contains the new action.  Keep the wheel test workflows on
actions/download-artifact for now; a follow-up PR will point them at the
action by commit SHA.
@juenglin

juenglin commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8deb152

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants