Skip to content

Always build and run Cython tests + other CI improvements #640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
May 19, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
c41a39a
try to also build cuda.bindings Cython tests at the build stage
leofang May 16, 2025
227e315
we can no longer lock host Python version
leofang May 17, 2025
2b38518
check if this hack from the test workflow also fixes cibuildwheel
leofang May 17, 2025
9fc20e1
also install test deps
leofang May 17, 2025
512e613
AGENT_TOOLSDIRECTORY needs to be writable
leofang May 17, 2025
de4db55
test
leofang May 17, 2025
be78f1b
test
leofang May 17, 2025
a0f964d
test
leofang May 17, 2025
19e5709
ensure AGENT_TOOLSDIRECTORY is persistent
leofang May 17, 2025
78c8afd
do this instead
leofang May 17, 2025
c626592
fixes
leofang May 17, 2025
dd86fe4
fix windows
leofang May 17, 2025
a13f0ef
fix
leofang May 17, 2025
4af05da
fix dir
leofang May 17, 2025
8592024
extra dot
leofang May 17, 2025
27418ef
ensure the Python include path is visible
leofang May 18, 2025
254748b
fix python include path; only install essential build deps; add tests
leofang May 18, 2025
ba0711b
fix escape; revert test dep installation handling
leofang May 18, 2025
0397b88
misc fixes
leofang May 18, 2025
3c3899c
fix win artifact location; pin action commits; enable cuda.core cytho…
leofang May 18, 2025
0a9ae25
revert pinning ilammy/msvc-dev-cmd for now
leofang May 18, 2025
a524b8c
set up a simple dependabot
leofang May 18, 2025
3554334
fix spdx identifier
leofang May 18, 2025
8f7a054
add cuda.core include path
leofang May 18, 2025
9ad27ac
try this
leofang May 18, 2025
08c5579
again
leofang May 18, 2025
f589987
ensure cython tests are only run when testing against build CTK
leofang May 19, 2025
452fced
try to avoid Cython 3.1 to speed up build time
leofang May 19, 2025
15ac320
pin exact commit for all non-NV GHA
leofang May 19, 2025
d6ada33
move some reusable logics into build_tests.sh
leofang May 19, 2025
f5ffde9
fix path format, again
leofang May 19, 2025
f409d19
try enforcing double slashes
leofang May 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/actions/doc_preview/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ runs:
# Note: the PR previews will be removed once merged to main (see below)
- name: Deploy doc preview
if: ${{ github.ref_name != 'main' }}
uses: JamesIves/github-pages-deploy-action@v4
uses: JamesIves/github-pages-deploy-action@6c2d9db40f9296374acc17b90404b6e8864128c8 # v4.7.3
with:
git-config-name: cuda-python-bot
git-config-email: [email protected]
Expand All @@ -31,7 +31,7 @@ runs:

- name: Leave a comment after deployment
if: ${{ github.ref_name != 'main' }}
uses: marocchino/sticky-pull-request-comment@v2
uses: marocchino/sticky-pull-request-comment@67d0dec7b07ed060a405f9b2a64b8ab319fdd7db # v2.9.2
with:
header: pr-preview
number: ${{ inputs.pr-number }}
Expand All @@ -47,7 +47,7 @@ runs:
# The steps below are executed only when building on main.
- name: Remove doc preview
if: ${{ github.ref_name == 'main' }}
uses: JamesIves/github-pages-deploy-action@v4
uses: JamesIves/github-pages-deploy-action@6c2d9db40f9296374acc17b90404b6e8864128c8 # v4.7.3
with:
git-config-name: cuda-python-bot
git-config-email: [email protected]
Expand All @@ -57,7 +57,7 @@ runs:

- name: Leave a comment after removal
if: ${{ github.ref_name == 'main' }}
uses: marocchino/sticky-pull-request-comment@v2
uses: marocchino/sticky-pull-request-comment@67d0dec7b07ed060a405f9b2a64b8ab319fdd7db # v2.9.2
with:
header: pr-preview
number: ${{ inputs.pr-number }}
Expand Down
4 changes: 2 additions & 2 deletions .github/actions/fetch_ctk/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ runs:

- name: Download CTK cache
id: ctk-get-cache
uses: actions/cache/restore@v4
uses: actions/cache/restore@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
continue-on-error: true
with:
key: ${{ env.CTK_CACHE_KEY }}
Expand Down Expand Up @@ -123,7 +123,7 @@ runs:
- name: Upload CTK cache
if: ${{ always() &&
steps.ctk-get-cache.outputs.cache-hit != 'true' }}
uses: actions/cache/save@v4
uses: actions/cache/save@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
with:
key: ${{ env.CTK_CACHE_KEY }}
path: ./${{ env.CTK_CACHE_FILENAME }}
Expand Down
2 changes: 1 addition & 1 deletion .github/actions/get_pr_number/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ runs:

- name: Get PR data (main branch)
if: ${{ github.ref_name == 'main' }}
uses: actions/github-script@v7
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
id: get-pr-data
with:
script: |
Expand Down
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
#
# SPDX-License-Identifier: Apache-2.0

version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
4 changes: 2 additions & 2 deletions .github/workflows/backport.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
}}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Load branch name
id: get-branch
Expand All @@ -32,7 +32,7 @@ jobs:
echo "OLD_BRANCH=${OLD_BRANCH}" >> $GITHUB_ENV

- name: Create backport pull requests
uses: korthout/backport-action@v3
uses: korthout/backport-action@436145e922f9561fc5ea157ff406f21af2d6b363 # v3.2.0
with:
copy_assignees: true
copy_labels_pattern: true
Expand Down
18 changes: 9 additions & 9 deletions .github/workflows/build-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
shell: bash -el {0}
steps:
- name: Checkout ${{ github.event.repository.name }}
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
ref: ${{ inputs.git-tag }}
Expand All @@ -56,7 +56,7 @@ jobs:
# is resolved

- name: Set up miniforge
uses: conda-incubator/setup-miniconda@v3
uses: conda-incubator/setup-miniconda@505e6394dae86d6a5c7fbb6e3fb8938e3e863830 # v3.1.1
with:
activate-environment: cuda-python-docs
environment-file: ./cuda_python/docs/environment-docs.yml
Expand Down Expand Up @@ -103,7 +103,7 @@ jobs:
echo "CUDA_BINDINGS_ARTIFACTS_DIR=$(realpath "$REPO_DIR/cuda_bindings/dist")" >> $GITHUB_ENV

- name: Download cuda-python build artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: cuda-python-wheel
path: .
Expand All @@ -117,14 +117,14 @@ jobs:

- name: Download cuda.bindings build artifacts
if: ${{ !inputs.is-release }}
uses: actions/download-artifact@v4
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ env.CUDA_BINDINGS_ARTIFACT_NAME }}
path: ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}

- name: Download cuda.bindings build artifacts
if: ${{ inputs.is-release }}
uses: actions/download-artifact@v4
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
pattern: ${{ env.CUDA_BINDINGS_ARTIFACT_NAME }}
merge-multiple: true
Expand All @@ -139,14 +139,14 @@ jobs:

- name: Download cuda.core build artifacts
if: ${{ !inputs.is-release }}
uses: actions/download-artifact@v4
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}

- name: Download cuda.core build artifacts
if: ${{ inputs.is-release }}
uses: actions/download-artifact@v4
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
pattern: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
merge-multiple: true
Expand Down Expand Up @@ -221,7 +221,7 @@ jobs:

# TODO: Consider removing this step?
- name: Upload doc artifacts
uses: actions/upload-pages-artifact@v3
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3.0.1
with:
path: artifacts/
retention-days: 3
Expand All @@ -236,7 +236,7 @@ jobs:

- name: Deploy doc update
if: ${{ github.ref_name == 'main' || inputs.is-release }}
uses: JamesIves/github-pages-deploy-action@v4
uses: JamesIves/github-pages-deploy-action@6c2d9db40f9296374acc17b90404b6e8864128c8 # v4.7.3
with:
git-config-name: cuda-python-bot
git-config-email: [email protected]
Expand Down
67 changes: 56 additions & 11 deletions .github/workflows/build-wheel.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
#
# SPDX-License-Identifier: Apache-2.0

Expand Down Expand Up @@ -36,7 +36,7 @@ jobs:
(inputs.host-platform == 'win-64' && 'windows-2019') }}
steps:
- name: Checkout ${{ github.event.repository.name }}
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0

Expand All @@ -47,17 +47,16 @@ jobs:
if: ${{ inputs.host-platform != 'win-64' }}

- name: Set up Python
if: ${{ startsWith(inputs.host-platform, 'linux') }}
id: setup-python
uses: actions/setup-python@v5
id: setup-python1
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
# WAR: setup-python is not relocatable...
# WAR: setup-python is not relocatable, and cibuildwheel hard-wires to 3.12...
# see https://github.com/actions/setup-python/issues/871
python-version: "3.12"

- name: Set up MSVC
if: ${{ startsWith(inputs.host-platform, 'win') }}
uses: ilammy/msvc-dev-cmd@v1
uses: ilammy/msvc-dev-cmd@v1 # TODO: ask admin to allow pinning commits

- name: Set environment variables
run: |
Expand All @@ -76,12 +75,14 @@ jobs:
echo "CUDA_CORE_ARTIFACT_BASENAME=${CUDA_CORE_ARTIFACT_BASENAME}" >> $GITHUB_ENV
echo "CUDA_CORE_ARTIFACT_NAME=${CUDA_CORE_ARTIFACT_BASENAME}-${{ github.sha }}" >> $GITHUB_ENV
echo "CUDA_CORE_ARTIFACTS_DIR=$(realpath "$REPO_DIR/cuda_core/dist")" >> $GITHUB_ENV
echo "CUDA_CORE_CYTHON_TESTS_DIR=$(realpath "$REPO_DIR/cuda_core/tests/cython")" >> $GITHUB_ENV
CUDA_BINDINGS_ARTIFACT_BASENAME="cuda-bindings-python${PYTHON_VERSION_FORMATTED}-cuda${{ inputs.cuda-version }}-${{ inputs.host-platform }}"
echo "CUDA_BINDINGS_ARTIFACT_BASENAME=${CUDA_BINDINGS_ARTIFACT_BASENAME}" >> $GITHUB_ENV
echo "CUDA_BINDINGS_ARTIFACT_NAME=${CUDA_BINDINGS_ARTIFACT_BASENAME}-${{ github.sha }}" >> $GITHUB_ENV
echo "CUDA_BINDINGS_ARTIFACTS_DIR=$(realpath "$REPO_DIR/cuda_bindings/dist")" >> $GITHUB_ENV
echo "CUDA_BINDINGS_CYTHON_TESTS_DIR=$(realpath "$REPO_DIR/cuda_bindings/tests/cython")" >> $GITHUB_ENV
echo "CIBW_BUILD=${CIBW_BUILD}" >> $GITHUB_ENV

- name: Dump environment
run: |
env
Expand Down Expand Up @@ -112,7 +113,7 @@ jobs:
twine check ${{ env.CUDA_CORE_ARTIFACTS_DIR }}/*.whl

- name: Upload cuda.core build artifacts
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}/*.whl
Expand Down Expand Up @@ -159,7 +160,7 @@ jobs:
twine check ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}/*.whl

- name: Upload cuda.bindings build artifacts
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: ${{ env.CUDA_BINDINGS_ARTIFACT_NAME }}
path: ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}/*.whl
Expand Down Expand Up @@ -187,8 +188,52 @@ jobs:

- name: Upload cuda-python build artifacts
if: ${{ strategy.job-index == 0 && inputs.host-platform == 'linux-64' }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: cuda-python-wheel
path: cuda_python/*.whl
if-no-files-found: error

- name: Set up Python
id: setup-python2
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: ${{ matrix.python-version }}

- name: Set up Python include paths
run: |
if [[ "${{ inputs.host-platform }}" == linux* ]]; then
echo "CPLUS_INCLUDE_PATH=${Python3_ROOT_DIR}/include/python${{ matrix.python-version }}" >> $GITHUB_ENV
elif [[ "${{ inputs.host-platform }}" == win* ]]; then
echo "CL=/I\"${Python3_ROOT_DIR}\include\python${{ matrix.python-version }}\"" >> $GITHUB_ENV
fi
# For caching
echo "PY_EXT_SUFFIX=$(python -c "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))")" >> $GITHUB_ENV

- name: Build cuda.bindings Cython tests
run: |
pip install $(ls ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}/*.whl)[test]
pushd ${{ env.CUDA_BINDINGS_CYTHON_TESTS_DIR }}
bash build_tests.sh
popd

- name: Upload cuda.bindings Cython tests
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: ${{ env.CUDA_BINDINGS_ARTIFACT_NAME }}-tests
path: ${{ env.CUDA_BINDINGS_CYTHON_TESTS_DIR }}/test_*${{ env.PY_EXT_SUFFIX }}
if-no-files-found: error

- name: Build cuda.core Cython tests
run: |
pip install $(ls ${{ env.CUDA_CORE_ARTIFACTS_DIR }}/*.whl)[test]
pushd ${{ env.CUDA_CORE_CYTHON_TESTS_DIR }}
bash build_tests.sh
popd
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have these built by a more standard mechanism and avoid needing to run a separate bash script. Any reason we wouldn't want these built when invoking pip install?

Doesn't need to be solved in this PR regardless.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the same thought, I have some local changes (started on build) I will rebase once this is merged.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because I don't want to include the built tests in the wheel. If there is a way to achieve this with the build frontend/backend, it'd certainly simplify build-wheel.yml here, but I couldn't find any.


- name: Upload cuda.core Cython tests
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}-tests
path: ${{ env.CUDA_CORE_CYTHON_TESTS_DIR }}/test_*${{ env.PY_EXT_SUFFIX }}
if-no-files-found: error
17 changes: 6 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
CUDA_BUILD_VER: ${{ steps.get-vars.outputs.cuda_build_ver }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
- name: Get CUDA build version
Expand All @@ -45,8 +45,7 @@ jobs:
name: Build ${{ matrix.host-platform }}, CUDA ${{ needs.ci-vars.outputs.CUDA_BUILD_VER }}
if: ${{ github.repository_owner == 'nvidia' }}
secrets: inherit
uses:
./.github/workflows/build-wheel.yml
uses: ./.github/workflows/build-wheel.yml
with:
host-platform: ${{ matrix.host-platform }}
cuda-version: ${{ needs.ci-vars.outputs.CUDA_BUILD_VER }}
Expand All @@ -66,8 +65,7 @@ jobs:
- ci-vars
- build
secrets: inherit
uses:
./.github/workflows/test-wheel-linux.yml
uses: ./.github/workflows/test-wheel-linux.yml
with:
build-type: pull-request
host-platform: ${{ matrix.host-platform }}
Expand All @@ -87,8 +85,7 @@ jobs:
- ci-vars
- build
secrets: inherit
uses:
./.github/workflows/test-wheel-windows.yml
uses: ./.github/workflows/test-wheel-windows.yml
with:
build-type: pull-request
host-platform: ${{ matrix.host-platform }}
Expand All @@ -106,8 +103,7 @@ jobs:
- ci-vars
- build
secrets: inherit
uses:
./.github/workflows/build-docs.yml
uses: ./.github/workflows/build-docs.yml
with:
build-ctk-ver: ${{ needs.ci-vars.outputs.CUDA_BUILD_VER }}

Expand All @@ -121,5 +117,4 @@ jobs:
- test-windows
- doc
secrets: inherit
uses:
./.github/workflows/status-check.yml
uses: ./.github/workflows/status-check.yml
6 changes: 3 additions & 3 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,16 @@ jobs:
build-mode: none
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Initialize CodeQL
uses: github/codeql-action/init@v3
uses: github/codeql-action/init@ff0a06e83cb2de871e5a09832bc6a81e7276941f # v3.28.18
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
queries: security-extended

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
uses: github/codeql-action/analyze@ff0a06e83cb2de871e5a09832bc6a81e7276941f # v3.28.18
with:
category: "/language:${{matrix.language}}"
2 changes: 1 addition & 1 deletion .github/workflows/release-upload.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
ARCHIVE_NAME: ${{ github.event.repository.name }}-${{ inputs.git-tag }}
steps:
- name: Checkout Source
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
ref: ${{ inputs.git-tag }}
Expand Down
Loading
Loading