Skip to content

Improve the CI tests speed via Codecov Automated Test Selection #2791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seisman opened this issue Nov 1, 2023 · 2 comments
Closed

Improve the CI tests speed via Codecov Automated Test Selection #2791

seisman opened this issue Nov 1, 2023 · 2 comments
Assignees
Labels
maintenance Boring but important stuff for the core devs

Comments

@seisman
Copy link
Member

seisman commented Nov 1, 2023

I haven't read the full documentation carefully, but it seems the new codecov tool, Automated Test Selection can help improve the tests speed significantly:

Automated Test Selection is a Codecov feature that ensures your CI pipeline only runs the tests needed to completely exercise the diff of a commit. Automated Test Selection is intended to greatly reduce the time needed to run test suites without compromising on test coverage or having to invest in creating the complex CI pipelines and tools (e.g., Bazel) that are typically required to facilitate test selection.

Testing tools typically have a stage of collecting tests followed by a stage of executing tests. The idea behind Automated Test Selection is to live exactly between those 2 stages as a filter to decide which tests necessarily need to be executed, considering the diff of the commit being tested.

I think it may be worth a try.

@seisman seisman added question Further information is requested maintenance Boring but important stuff for the core devs labels Nov 1, 2023
@weiji14
Copy link
Member

weiji14 commented Nov 1, 2023

Linking to #584. I mentioned something similar with pytest-fastest before, but maybe this is easier to set-up.

Ok to try this Automated Test Selection for Pull Requests, but run full tests still on the main branch (and also for release PRs)? Looking at https://docs.codecov.com/docs/automated-test-selection-quick-start, it sounds like the GitHub Action is only tested on Linux and macOS, so need to check if Windows (our slowest tests) works. We could file an issue at https://github.com/codecov/codecov-ats requesting for Windows support if it doesn't.

@seisman seisman added this to the 0.11.0 milestone Nov 1, 2023
@seisman seisman removed the question Further information is requested label Nov 1, 2023
@seisman seisman changed the title Improve the CI tests speed via Codecov Automated Test Selection? Improve the CI tests speed via Codecov Automated Test Selection Nov 1, 2023
@seisman seisman self-assigned this Nov 1, 2023
@seisman seisman modified the milestones: 0.11.0, 0.12.0 Dec 11, 2023
@seisman seisman removed this from the 0.12.0 milestone Feb 26, 2024
@seisman
Copy link
Member Author

seisman commented Jun 19, 2024

As mentioned in #584 (comment), currently running the tests takes < 1 minute on Linux/macOS and 3 minutes on Windows (see https://github.com/GenericMappingTools/pygmt/actions/runs/9574409146/job/26397597813 for the latest run), which is already fast enough. I don't think we need the Codecov ATS anymore. I'm closing the issue and the related PR #2792.

We may revisit it if the number of tests increases significantly in the future.

@seisman seisman closed this as completed Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Boring but important stuff for the core devs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants