Implement code coverage in GitHub actions #6441

nicoddemus · 2020-01-11T15:22:00Z

This overwrites the codecov.yml file in the root of the repository with
codecov-upstream.yml file (which contains the code-cov token), so PRs
and branches on the repository can upload coverage.

Suggestion from here:

#6421 (comment)

Security concerns: the token might be misused, but only to upload bogus coverage
to codecov.io, so the team believe this is harmless. If we decide to fallback
from this decision , we just need to revoke the token.

Related to #6369

(Following up from #6421, but now on my fork)

This overwrites the `codecov.yml` file in the root of the repository with `codecov-upstream.yml` file (which contains the code-cov token)´, so PRs and branches on the repository can upload coverage. Suggestion from here: pytest-dev#6421 (comment) Security concerns: the token might be misused, but only to upload bogus coverage to `codecov.io`, so the team believe this is harmless. If we decide to fallback from this decision , we just need to revoke the token. Related to pytest-dev#6369

nicoddemus · 2020-01-11T15:23:31Z

.github/workflows/main.yml

+      run: "tox -e ${{ matrix.tox_env }}"
+
+    - name: Prepare coverage token
+      if: success() && !matrix.skip_coverage && ( github.repository == 'pytest-dev/pytest' || github.event_name == 'pull_request' )


@hugovk small difference here: we still upload coverage on PRs. Of course this may happen on PRs on fork -> fork, but this should be rare and not a problem.

OK, should be fine.

So, when github.event_name == 'pull_request', is it something like github.repository == 'nicoddemus/pytest'?

Didn't check, but I think so

nicoddemus · 2020-01-11T15:34:11Z

I've tried to remove the Azure pipelines/add it back to see if I could fix the failures, but the Pipelines page no longer shows pytest-dev/pytest for me to select... weird.

Anyway, I think we can disable azure entirely now because all Windows builds will be done on GitHub actions.

Still keeping Travis for deploy, at least until #6369 is fully resolved.

nicoddemus · 2020-01-13T23:39:27Z

It would be nice to merge this soon, as Azure is broken.

blueyed · 2020-01-14T00:13:17Z

I do not get why we need .github/codecov-upstream.yml.
We could also skip reporting coverage for when it would not be copied, no?

blueyed · 2020-01-14T00:14:32Z

Anyway, I think we can disable azure entirely now because all Windows builds will be done on GitHub actions.

Makes sense already now if it cannot be fixed.

blueyed · 2020-01-14T00:16:33Z

codecov.yml

@@ -1,3 +1,5 @@
+# note: `.github/codecov-upstream.yml` is basically a copy of this file, please propagate
+# changes as needed


If (the concept of) .github/codecov-upstream.yml needs to be kept it could also just append necessary lines to this file, making this comment unnecessary.

append necessary lines to this file

What do you mean? Is there a way to merge the two files?

cat >> codecov.yml <<EOF codecov: # token from: https://codecov.io/gh/pytest-dev/pytest/settings # use same URL to regenerate it if needed token: "1eca3b1f-31a2-4fb8-a8c3-138b441b50a7" EOF

/cc @hugovk in case you want to adopt it also.

Good idea, done. 👍

Used a python script so I can put docs, do assertions, etc.

nicoddemus · 2020-01-14T00:37:21Z

I do not get why we need .github/codecov-upstream.yml.
We could also skip reporting coverage for when it would not be copied, no?

From here: #6421 (comment)

If you put it inline, builds on forks (even if no PR is made) will be sent to the pytest's own Codecov, which clutters up the logs somewhat, and fork users cannot see their own coverage before creating PRs.

Not sure how common it is to use codecov on forks though.

blueyed · 2020-01-14T00:57:04Z

Ok, let's only add the token manually then for now.
But I suggest just appending it instead of having a separate file.

nicoddemus · 2020-01-14T01:51:56Z

But I suggest just appending it instead of having a separate file.

But won't the below be a problem?

If you put it inline, builds on forks (even if no PR is made) will be sent to the pytest's own Codecov, which clutters up the logs somewhat

blueyed · 2020-01-14T02:46:21Z

@nicoddemus
It's not put inline / used always, but instead of having a separate file and copying it you'll do the cat (with the same pre-conditions).

hugovk · 2020-01-14T05:51:10Z

Appending should be fine too (with the same pre-conditions), it avoids needing to duplicate and keep the rest of the file in sync.

The catted snippet could also be in a file of you'd prefer to keep it out of the yml config.

blueyed

Looking forward to having coverage for Windows again.. 👍
Please squash.

blueyed · 2020-01-14T12:01:38Z

The catted snippet could also be in a file of you'd prefer to keep it out of the yml config.

Can be even rewritten in Python.. :)

blueyed · 2020-01-14T12:04:15Z

@nicoddemus
GH actions are not done for PRs to features?

nicoddemus · 2020-01-14T12:14:53Z

GH actions are not done for PRs to features?

Not yet it seems. I will go ahead and merge this one.

nicoddemus · 2020-01-14T12:20:43Z

Sorry, just realized we still needed a small change in main.yaml, done it in #6458 (which only after pushing realized you had already opened #6457, sorry about the wasted time).

blueyed · 2020-01-14T13:19:20Z

Please squash.

Guess I should request changes for this in the future.

hugovk · 2020-01-14T13:28:49Z

🎉 GitHub Actions is now showing up under the builds for new commits on Codecov!

https://codecov.io/gh/pytest-dev/pytest/commit/d291905825d2ff3e5bbbaa245d8fecd24c980db9/build

Compare to an older commit from last week:

https://codecov.io/gh/pytest-dev/pytest/commit/36531599a4192c9c4f4f39ecdf2cdb32cde91628/build

nicoddemus · 2020-01-14T13:46:14Z

Guess I should request changes for this in the future.

Sorry didn't see that; my initial take was to keep the commit, but in retrospect I agree it would have been better to squash it.

webknjaz · 2020-01-14T14:35:41Z

.github/workflows/main.yml

@@ -1,8 +1,7 @@
-# evaluating GitHub actions for CI, disconsider failures when evaluating PRs
+# evaluating GitHub actions for CI, disregard failures when evaluating PRs
 #
 # this is still missing:
 # - deploy


FYI: https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ / https://github.com/marketplace/actions/pypi-publish

Thanks for the links!

Feel free to ping me for help/review. I'm kinda developing my own framework for GH Apps&Actions + I authored that guide & action...

Sure, will do! I don't expect many problems, I've already setup deployment for a number of projects (including pytest-mock), but you never know!

nicoddemus mentioned this pull request Jan 11, 2020

Include codecov on GitHub Actions #6421

Closed

nicoddemus commented Jan 11, 2020

View reviewed changes

blueyed reviewed Jan 14, 2020

View reviewed changes

Append token to codecov.yml instead of duplicating the file

d291905

nicoddemus requested review from hugovk and blueyed January 14, 2020 10:50

blueyed approved these changes Jan 14, 2020

View reviewed changes

nicoddemus merged commit 00adb4e into pytest-dev:master Jan 14, 2020

nicoddemus deleted the gh-actions-cov branch January 14, 2020 12:14

webknjaz reviewed Jan 14, 2020

View reviewed changes

nicoddemus mentioned this pull request Jan 14, 2020

Include deploy and coverage in GitHub actions #6369

Closed

4 tasks

hugovk mentioned this pull request Jan 30, 2020

GitHub Actions #5778

Closed

hugovk mentioned this pull request Feb 10, 2020

Lint and test on GitHub Actions PyCQA/isort#1122

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement code coverage in GitHub actions #6441

Implement code coverage in GitHub actions #6441

nicoddemus commented Jan 11, 2020 •

edited

Loading

nicoddemus Jan 11, 2020

hugovk Jan 11, 2020

nicoddemus Jan 11, 2020

nicoddemus commented Jan 11, 2020 •

edited

Loading

nicoddemus commented Jan 13, 2020

blueyed commented Jan 14, 2020

blueyed commented Jan 14, 2020

blueyed Jan 14, 2020

nicoddemus Jan 14, 2020

blueyed Jan 14, 2020 •

edited

Loading

blueyed Jan 14, 2020

nicoddemus Jan 14, 2020

nicoddemus commented Jan 14, 2020

blueyed commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

blueyed commented Jan 14, 2020

hugovk commented Jan 14, 2020

blueyed left a comment

blueyed commented Jan 14, 2020

blueyed commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

blueyed commented Jan 14, 2020

hugovk commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

webknjaz Jan 14, 2020

nicoddemus Jan 14, 2020

webknjaz Jan 14, 2020

nicoddemus Jan 14, 2020

		@@ -1,3 +1,5 @@
		# note: `.github/codecov-upstream.yml` is basically a copy of this file, please propagate
		# changes as needed

Implement code coverage in GitHub actions #6441

Implement code coverage in GitHub actions #6441

Conversation

nicoddemus commented Jan 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicoddemus commented Jan 11, 2020 • edited Loading

nicoddemus commented Jan 13, 2020

blueyed commented Jan 14, 2020

blueyed commented Jan 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueyed Jan 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicoddemus commented Jan 14, 2020

blueyed commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

blueyed commented Jan 14, 2020

hugovk commented Jan 14, 2020

blueyed left a comment

Choose a reason for hiding this comment

blueyed commented Jan 14, 2020

blueyed commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

blueyed commented Jan 14, 2020

hugovk commented Jan 14, 2020

nicoddemus commented Jan 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicoddemus commented Jan 11, 2020 •

edited

Loading

nicoddemus commented Jan 11, 2020 •

edited

Loading

blueyed Jan 14, 2020 •

edited

Loading