Run test script with/without coverage

[duckontheweb]

Related Issue(s): #

• Improve coverage checks #466 (specifically this comment)

Description:

Adds ability to run scripts/test with or without coverage (default outside of CI is still to run with coverage).

Also moves generation of coverage.xml out of scripts/test. The XML file is mostly used when uploading coverage to Codecov.io. Generating this file would cause all test jobs in the matrix to fail if the coverage dropped, which is misleading. Instead, this PR updates the coverage CI job to first upload the coverage report to Codecov.io so we can use it for debugging, then generates the report again using the configured fail-under value, which will cause coverage job to fail if coverage drops. This should make the source of the workflow failure a bit more obvious to developers.

PR Checklist:

• Code is formatted (run pre-commit run --all-files)
• Tests pass (run scripts/test)
• Documentation has been updated to reflect changes, if applicable
• This PR maintains or improves overall codebase code coverage.
• Changes are added to the CHANGELOG. See the docs for information about adding to the changelog.

[duckontheweb]

cc: @l0b0

[codecov-commenter]

Codecov Report

Merging #486 (0eaef55) into main (f9573f8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #486   +/-   ##
=======================================
  Coverage   91.66%   91.66%           
=======================================
  Files          40       40           
  Lines        5245     5245           
=======================================
  Hits         4808     4808           
  Misses        437      437           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9573f8...0eaef55. Read the comment docs.