Skip to content

Cut out evaluation in the "pythoneval" tests #1671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gnprice opened this issue Jun 7, 2016 · 1 comment
Closed

Cut out evaluation in the "pythoneval" tests #1671

gnprice opened this issue Jun 7, 2016 · 1 comment

Comments

@gnprice
Copy link
Collaborator

gnprice commented Jun 7, 2016

The "pythoneval" tests are by far the biggest contributor to the time it takes to run our test suite. For each of 171 sample programs, we type-check them and then run them through the Python interpreter.

From discussion in #1668, running the programs is no longer important for our testing. So cut it out. This will mean

  • cutting out that step in testpythoneval.py,
  • cutting out the program output (as opposed to the type-checker output) from all the test cases, and
  • cutting out entirely any cases that no longer seem to have any meaningful content when they're only for the type-checker.

Some of the test cases may exercise features of typing in interesting ways -- any of those should be moved to the typing repo's own test suite if they aren't covered there already.

After this is done, we can probably get a big further speedup by folding the remaining test cases into just more cases for testcheck.py, so that we're not forking a mypy subprocess for each one. Will make a separate issue for that.

emmatyping added a commit to emmatyping/mypy that referenced this issue Sep 27, 2017
This replaces the old subprocess based method. In addition, the test cases
are no longer run, in order to reduce test time. This has led to a 10-20%
speedup in pytest. This might help with python#3895, as it removes the
subprocesses.

Fixes python#1671
@emmatyping
Copy link
Member

I just tested completely removing this and it doesn't seem to actually be that big of a runtime hit. WIth Guido's points about checking real code: #4025 (comment), I think we can just close this, the majority of the runtime seems to be in type checking the files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants