Slow incremental run when single module has errors

[JukkaL]

mypy -i is slower than expected in the following scenario:

• Check out fresh mypy repository
• Create tiny module mypy.x that generates an error: echo '1 + ""' > mypy/x.py
• Run mypy -i mypy
  • This takes a while, generates an error for mypy/x.py as expected
• Run mypy -i mypy
  • This is as slow as the previous run, but caching should make this much faster

[JukkaL]

@gvanrossum in #4130 (comment):

The performance problem is apparently not new -- I can reproduce this with standard -i mode as well. No cache entries for the mypy package are written somehow -- perhaps because mypy.x precedes all other submodules of mypy. This could be due to an edge case in the topsort.

[gvanrossum]

Adding topic-daemon label because this impedes one of the important use cases for the daemon (quickly rechecking code with errors).

[JukkaL]

This is blocking progress in fine-grained incremental mode. ASTs can't be reused when there is an error. In fine-grained incremental mode we want to cache every module that has no blocking errors. I'll try to fix or work around this.

[JukkaL]

I've worked around this in my local branch (but have not fixed the underlying issue).

[gvanrossum]

I've got a clue about why this is happening: in build.py, in State.write_cache(), on line 1943, there's a check for self.manager.errors.is_errors(). But this returns True when there was an error in any previous file! And then two lines down we actually call delete_cache(). (This was added recently in PR #4045.)

What we actually need is a little different -- we want to check if the current SCC or any of its dependents has errors.