Integrate mypy_primer into our CI

[intgr]

mypy_primer is a project that runs mypy on lots of public open source repositories.

In CI, it can do a mypy run before and after some proposed changes in a pull request, and report the diff between those results. It's being used by typeshed and mypy itself among others. Example:

If we add this to django-stubs and djangorestframework-stubs, it would give us better visibility into possible breakage we could be causing and help troubleshoot issues earlier.

I have no immediate plans to implement this and no experience with mypy_primer. I opened this issue to facilitate some discussion and feedback.

Most projects from mypy_primer's built-in project definitions wouldn't be very useuful for us, because they don't use Django. So we would probably want to create our own list of open source Django and django-rest-framework projects for mypy_primer. Suggestions for which ones to add would be helpful.

[sobolevn]

Related hauntsaninja/mypy_primer#8

[sobolevn]

CC @hauntsaninja

[hauntsaninja]

From the mypy_primer corpus, some potentially useful projects are: zulip, dd-trace-py, websockets, pyinstrument (from a very quick skim, probably zulip is the main thing that's useful)

[intgr]

One of the motivations for me is that maintaining the list of ignores for our typecheck_tests.py (https://github.com/typeddjango/django-stubs/blob/master/scripts/enabled_test_modules.py#L60) by hand is quite a chore (there are over 450 lines of ignores!). If we had mypy_primer, mayhaps we can consider deprecating typecheck_tests and can get rid of the maintenance burden. Although there are also a few pros of typecheck.

[RyanWalker277]

Hey there! just wanted to chime in with some thoughts. While it's great to have better visibility into potential breakage and troubleshoot issues earlier, maintaining project-specific configurations might still be a challenge and require ongoing effort. However, if it means we can get rid of the maintenance burden of manually maintaining the list of ignores for our typecheck_tests, that would definitely be a win! Just some food for thought.

We can also consider Wagtail and Django-cms for the custom project list. Would like to hear everyone's thoughts on this.

[intgr]

In case someone wants more background on typecheck_tests.py, here's an explanation I wrote for djangorestframework-stubs: typeddjango/djangorestframework-stubs#345 (comment)

[intgr]

Cross-posting some relevant discussion from a PR.

Comment by @sobolevn #1552 (review)

A good question is: do we still need this typecheck test?
Maybe we can drop it and use stubtest instead?

Comment by @intgr #1552 (comment)

Actually I'm not sure it does anything at all. I suspect not! 🤷

In djangorestframework-stubs I have needed to tweak the ignores lists regularly, but I don't remember the last time I saw it fail in django-stubs. Even when adding typecheck for Django 4.2, there weren't any changes.

Adopting stubtest definitely sounds like a good idea. Is it something you're interested in implementing?

One concern is, who will be responsible for fixing the issues from stubtest? Our maintainer bandwidth is limited. But if we could somehow get more contributors involved in fixing issues, then that would be awesome.

[flaeppe]

I contributed a PR a little while back that should enable support for running mypy plugins with mypy_primer: hauntsaninja/mypy_primer#121

It hasn't been merged yet, but if we want to we could probably start by running off that PR if we want to try to get started

[intgr]

For context: typecheck_tests has been gone for ages now (#1556) and life has been much easier. Collapsing some outdated comments about that.

[sobolevn]

I want to resurrect this issue due to latest release troubles :)