Faster cache deserialization

[JukkaL]

Deserialization of cache files is a major bottleneck in incremental mode, so any performance improvements there would be valuable.

At least @pkch has previously worked on this, and it's unclear if any significant wins are achievable without a lot of work. However, there seem to be some redundant information in the cache files that we haven't looked at eliminating yet.

Here are some random ideas:

• Use shared type objects at least for common types such as int and str.
• Don't include default attribute values for the most common AST nodes. For example, we could have "defaults": true to mark that we can skip processing non-default values altogether.
• Use shared dummy objects for things like empty function bodies.
• Micro-optimize deserialization of the most common node types.
• Avoid repeating information about arguments in the serialized form of FuncDefs.

Idea for how to move forward:

• Run a profiling run when type checking mypy and looks for promising hot spots (spoiler: there probably aren't any major hot spots visible in the profile). Post results here.
• Determine which node types are the most common ones during deserialization. Also calculate cumulative figures and percentages (of all nodes). Post results here.
• Prototype various potential optimizations and benchmark them individually. Note that getting reliable benchmark results is hard because of turbo modes, background processes, etc. I may look at setting up a dedicated system for running benchmarks.
• If speedups from individual optimizations are negligible (under 0.5%) or negative, give up on them and post the negative results here. Otherwise we risk others doing the same experiment in the future.
• If the total speedup from all optimizations would be under, say, 10%, it probably doesn't make sense to land any of them, unless some optimizations are utterly trivial.

In any case, this issue isn't important enough to spend a lot of time on it. We have other plans to speed up mypy, but as they will take more effort, so some quick wins here would be nice.

[JelleZijlstra]

Random idea: Could we use a different format than JSON, perhaps marshal?

[gvanrossum]

JSON serialization itself is not the bottleneck. The json stdlib module uses highly optimized C code when (de)serializing using default options, which we are careful to use unless you pass --debug-cache. The cost is all in constructing dicts that the json module can serialize, and (after deserialization) constructing objects from those same dicts. This is what the serialize() and deserialize() methods do.

[ilevkivskyi]

Another random idea (probably I have mention it before): use pickle. It supports custom stateful objects via __getstate__ and __setstate__. The downside is that cache will be human-unreadable (unlike JSON). But if this will give a significant speed-up, maybe we can still consider this?

(Also I think stdlib __getstate__/__setstate__ will be more familiar to new contributors than custom serialize/deserialize methods.)

[gvanrossum]

Hm, but the __getstate__/__setstate__ methods have to do about the same amount of work as serialize/deserialize so I don't expect a big benefit from this. Also people have generally soured on pickle due to various vulnerabilities and incompatibilities.

[ilevkivskyi]

Yes, but on the other hand, some of the methods could be simpler, since intermediate dicts are not created (everything is put in objects __dict__ directly), plus some work is done already in __init__, plus some method calls are C-level. I agree it is not obvious, and benchmarking is needed to prove any win with this approach.

[ilevkivskyi]

Just another random idea. I have heard from a friend who works on a project that has a feature very similar to our --incremental mode: they recently switched from JSON to using SQL and have seen a reasonable performance boost. I am not aware of any details, and my experience with SQL is very limited, just wanted to share this idea.

I understand that the bottleneck is probably not parsing JSON but creating actual instances, but I have heard that SQL supports self-referential data structures, so that maybe we can save some time also in fixup.py.

[JukkaL]

Decreasing priority for now. We'll be focusing on fine-grained incremental checking first.

[emmatyping]

Is this still a concern? I believe we now have a SQL store for the cache as an option. If it is still a concern, I do like the idea of using __getstate__/__setstate__.

[JukkaL]

We now have an option of using sqlite for storing cache information, which is helpful. Deserialization performance is still an issue for large builds, though.

Using __getstate__/__setstate__ can be tricky because we must allow cache files to be loaded incrementally, and cache files can have references to things defined in other cache files. Import cycles can have cyclic references.

Another idea not discussed above is to deserialize cache files lazily. For example, consider a module m that (directly or indirectly) depends on 5000 other modules. Currently we will deserialize all the 5000 modules if m is dirty. However, usually most of the dependencies are not actually required to type check m. We could plausibly delay the deserialization of a module until we actually need some definition from the module while type checking. The details are still hazy, but this seems like a big potential win, though it's non-trivial to implement.

[msullivan]

Our use of SQL is totally half-assed, though. We have a table with filename, data, mtime columns and we store our json blobs in it.