Skip to content
This repository was archived by the owner on Apr 14, 2022. It is now read-only.

Lazy types in caching #1757

Merged
merged 116 commits into from
Dec 9, 2019
Merged

Lazy types in caching #1757

merged 116 commits into from
Dec 9, 2019

Conversation

MikhailArkhipov
Copy link

@MikhailArkhipov MikhailArkhipov commented Nov 6, 2019

Fixes #1728
Fixes #1758
Fixes #1514
Fixes #1515
Fixes #1516
Fixes #1517
Fixes #1518
Fixes #1569
Fixes #1680

  • Instead of restoring analysis, restore complete module and claim there are no dependencies.
  • Introduces lazy types. Lazy types expose documentation and names, but members, function return types and parameters are populated only when requested.
  • Allow creating of caches for stub-only modules like posix or mmap.
  • Delay cache writing to after analysis completes so it does not steal CPU time from analysis.
  • Limit how much IO threads run at the same time in the cache writer.

TODO (later):

  • Functions should store signatures so documentation in completion list won't need to fully restore type.

Related

MikhailArkhipov added 30 commits September 30, 2019 12:03
MikhailArkhipov added 5 commits December 5, 2019 13:52
var (modulesCount, totalMilliseconds) = ActivityTracker.EndTracking();
totalMilliseconds = Math.Round(totalMilliseconds, 2);
_analyzer.RaiseAnalysisComplete(modulesCount, totalMilliseconds);
_log?.Log(TraceEventType.Verbose, $"Analysis complete: {modulesCount} modules in {totalMilliseconds} ms.");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I'm seeing here is that due to the lazy loading, we're (in a way) under-reporting the number of modules we're working with. Hopefully that's not such a big deal if the work also disappears, but it may end up appearing as though we're not "improving" per-module because for the N modules we still have to load, we have to do the same work.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not entirely happy with RaiseAnalysisComplete being bool, but I can make it return if it actually raised the event. Somewhat ugly, but implementation uses data internal to the analyzer which i'd rather not make public.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean by returning a bool, I'm more talking about the fact that we'll log "123 modules analyzed", when maybe there are a bunch more that weren't "loaded" because they'll be fetched from disk.

You don't have to change anything for this comment, I was more thinking about the impact of lazyness on our reporting.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean returning bool from RaiseAnalysisComplete and then only reporting when it is really done.

if (id == null) {
var parent = moduleResolution.CurrentPathResolver.GetModuleParentFromModuleName(moduleName);
if (parent == null) {
id = moduleName;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this string just be something like "foo.bar" without any versioning? If the version needs to come from a parent, I'd expect to have seen recursion or something. Maybe I'm just reading this wrong.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an edge case just to handle GetModuleParentFromModuleName that can return null since hashing needs parent IImportChildrenSource.

@jakebailey
Copy link
Member

Caching isn't perfect for all libraries, but the default is None and we want the rest of the fixes too. Approving.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.