Analyzer: Flaky results in IDE

[btrautmann]

• Dart SDK Version (dart --version) Dart SDK version: 2.13.3 (stable) (Wed Jun 9 12:44:44 2021 +0200) on "macos_x64"
• Whether you are using Windows, MacOSX, or Linux (if applicable) MacOS 11.6
• Whether you are using Chrome, Safari, Firefox, Edge (if applicable) N/A

This issue is a follow-up to discussion occurring in #57588, starting with this comment.

During my attempts at developing a custom analyzer plugin, I've found the IDE feedback to be very flaky. I figured it was just my own implementation (even though it was mostly based off the same approach as string_literal_finder), but once I actually pulled down string_literal_finder and pointed to it from source, I saw similar results, shown below. You can find the source for my reproduction here.

Screen.Recording.2021-10-28.at.10.34.07.AM.mov

At the time the above was recorded, this was what analyzer diagnostics was telling me. Unfortunately I did not capture instrumentation logs (well I did, but have since restarted the analyzer server which wiped those logs out).

Please let me know if I can provide any more helpful information.

[btrautmann]

I continued investigating this and I'm actually seeing the same when importing the package from pub.dev, but I think this is still worth leaving here as it may point to something wrong with how the analysis server invokes the plugin.

Furthermore, my question at the bottom of this comment still stands, which is:

One additional thing that I noticed, is that when the analysis server creates a directory for the analyzer_plugin, it copies over it's pubspec.yaml but relative paths not made absolute, i.e a bootstrap package that relies on the host package like this:

dependencies:
  better_lint:
    path: ../../

Will not be able to resolve things packages that exist in the host package because that package is not copied over to the temp directory as well. Am I off base/misunderstanding something here? In the case of string_literal_finder its pubspec looks like this:

dependencies:
  string_literal_finder: ^1.0.0-dev.1

If I'm correct, does this mean that you cannot use relative paths when referencing the host package from the bootstrap backage?

[btrautmann]

Hi there, just checking in. Been exploring what it might look like to build out our analyzer plugin, and I haven't seen any traction here or in #57588. I'm guessing this isn't much of a priority for the team?

I have found dart-code-metrics which successfully (at least in my playing around with it) adds an IDE integration, but doesn't support custom lint rules and doesn't plan to, so before I go and build out our own, I was hoping to hear if there were any plans to make this easier for developers or if rolling your own is the only solution for the time being.

[DanTup]

@btrautmann do you have steps to reproduce the issue with diagnostics disappearing when you change files? There seemed to be some version mismatches when I tried in flutter/devtools#3585, if you have something that works on the latest code I can try to debug what's happening.

I don't know the answer to the question about using relative paths or if there are any changes in this area - @bwilkerson may have more knowledge about those.

[bwilkerson]

If I'm correct, does this mean that you cannot use relative paths when referencing the host package from the bootstrap backage?

It's been a while since I've looked at the plugin implementation, but I wouldn't be at all surprised to find that it doesn't support having relative paths in the bootstrap package's pubspec.

I'm guessing this isn't much of a priority for the team?

I think we're very interested in making it possible for the community to provide enhancements to the development experience. We value community involvement and even if we didn't, we don't have the resources to do everything. The analyzer plugin support was an experiment to see how well a plugin approach might work to satisfy the need to make it easier for the community to provide enhancements. The current implementation seems promising to me, but there are some issues that we need to resolve before it can reasonably be considered to be production quality.

It's true that this hasn't yet been a higher priority that some other projects, but that doesn't mean that it has a low priority, only that we have limited resources and a lot of work to do. Unfortunately, several important tasks have not been fully addresses, and this is one of them.

I was hoping to hear if there were any plans to make this easier for developers or if rolling your own is the only solution for the time being.

Rolling your own plugin is the only approach that's currently been implemented. We've certainly discussed other options, but there are lots of advantages to the plugin approach and lots of UX and technical challenges to some of the other options that have been suggested. We haven't entirely ruled out other options, but plugins still seem, to me, to be the best option available to us.

[btrautmann]

Wanted to leave an update here:

I wouldn't be at all surprised to find that it doesn't support having relative paths in the bootstrap package's pubspec

I believe this ended up being the case. We ended up using a git reference for the bootstrap package and the flakiness subsided. I also suspect that the version field that we were overriding in ServerPlugin was causing us issues; during our original pass at creating an analyzer plugin, we weren't aware that the string needed to align with an analysis server version (I read that somewhere, but I'm not sure if it's 100% correct). As it stands, we're using '1.0.0-alpha.0'. I do believe that if the tutorial docs here were improved, it would go a long way in shedding light on the specifics around writing an analyzer plugin!

We haven't entirely ruled out other options, but plugins still seem, to me, to be the best option available to us.

Thanks for highlighting the team's current thoughts. Our team is interested in possibly finding a solution such that makes writing analyzer plugins trivial. From our perspective this is really the only path forward, since there's no way to basically ship a reusable analyzer plugin capable of querying a consumer for its custom lint rules (otherwise y'all would've just done this with linter, right?).

If y'all end up working on this, it would be great if you could keep the community posted here or on #57588 so we avoid duplicate efforts 😄

I am going to close this issue, though, because I believe it was the relative path of the bootstrap package that was causing flakiness.