Repository navigation
Regenerate relevance tables in analysis_server #48771
Description
The relevance tables were generated before the prevalence of null safe code. We need to regenerate them, using null safe code, aspects of which may look quite different.
CC @bwilkerson
Activity
bwilkerson commentedon Apr 9, 2022
Konstantin has suggested that we run an experiment to test whether relevance scores actually improve the ranking over using just the prefix matching score. If relevance scores aren't helping then we won't need to have these tables, so we might want to run that experiment first.
If we do need the tables, I'll note that the current tables were built using a small number of small Flutter apps, all of which are now quite old. We should think about the right mix of inputs in order to maximize the quality of the results for whatever kind of code a user might be writing.
The tables are built by
tool/code_completion/relevance_table_generator.dart.When we build new relevance tables we should test them against the current tables to see what the impact on the metrics is. There's support for that built in to
code_metrics.dart. I'll be happy to walk someone through the process if it isn't me.srawlins commentedon Apr 12, 2022
This doesn't have to be super rigorous to get some good benefits, right? Like if I switched the inputs to be just the flutter/gallery app, and regenerate the tables, and run completion_metrics (over a corpus different from flutter/gallery, I think), and the metrics are generally better, then we can call that a day, right? A separate task might focus on increasing the inputs by an order of magnitude, and trying to maximize the gains by selecting different inputs.
bwilkerson commentedon Apr 14, 2022
Yes. You can't evaluate the quality of the relevance tables by running them over the same code used to build them for the same reasons you can't test an ML model by running it over the training data.
I ran an experiment similar to what you're describing a while back (maybe a year ago? my sense of time has been thrown off by the pandemic) and what I found is that the relevance tables didn't really change enough to show any improvement over what we currently have. I don't know whether the same will be true now. Let me know if you want details.
srawlins commentedon Apr 14, 2022
That's very interesting. The one reason I filed this task is the pre-null safety / post-null safety change which has taken place. I think it'd be worth investigating, if we make a push to improve completion quality.
All that being said, I have no intention of investigating completion quality any time soon; my current charge is strictly for performance.
bwilkerson commentedon Apr 14, 2022
I don't think null-safety had as big an impact on completion suggestions as we might expect. We do have a feature that looks at types, and the types of expression has changed, but that's not part of the content of these tables.
That said, I'd forgotten before but one aspect of this, which will continue to be an issue into the future, is that newer language features that didn't exist when the tables were last built aren't represented in the tables at all. That by itself is a good reason for rebuilding them.
DanTup commentedon Sep 23, 2025
Possibly related to this - Dart-Code/Dart-Code#5719 requests that extension methods are ranked lower. While looking at the data, I noticed that all methods (regardless of whether they're from the class, a subclass, or an extension) all seem to rank equally and therefore end up alphabetical:
I don't know if this is good or bad, but if the tables are being computed maybe it's worth including whether they're extensions or not and the "depth" into the tree as factors to see if they affect the scores?
bwilkerson commentedon Sep 23, 2025
This is definitely an area that needs more work. I've added it to the list of tasks to consider for Q4 resourcing.