Improve performance when opening large projects

[citizenmatt]

This PR fixes a number of performance issues when opening projects that have a very large number of assets and meta files:

• Reuse cached file metadata when creating IPsiSourceFile instances for asset and meta files. Fixes RIDER-53358
  We efficiently get this information when calling GetDirectoryEntries to discover asset and meta files in Assets, ProjectSettings and package folders, but creating the PSI source file will go back to the disk for every single file to get timestamp, length and file attributes. This has a massive impact on very large projects (tens of thousands of files), and can cause further problems on certain OS/drive/file system combos (e.g. exFAT on USB on Windows can take 12 minutes for 24,000 files just to get the information we've already got in memory)
• Reduce excessive memory allocations in UnityExternalFilesPsiModule.SourceFiles for very large projects.
  The module's trie isn't enumerable, and would allocate and populate a new collection on each access, and it is frequently accessed. The module now maintains a separate set of files as well as the trie.
• Reduce excessive allocations when trying to get the PSI language of UnityExternalPsiSourceFile.
  The logic to look up the PSI language would allocate, first to get file extension to look up the ProjectFileType, then allocate again in UnityYamlProjectFileLanguageService.GetPsiLanguageType when fetching the PsiLanguageType. This has been simplified to pass the ProjectFileType in to the constructor, so no more allocations and lookups are required.
  This required introducing MetaProjectFileType, which derives from YamlProjectFileType and registers the .meta file (which was previously handled by UnityYamlProjectFileType, even though meta files didn't use the UnityYaml PSI language). The appropriate project file type is calculated in the external module processor and passed in - the simple YAML project file type is used when creating files for the ProjectSettings folder.
• Replaced a tiny allocation in a hot path in the YAML parser with a statically allocated empty NodeTypeSet. Saves several hundred megabytes on a very large project!

[citizenmatt]

A further note on ProjectFileLanguageService, because it confused me for a while. The PsiLanguageType property is the "default" language type of the langauge service, and is only used by the language service itself. Other consumers will try to get the language type of specific files, and the base implementation will end up using the default value from PsiLanguageType.

The UnityYamlProjectFileLanguageService used to have YamlLanguage as PsiLanguageType. It would also override GetPsiLanguageType(IPsiSourceFile) and check to see if the file is .meta or ProjectSettings/*.asset. If so, it defers to base.GetPsiLanguageType(IPsiSourceFile), which ultimately returns PsiLanguageType - YamlLanguage. If it's not .meta or ProjectSettings/*.asset, then it's treated as UnityYamlLanguage.

The UnityYaml language does not have a PSI LanguageService, so won't create a lexer or parser. We manually create lexer/parser when indexing the file. This gives us control of when these potentially massive files are parsed.

UnityYamlProjectFileLanguageService has been simplified so that it only handles the UnityYaml language, and file extensions registered with UnityYamlProjectFileType - .unity, .asset, .prefab, etc. This means that if a UnityYaml file is asked for its PSI language, it will always return UnityYaml.

Asset files under ProjectSettings are now created with a YamlProjectFileType, so they will always return YamlLanguage as the PSI language.

Meta files now have MetaProjectFileType, which derives from YamlProjectFileType, so we have somewhere to register .meta, but everything else behaves like YamlProjectFileType.

[wqaetly]

However, this is very slow for our project, which is 4T in size. So can we provide an option to directly block meta file processing, because we only need to use Rider's Debug function?

[van800]

@wqaetly would you please create a separate report in Rider with Help-> Report a Bug?

[wqaetly]

@wqaetly would you please create a separate report in Rider with Help-> Report a Bug?您可以在 Rider 中使用“帮助”->“报告错误”创建单独的报告吗？

No, it's not a bug of rider, because after I disabled Unity Plugin, it only took one minute to refresh.

[van800]

Since this plugin is bundled in Rider, we started to use youtrack for all major planning and Q&A.
If you also agree to attach the logs, it would help a lot understanding all the circumstances.

[van800]

Have you already tried this setting?

[wqaetly]

Have you already tried this setting?您是否已经尝试过此设置？

no, i turn off all possible toggle

[wqaetly]

Since this plugin is bundled in Rider, we started to use youtrack for all major planning and Q&A.由于此插件捆绑在 Rider 中，我们开始使用 youtrack 进行所有主要规划和问答。 If you also agree to attach the logs, it would help a lot understanding all the circumstances.如果您也同意附加日志，这将有助于了解所有情况。

sorry i cannot , because it's company project

[wqaetly]

Since this plugin is bundled in Rider, we started to use youtrack for all major planning and Q&A.由于此插件捆绑在 Rider 中，我们开始使用 youtrack 进行所有主要规划和问答。 If you also agree to attach the logs, it would help a lot understanding all the circumstances.如果您也同意附加日志，这将有助于了解所有情况。由于这个插件捆绑在 Rider 中，我们开始使用 youtrack 进行所有主要规划和问答。由于此插件捆绑在 Rider 中，我们开始使用 youtrack 进行所有主要规划和问答。如果您也同意附加日志，这将有助于了解所有情况。如果您也同意附加日志，这将有助于了解所有情况。

sorry i cannot , because it's company project抱歉我不能，因为这是公司项目

i am using Rider 2025.1, the problem remain existing

[van800]

Please start a new issue (in Rider) or a support ticket. Otherwise, unfortunately in this thread to an issue, which was closed long ago, you would not get any help.