Support extensions from globally activated pub packages

[kenzieschmoll]

An example of an extension that fits this use case would be an arbitrary development tool package (like an AI-assisted development experience extension or a tool that interacts with the Dart analysis server). These extensions may be runtime or static extensions.

It would be a bit awkward to require a user to add a dev_dependency to every Dart project where they want to use this DevTools extension, since tools like these are not specific to a Dart / Flutter project, but rather to the general Dart / Flutter development experience. The workflow for an end user would be:

1. dart pub global activate some_pkg
2. If some_pkg provides a DevTools extension, we should be able to load this in DevTools (and also embedded in the IDE).

@jonasfj @sigmundch is there a way to tell from the pub cache what the globally activated pub packages are?

[sigmundch]

@sigurdm 😄

[sigurdm]

We have dart pub global list, however I'm not sure it gives you enough to go on.

You probably want a way to find the .dart_tool/package_config.json for each of these to detect extensions - right?

@jonasfj should we extend the list command with more info? Any other ideas?

[jonasfj]

I'm not sure globally activated packages should be used for this.

Global package activation is a mechanism for installing a package onto your system.
I'm not sure we should use it to configure extensions for devtools.

That said, a somewhat ludicrous idea would be for devtools to discover system-level extensions by searching PATH.

Imagine that devtools discovers system-level extensions by:

• Listing all binaries from any folder in PATH.
• Filter for executables.
• Take all executables prefixed _dart_devtools_extensions_
• These executables are now considered extensions, if each extension is supposed to provide a Flutter Web application or something like that, you simply run _dart_devtools_extensions_myglobaldevtoolsextension --print-extension-location and have it print where the resources are located.

If you are authoring a global devtools extension, you don't have to ship it as a pub package to be globally activate. You can ship as a globally activated package, because you can use the executables section in pubspec.yaml to specify executables to be installed on PATH. Provided that the user added $PUB_CACHE/bin to $PATH.

I suppose it would also be possible for devtools to try and guess the location of the global PUB_CACHE, and then also search PUB_CACHE/bin for executables prefixed _dart_devtools_extensions_.

With this approach system-level devtools extensions could also be installed using system package manager like apt-get, or just extracted from a zip to /opt and manually added to PATH. Or it could be installed with a different package manager like homebrew, npm or pypi. Because devtools is just searching for executables on PATH prefixed by _dart_devtools_extensions_ (or whatever string you want).

Maybe, it's a bit crazy, but:

• dart global activate is for installing tools.
• I think searching PATH is the best way to discover tools.
• Listing all executables on PATH is probably fast.
• Executables like _dart_devtools_extension_my_extension_name is not going to pollute bash-autocompletion (a prefix like dart_... would probably be annoying).
• A globally activate pub package can find its own location in PUB_CACHE using something like Isolate.resolvePackageUriSync('package:my_own_package_name/my_own_package_name.dart') (and then do a relative path from there)

It might just work :D

[kenzieschmoll]

You probably want a way to find the .dart_tool/package_config.json for each of these to detect extensions - right?

@sigurdm package:extension_discovery's findExtensions method expects a package_config.json location so that it can:

1. Look for any pub package that contains an extension/devtools/ directory
2. Return a list of Extension objects that describe the extension and provide a location for the extension's assets.

For a globally activated package, we would really just need to look at its contents for the extension/devtools directory. Maybe package:extension_discovery could be extended to support this use case as well, if it is non-trivial to find a package_config.json file for globally activated packages.

Global package activation is a mechanism for installing a package onto your system. I'm not sure we should use it to configure extensions for devtools.

@jonasfj Since DevTools extensions are shipped as part of the pub ecosystem, I actually think this is a reason we need to support this use case. As a user, I would expect that if I install a package onto my system, that I can use anything that package provides, including executables, extensions, etc.

if each extension is supposed to provide a Flutter Web application or something like that, you simply run _dart_devtools_extensions_myglobaldevtoolsextension --print-extension-location and have it print where the resources are located.

While this could work, my concern is that this is one more thing that extension authors have to manually set up, and they have to know when they need this and when they do not. I don't expect extension authors to have as much familiarity with what it means to be a "static" extension vs. a "runtime" extension and "global" vs "non-global". We want to make it as simple as possible for authors to build a tool, and know that no matter how users install their package (as a global or as a project dependency), that their users will be able to discover the tool they have built and use it.

With this approach system-level devtools extensions could also be installed using system package manager like apt-get, or just extracted from a zip to /opt and manually added to PATH. Or it could be installed with a different package manager like homebrew, npm or pypi. Because devtools is just searching for executables on PATH prefixed by dart_devtools_extensions (or whatever string you want).

I don't think we want to diverge the extension distribution (and enablement) mechanism, since this will be less discoverable for users and will make the development process more difficult for extension authors. Having one way to ship and install an extension (via pub) is ideal. Shipping / installing with pub is straight forward and keeps DevTools extensions in alignment with the Dart ecosystem standards.

[kenzieschmoll]

Pinging this issue again since another use case has come up where we'd like to be able to load a DevTools extension for a globally activated package.

Rationale:

1. Requiring users to add a dependency to a Dart or Flutter project in order to use an extension is not always ideal. The project may be a pure Dart project, but the package containing the extension may have a Flutter dependency (in the case of standalone extensions, the extension requires a Flutter dep).
2. Adding the dependency for a devtools extension or the package containing a devtools extension may cause unnecessary version solve issues for the project that are unrelated to using the tool.

For these reasons and the ones I outlined above, extension detection for globally activated packages is something we'd like to support.

Could package:extension_discovery be extended to support this use case? @jonasfj @sigurdm Or the other option that Sigurd suggested using dart pub global list would work fine. Could we add dart pub global list --verbose or similar that includes the location of the package_config.json for globally activated packages, and then we could use package:extension_discovery as it is today to find the extensions for global packages.

[jonasfj]

In an ideal world:

(1) is something that should be solved by:

• Using a dev_dependency, or,
• Support publishing Flutter plugin + Dart standalone hybrid packages dart-lang/pub#3563

(2) should be addressed by allowing multiple versions of the same package in a resolution (probably by marking some dependencies as private_dependencies).

The motivation for global extensions should be that there may be extensions which:

• (i) Make sense for all projects on a particular developer machine, say an extension that makes working over a remote session (SSH) easier...
• (ii) Make sense for some users working on a particular project (though a pubspec_override.yaml or other configuration option to disable/enable an extension for a particular checkout might be better).

---

In practice, I think (2) private_dependencies (or something similar) allowing multiple versions is highly requested, but it's also a large project and it may be quite a while before we have consensus for a thing like that.

I suspect that (1) is something we do need to solve, we haven't found a good way yet. But I don't think that an optional Flutter SDK constraint is a controversial feature. And progress on native hooks will probably only make this more urgent.

---

This is my thoughts of the top of my head. Not sure what we should do. Or if there are other reasons to do global extensions. Or maybe, tracking devtools extensions in pubspec.lock is just overkill and we should allow some things to be chaotic.

[sigurdm]

Given that devtools extensions are "just" precompiled flutter(-web) binaries. There should (from the point of view of distributing the extension) not be the need for any other dependencies?
I am not sure pub.dev is the best place for distributing these binaries, but it seems to me one strategy is to publish packages that contain just the precompiled extension and none of the source code or dependencies used for compiling the extension.

With no extensions there is no potential for conflicts?

[kenzieschmoll]

In an ideal world:

(1) is something that should be solved by:

• Using a dev_dependency, or,

Using a dev_dependency is already how we recommend people do this today, but the friction for adding a dependency (dev or not) to your project in order to use the extension is the same. This does not avoid needing the dependency to solve with the remainder of your project nor the issue of adding a Flutter dep to a pure Dart project.

• Support publishing Flutter plugin + Dart standalone hybrid packages dart-lang/pub#3563

I'm not following how this would solve the issue for extensions. Can you provide an example? Here is the potential set up for an extension user:

my_dart_app/pubspec.yaml

name: my_dart_app
version: 1.0.0
environment:
  sdk: ^3.5.0

dependencies:
  # dependencies for the my_dart_app project

dev_dependencies:
  foo: ^1.0.0 # add dependency for standalone devtools extension

foo/pubspec.yaml

name: foo
version: 1.0.0
environment:
  sdk: ^3.8.0
  flutter: ^3.32.0

dependencies:
  flutter:
    sdk: flutter
  # other dependencies for the foo devtools extension

(2) should be addressed by allowing multiple versions of the same package in a resolution (probably by marking some dependencies as private_dependencies).

How would this solve the problem? I assume even private_dependencies still have to solve with the remaining dependencies in the pubspec, which would leave us with the same friction points I described above.

The motivation for global extensions should be that there may be extensions which:

• (i) Make sense for all projects on a particular developer machine, say an extension that makes working over a remote session (SSH) easier...

Agreed. For example, we want to provide a DevTools extension for inspecting the state of the Dart Tooling Daemon. This tool is not specific to a single Dart / Flutter project, but rather is useful for any Dart / Flutter project. It is cumbersome set up to have to manually add this dependency to a project (which may mess up the pub solve and cause toilsome dependency issues) in order to use this tool.

• (ii) Make sense for some users working on a particular project (though a pubspec_override.yaml or other configuration option to disable/enable an extension for a particular checkout might be better).

For this case, I'd recommend using the devtools_options.yaml file to enable / disable the extension for particular users of a project. If this file is .gitignored, then extension configuration can be user-specific even with the dependency on the DevTools-extension-containing package checked in.

I don't think that an optional Flutter SDK constraint is a controversial feature.

I do believe this would be controversial for any package developed in the Dart SDK.

In general though, I don't understand a reason not to support extensions for globally activated packages? This seems like the most simple way to support the global tool use case and would require minimal changes to the extension discovery mechanism. We could even hack this together today with the information we have available from ~/.pub-cache/global_packages/foo/pubspec.lock. There is enough information there to piece together the location of the package_config.json file for a package and that is all we need to send to the extension_discovery package to get back a list of extensions. However, doing this would not be as robust as providing first party support from extension_discovery or from pub itself, so we don't want to hack something together when we could provide a robust alternative.

[kenzieschmoll]

Given that devtools extensions are "just" precompiled flutter(-web) binaries. There should (from the point of view of distributing the extension) not be the need for any other dependencies? I am not sure pub.dev is the best place for distributing these binaries, but it seems to me one strategy is to publish packages that contain just the precompiled extension and none of the source code or dependencies used for compiling the extension.

While this is true, this would not be an ideal development story for extension authors nor an ideal discovery mechanism for extension users. One of the benefits of how DevTools extensions are designed is that users get the extension for any package they depend on for free; they do not have to discover this in some other way. If the extension was shipped as a separate package, this benefit would go away completely and extension authors would likely struggle to get tool adoption. We saw this with the shared_preferences extension for example. Initially shipped as a tool separate from the shared_preferences package, users could not easily discover it. Once this tool was moved over to ship with the parent shared_preferences package, we saw usage drastically go up (now the second highest used DevTools extension).

Another benefit for authors of standalone extensions (extensions that do not ship with a parent package but rather are a standalone tool) is that development is simple and can be done in a single package. The source code and the precompiled extension that is shipped to users are in the same package. Requiring extension authors to ship as a separate package that only contains the binary seems like unnecessary overhead when we could just allow users to globally activate a single package to use the extension.

With no extensions there is no potential for conflicts?

Can you elaborate? I'm not sure I understand the question.

[sigurdm]

users get the extension for any package they depend on for free;

Right, but when you are using the package, you most likely want to resolve all the constraints along with the rest of the package - no? I was arguing about stand-alone extensions. The argument kind of extends though: the dependencies for compiling the extension itself should not necessarily be dependencies of the package. See suggestion for a nested package layout below.

The source code and the precompiled extension that is shipped to users are in the same package

I think I would suggest a nested package layout in that case such that the code for the extension is contained inside the package, but not really part of it. Something like:

pubspec.yaml # pubspec for distributing the extension, the one that is published. No dependencies.
  extension/
    devtools/
      build/
        ...  # pre-compiled output of the Flutter web app.
      pubspec.yaml # pubspec for the extension code itself, defines the dependencies for compiling the web app.
      lib/  # source code for your extension Flutter web app
        src/
          ...

Connecting the two packages into a pub workspace might make development bearable.

With no extensions there is no potential for conflicts?

Can you elaborate? I'm not sure I understand the question.

Sorry that was a brain-hiccup - I meant "With no dependencies there is no potential for conflicts" (the question mark was kinda rhetorical).

[kenzieschmoll]

Right, but when you are using the package, you most likely want to resolve all the constraints along with the rest of the package - no?

Not necessarily for standalone extensions. For example, someone may have built a tool that does something interesting with the connected VM service instance but that doesn't have any need to pub solve with the user's project. The DevTools extension is precompiled, so the dependencies for a standalone extension package should not matter for the end user. For non-standalone extensions (extensions that are tightly coupled to a parent package like provider or shared_preferences), the user would care about the extension-providing package's dependencies resolving with the rest of their project dependencies.

Re: nested layout above - it feels a bit strange to have an entire Flutter web app nested under the extension/devtools/ directory. Additionally, the build/ directory gets changed each time the Flutter app is built, so for local development, you'd have to blow up the contents of your extension build/ that you actually want to ship with your package, which feels dangerous.

I do think the topic of how to structure your extension for development is a separate topic than enabling extensions for globally activated packages. In many cases, it will depend on the extension author's use case, repository standards, and preference. So I propose we keep this issue focused on how to support DevTools extensions for globally activated packages and work toward a solution for that problem.

Essentially the ask for the pub team is this: can we have a way to list the package_config.json locations for globally activated packages? If we have that, we can use package:extension_discovery to find out if any globally activated packages provide an extension. I am not concerned about version conflicts, as we already have logic to de-duplicate extensions and take the latest available version.