Consider creating macros repo; how to use it for dart_model

[davidmorgan]

Re: #3872 #3706 #3847, @jakemac53 what do you think, please? :)

Proposal

I propose that we create a new repository dart-lang/macros, and use that for

1. macros packages that don't need SDK internals, such as dart_model, and related tooling;
2. issues and docs related to the implementation rather than the language feature,
3. (maybe?) end to end testing and benchmarks that don't need SDK internals.

I propose to use dart_model and other macros packages from SDK tools via a version in DEPS pulling into third_party/pkg.

Discussion

This idea came out of thinking about where dart_model code should live, then noticing we could create a repo for it, then noticing that we are currently overwhelming the language repo with macro things when much of the noise is now about the implementation, not the spec.

Re: where dart_model should live, the package should be a "lowest common denominator" in the sense that we want everyone to depend on it, including internal SDK tools:

flowchart LR
Analyzer --> dart_model
CFE --> dart_model
package:some_user_macro --> dart_model

Loading

The analyzer and CFE want to depend only on code in the SDK, which suggests putting dart_model under pkg in the SDK repo. But, this has some pretty big downsides:

• dart_model wants to come with a bunch of related tooling, helper packages (codegen/schema validation), possibly utility packages (json_buffer) and probably additional macro-related packages (macro protocol) , ... squashing these into the flat pkg namespace is awkward.
• it doesn't look quite correct; if we develop dart_model under pkg then the analyzer and CFE always use it from main. That makes it look like the analyzer and CFE always have the latest dart_model; but that's simply not true: we care about cases where the user has an older SDK.

I suspect that keeping them physically decoupled so that the setup matches the design could work better. For example, it's perfectly valid to work on dart_model ahead of what the SDK supports, when adding support for unlaunched features; and it's valid to work on dart_model features that are only used by the client, with no relevance at all for the SDK. There is no obvious advantage to coupling these changes to the SDK test and release, and decoupling may have the advantage of encouraging a cleanly decoupled package and tests.

So, let's keep that physical separation by using dart_model like an external dep.

This layer of separation also opens possibilities around doing source transformations useful to the host, for example if we discover that it's convenient for the analyzer to have access to multiple versions of the package we can copy in multiple versions, renaming the old one(s), so they are available simultaneously.

Edited to refer to the DEPS mechanism for pulling in packages by version.

[davidmorgan]

Also @scheglov @johnniwinther for your thoughts as prospective users of dart_model, please.

[jakemac53]

Agreed that developing this in a separate repo is a better option than the SDK.

[davidmorgan]

Thinking this through a little more :)

We are going to want to make frequent breaking changes during v0, which with the suggested setup would require a tightly pinned dep from the analyzer, and then to publish dart_model and analyzer; much as with package:macros today.

For v0, how about instead we:

• Develop dart_model on a new repo as a standalone package as described.
• But then "release" it by copying into _fe_analyzer_shared/src/dart_model, (with updated import statements), and the analyzer and CFE use it from there?

We can also release to pub when it's convenient e.g. for experimentation but that would not be tied to the SDK for v0.

Reaching a stable v1 that we can depend on properly without it becoming painful would be a headline goal / milestone that needs to happen well before the feature release :)

What do you think please @jakemac53 @scheglov @johnniwinther?

[johnniwinther]

Would it make sense to do the v0 development directly in the sdk and then moving it to dart_model when it becomes more stable towards v1?

[davidmorgan]

@johnniwinther it's more convenient to have a separate monorepo for things like codegen, schema tooling, examples and benchmarks that don't need to be in the SDK at all. So without having tried it ;) I prefer the separate repo option at the moment, but happy to hear opinions/suggestions.

[johnniwinther]

I was just thinking that while developing from prototype to releasable version, having a boundary between the model and the primary use sites might slow down development. I does sound like a good separation, in particular long term.

[davidmorgan]

We have the repo now, so closing.

#3904 for next steps related to where we put / develop the code :)