Chicken and egg problem when evolving `dart:_macros` library

[jakemac53]

See https://dart-review.googlesource.com/c/sdk/+/353561, this exercises a pattern where we both add a new API to dart:_macros and expect to be able to use it in the compilers, atomically.

In particular, I added the exception types to dart:_macros in this CL, and migrated analyzer/CFE to use package:macros which exports dart:_macros at the same time. They expect these new exception APIs to exist, but they don't.

Presumably this is related in some way to the pre-built SDK, but I don't fully understand all the context here.

I could likely avoid the issue in this case, but we will have this same pattern but not be able to avoid it in the future.

cc @johnniwinther @whesse

[bwilkerson]

I don't see where macros is mentioned in the DEPS file, which means that it isn't in the package_config.json file against which the SDK code is analyzed / compiled.

Aside from that, you might want to consider developing package:macro in the Dart SDK. I suspect that doing so would prevent this and several other problems.

[jakemac53]

I don't see where macros is mentioned in the DEPS file, which means that it isn't in the package_config.json file against which the SDK code is analyzed / compiled.

See the relation chain, it is added one CL earlier :).

All the APIs are actually coming from dart:_macros though. This is actually what causes the problem. If it was purely a package it wouldn't be an issue. But dart: libraries are to some extent pre-compiled or something. I don't fully understand how the pre-built SDK exactly interacts with them. But APIs are not immediately available to the compiler itself, is what I am observing.

[jakemac53]

One idea here, but it might be hard to implement, is to somehow extract out the support for running macros such that it can compile the platform without referencing any macro related types (no imports at all to package:macros and thus no transitive ones to dart:_macros).

I don't know how feasible that would be though.

[bwilkerson]

See the relation chain, it is added one CL earlier :).

Ah, I missed that, sorry.

I have to wonder whether it might be cause by the chaining. That is, I wonder whether the second CL would be having a problem if the first had landed so that the SDK being used to analyze the second CL already had the package defined (in package_config.json).

[dart-github-bot]

Item	Details
Summary	Atomically adding an API to dart:_macros and expecting to use it in compilers causes an error.
Triage to	area-library, area-analyzer

(what's this?)

[jakemac53]

That is, I wonder whether the second CL would be having a problem if the first had landed so that the SDK being used to analyze the second CL already had the package defined (in package_config.json).

Most of the package works fine - it is only the new APIs added to it which don't work.

[mraleph]

The error makes sense: currently when bootstrapping we will to run HEAD CFE on the prebuilt dart - which does not contain any dart:_macros in the core libraries. This means CFE can't really use any Dart language features or APIs which are not yet in the core-libraries of the prebuilt dart.

For Dart only code (i.e. things that don't require or depend on VM changes) we can probably make it work by having a special mode where we compile HEAD CFE using core library source rather than precompiled platform embedded into prebuilt dart - but that will still break if compiling CFE requires invoking macros and there is an API skew between prebuilt CFE and macroses at HEAD.

[jakemac53]

It is, at least from my perspective, acceptable if we cannot use any macros in the CFE itself.

[mraleph]

We have discussed this problem with @johnniwinther a bit.

@jakemac53 Before I jump into nitty gritty details of how we could change bootstrapping I want to ask metaquestion: are we sure we want to make dart:_macros part of the platform file (i.e. core libraries)?. This seems to be compiler API rather than platform API and as such I find it a bit suspicious that we include it there. Consequently I would think it should live in a separate place.

Can we make it a builtin package instead (e.g. package the import of which is handled in a special way by the tooling)?

e.g. if you have you have import 'package:_macros/macros.dart' then

1. If package _macros is resolvable through package_config.json - then just use that.
2. Otherwise load from a special tool-specific builtin location.

Taking one step further: package_config.json could support referring to special builtin packages using custom scheme,

    {
      "name": "_macros",
      "rootUri": "dart-builtin://_macros/",
      "packageUri": "lib/",
      "languageVersion": "3.2"
    },

Then you don't need otherwise clause at all (but you would need pub changes - which I think are not that big, we already have special wiring for Flutter SDK there).

Resolving bootstrapping problem

First let me summarize the build process as we have it now:

• We have prebuilt Dart SDK at version v1 which contains a dart binary which contains inside VM, as well as prebuilt CFE and platform file (core libraries in Kernel binary form).
• We have CFE and Dart core libraries (aka platform) in source form checked out at version v2.

I am gonna write $\mathrm{CFE}_{v1}^S$ to denote CFE in source form ($S$) at version $v1$, similarly $\mathrm{Plat}_{v1}^K$ would mean platform precompiled to Kernel¹ ($K$) at version $v1$.

The current bootstraping process goes like this: we want to produce Kernel binary from platform sources (compile $\mathrm{Plat}_{v2}^S$ to $\mathrm{Plat}_{v2}^K$). To do that we try to execute $\mathrm{CFE}_{v2}^S$ (CFE v2 from source) on prebuilt dart binary and give it platform v2 sources:

$$\mathrm{Plat}_{v2}^K = [\mathrm{CFE}_{v2}^S / \mathtt{dart}_{v1}](\mathrm{Plat}_{v2}^S)$$

Executing $\mathrm{CFE}_{v2}^S$ on $\mathtt{dart}_{v1}$ requires compiling it to Kernel first and then running the result of that on $\mathrm{VM}_{v1}$:

$$\begin{align} \mathrm{CFE}_{v2}^K &= [\mathrm{CFE}_{v1}^K / \mathrm{VM}_{v1}](\mathrm{CFE}_{v2}^S + \mathrm{Plat}_{v1}^K) \\\ \mathrm{Plat}_{v2}^K &= [\mathrm{CFE}_{v2}^K / \mathrm{VM}_{v1}](\mathrm{Plat}_{v2}^S) \end{align}$$

Attempt to compile $\mathrm{CFE}_{v2}^S$ using $\mathrm{Plat}_{v1}^K$ is the step that fails.

The work around which I previously described would be to change this step to:

$$\begin{align} \mathrm{CFE}_{v2}^K &= [\mathrm{CFE}_{v1}^K / \mathrm{VM}_{v1}](\mathrm{CFE}_{v2}^S + \textcolor{red}{\mathrm{Plat}_{v2}^S}) \\\ \mathrm{Plat}_{v2}^K &= [\mathrm{CFE}_{v2}^K / \textcolor{red}{\mathrm{VM}_{v2}}](\mathrm{Plat}_{v2}^S) \end{align}$$

That is instruct $\mathrm{CFE}_{v1}^K$ to compile using v2 platform from source, rather than precompiled v1 platform from Kernel. At this point we would also want² to run the resulting $\mathrm{CFE}_{v2}^K$ on freshly built $\mathrm{VM}_{v2}$ rather than running it on $\mathrm{VM}_{v1}$ embedded into prebuilt $\mathtt{dart}_{v1}$ binary.

Currently prebuilt CFE is hidden inside the monolithic dart binary - it is part of kernel-service. I think we should instead produce a separate bootstrapping compiler tool $\mathtt{bc}$³ which supports compiling Dart source against source platform for bootstrapping purposes. Then bootstrapping could look like this⁴:

$$\begin{align} \mathrm{CFE}_{v2}^{K_{v1}} &= \mathtt{bc}_{v1}(\mathrm{CFE}_{v2}^S + \mathrm{Plat}_{v2}^S) \\\ \mathrm{Plat}_{v2}^{K_{v2}} &= [\mathrm{CFE}_{v2}^{K_{v1}} / \mathrm{VM}_{v2}](\mathrm{Plat}_{v2}^S) \\\ \mathrm{CFE}_{v2}^{K_{v2}} &= [\mathrm{CFE}_{v2}^{K_{v1}} / \mathrm{VM}_{v2}](\mathrm{CFE}_{v2}^S + \mathrm{Plat}_{v2}^{K_{v2}}) \end{align}$$

Here $\mathrm{Plat}_{v2}^{K_{v2}}$ and $\mathrm{CFE}_{v2}^{K_{v2}}$ are the final artifacts we want to ship with the SDK.

Footnotes

1. Strictly speaking Kernel binary format itself is versioned but we assume for simplicity that kernel format did not change between v1 and v2. ↩

2. The reason for running it on v2 VM is to avoid version skew between platform which was used for compilation and the VM which is used to run the resulting Kernel blob. Otherwise boostraping would break, for example, when we make changes to native parts of dart:io. This however makes an assumption that Kernel produced by v1 CFE is runnable on v2 VM. ↩

3. Note that we assume that v1 Kernel format can run on v2 VM. This means that bc tool itself can just be distributed as a v1 Kernel binary rather than a native executable. This unlocks bootstrapping on platforms which don't have prebuilt dart binary available. See also https://github.com/dart-lang/sdk/issues/51788 ↩

4. Here I made a choice to highlight that the Kernel produced by older CFE might not be the same as Kernel produced by newer CFE and we don't want to ship older Kernel with newer SDK. ↩

[jakemac53]

are we sure we want to make dart:_macros part of the platform file (i.e. core libraries)?

The important part, is that the version of the library is identical between the binaries shipped with the SDK and the users local version solve. You are correct in saying that if we supported SDK vendored packages in Dart (similar to Flutter), we could achieve this.

I am trying to not invent more work than is necessary though, because the feature is already large enough, and every additional feature adds additional risk and complexity. We already have dart: libraries, and they already achieve this same goal.

Note that I would also not want this to work how Flutter SDK packages work. They allow pub package dependencies, which is IMO a mistake (massive pain for both owners and users of those packages, they all become in effect SDK vendored due to pinning). Only other SDK vendored dependencies should be allowed. We also want proper versioning, that is the whole point of having the package as a layer on top of the dart:_macros library. The current vendoring strategy that flutter uses does not allow for versioning at all.

My question would be, what advantage does being a package provide over a dart: library? I am not currently aware of advantages other than being able to have pub deps, which I strongly believe to be a mistake and wouldn't want to replicate. It seems to me to be basically just a way to always have the dependencies come from source, and not much different from providing the platform from source.

If it allowed us to use macros in the CFE or something I would possibly be in favor, but I don't believe it would, we still have the issue of the package being pre-compiled as a part of the CFE in the pre-built dart binary. So the protocol would have to match up exactly in order to compile the CFE.

[mraleph]

My question would be, what advantage does being a package provide over a dart: library?

My point is that it does not have to be a real package - it just that package:-scheme already has flexibility of being retargetable via package config and that is something that all tools understand. I concede that package: does scheme does not have all the functionality we need so we will have to implement some additional stuff - and it might not be worth it for this particular case (though it might be worth it as a general feature).

I think what is more important here is that _macros does not look like something that belongs to the platform (core libraries) to begin with: it is part of the compiler itself and not interface to the execution platform.

When compiling the compiler dart:_macros have to resolve to the source that lives next to the compiler. When compiling with the compiler dart:_macros have to resolve to the library shipped with the compiler. Maybe this can be just special case for dart:_macros in the CFE, but it does seem like it is loosely related to the existing concepts.

This made me think that another option is to simply redirect package:macros itself in the SDK environment, e.g. we could do:

diff --git a/sdk/lib/_macros/api.dart b/sdk/lib/_macros/api.dart
new file mode 100644
index 00000000000..e9b4f9b2ac7
--- /dev/null
+++ b/sdk/lib/_macros/api.dart
@@ -0,0 +1 @@
+export 'macros.dart';
diff --git a/tools/generate_package_config.dart b/tools/generate_package_config.dart
index 3f9a24207b0..9dee64683d1 100644
--- a/tools/generate_package_config.dart
+++ b/tools/generate_package_config.dart
@@ -148,6 +148,12 @@ Iterable<Package> makePackageConfigs(List<String> packageDirs) sync* {
     var hasLibDirectory =
         Directory(join(repoRoot, packageDir, 'lib')).existsSync();
 
+    // Hack: reroute macros package to point directly into SDK sources.
+    if (name == 'macros') {
+      packageDir = 'sdk/lib/_macros';
+      hasLibDirectory = false;
+    }
+
     yield Package(
       name: name,
       rootUri: packageDir,

And then your CL should build just fine.