Define the macro API for introspecting on metadata annotations (and enum values)

[munificent]

The proposal has a section discussing this, but it's written like an open-ended design discussion. We should figure out what we actually want and then pin it down in the proposal. The key question is how arguments in metadata annotations are seen by the macro. Are they actual values, or unevaluated syntax? The former might not be possible (since it may refer to code produced by macros). The latter might not be useful.

[jakemac53]

I think we should basically do what the analyzer does here - provide a DartObject like object https://pub.dev/documentation/analyzer/latest/dart_constant_value/DartObject-class.html.

[GregoryConrad]

From a user's perspective, it would be very helpful to access annotations that depend on constant macro-produced types and declarations.

Use Case

Say I have:

class InfoAnnotation {
  const InfoAnnotation(Object obj);
}

@someFunctionDeclarationMacro
void fn(
  @InfoAnnotation(macroProducedObj) int someIntParameter,
) {}

In the above example, someFunctionDeclarationMacro could do something interesting with the macroProducedObj and I think that it is reasonable that someFunctionDeclarationMacro should be able to get the InfoAnnotation (and its members) from the ParameterDeclaration.

The Problem

I think I see why this would be problematic as-is: there is no way to know for certain whether this macroProducedObj has been declared yet in phase 2 (if I am wrong here, please do correct me).

Would it be possible to hint at the order in which some phase 2 macros are evaluated to allow for this? I would hate to have to resort to build_runner to handle something like this.

Proposed Solution

I have what I think is a good solution/compromise: a macro can directly specify what other macros it "depends on." I.e., In the example above, someFunctionDeclarationMacro "depends on" the macro that produces the macroProducedObj. For my particular use case, and I imagine most others, declaring macro dependencies would be sufficient.

Seeing as a macro that that depends on declarations produced by another macro would (in most cases) have access to the other macro, the above solution makes a lot of sense to me. It would act just like dependency resolution, giving a deterministic way to run macros in phase 2. You could even give users a helpful warning/error at compile time hinting that their macro should depend on another macro if the declaration the first macro needed was created later on in phase 2.

Could look something like:

macro class IntroducesSomeDeclaration {
// ...
}

@DependsOnMacros([IntroducesSomeDeclaration])
macro class UsesAnnotationWithSomeDeclaration {
// ...
}

Anti-Solution

An anti-solution would be for a macro class to ask to be run at a certain point in phase 2, perhaps even a "phase 2.5." This is non-deterministic, and because of that, nothing like this (IMO) should be supported.

[GregoryConrad]

I actually just realized my own use case relies on having the macro depend on other instances where it was applied; which is an issue. But, I know exactly what I want to declare in phase 1 given the FunctionDeclaration of the function the macro is applied to. And then in phase 2, my macro can utilize those phase 1 declarations from other macro invocations. Will it be possible to contribute more information in a phase than is required? That seems reasonable. I have some other ideas in which this would ease implementation (like when I just need to generate some quick library-specific boilerplate that I don’t want to expose publicly in a package).

Or, if you go the unevaluated syntax route mentioned above, would we be able to spit that out (as a string) in a code declaration elsewhere verbatim? For my case, I don’t really need to know anything about the metadata other than the name (as a string) of the object it contains. It would be nice to have other information, but I could get by without.

Ideally, in a final API, I’d love to be able to do either of what I just mentioned. I can give a code example of each tomorrow to help illustrate what I’m suggesting

[jakemac53]

Question: Is the macroProducedObj generated from a macro in the same library, or a dependency?

We do already have a general mechanism for taking an unresolved piece of code, and emitting that unresolved code in your own generated code. It is better than just copying the syntax because it will actually ensure all the contained identifiers resolve to the same thing as the original (the augmentation library where macro code goes has its own scope, so this is necessary).

[GregoryConrad]

Is the macroProducedObj generated from a macro in the same library, or a dependency?

Either is possible. It could be specified in the same library, or it could be specified in an imported library (which the user must import). Here's a concrete example:

// library a.dart

@myMacro // generates: const someIntObject = ...;
int someInt() => 0;


// library b.dart

import 'a.dart';

@myMacro // generates: const someIntPlusOneObject = ...;
int someIntPlusOne(
  @C(someIntObject) int i,
) => i + 1;

@myMacro // generates: const someStrObject = ...;
String someStr(
  @C(someIntPlusOneObject) int i,
) => i.toString();


// From some other package:
macro class MyMacro {...}

class C {
  const C(this.object);
  final MyCustomObject object;
}

In the above example, myMacro would need to:

1. Access the class of the supplied metadata annotation, in this case C.
2. Be able to reproduce the contents of @C(...) in generated code, i.e.:

int _userDefinedFunction(
  @C(contentsOfTheAnnotation) int a,
  @C(contentsOfAnotherParameterAnnotation) int b,
) => 0;

int _$generatedFunction() {
  final d1 = doSomething(contentsOfTheAnnotation);
  final d2 = doSomething(contentsOfAnotherParameterAnnotation);
  return _userDefinedFunction(d1, d2);
}

[jakemac53]

If all you need is access to the code chunk representing the argument to the annotation, then yes this should be feasible, regardless of if its referencing generated identifiers or not. But note that annotations can only reference const objects, so that may be a limitation you run into as well.

[GregoryConrad]

Great, thanks! Gives me good faith for a package I'm working on based around macros. The current prototype implementation is already using const objects at the top level, as I was planning on that being a possible issue with annotations from the start.

[GregoryConrad]

Not sure if this is the right issue for this question (putting it here because it is related to accessing a member of an annotation/macro), but say I have a macro like:

macro class MyMacro<T, R> {
  const MyMacro(this.func);
  final R Function(T) func;
  // ...
}

Would I then be able to introspect on the func provided to the macro (for this case, we can assume func is not generated by another macro)? I don't see a builder API for that (but perhaps I missed something).

Specifically, can we introspect/use T and R in generated code, and pass the given func into generated code?

Alternatively, if that is not possible, could we pass in an instance of a class type or similar into a macro and then be able to introspect on it, like:

class A {}

macro class MyMacro {
  const MyMacro(this.clazz);
  final Type clazz;
}

@MyMacro(A)
extern void fn();

[jakemac53]

@GregoryConrad See https://github.com/dart-lang/language/blob/master/working/macros/feature-specification.md#macro-arguments which describes how arguments of various types to macros are handled.

This is an interesting corner of the proposal to be sure, feel free to ask additional questions about that if you have them.

The basic summary though is that for function arguments the only really useful thing you can accept is an Identifier parameter, which would be a reference to a function, that you could then use in generated code. You are not allowed to actually invoke the function inside the macro itself, because macros run as a separate application at compile time, and only have access to the code from their own transitive imports. User code is not available other than as abstract references (Identifiers or other Code objects) which can be emitted into generated code.

[jakemac53]

It looks like we don't talk about generic type parameters for macro classes in the proposal today, I can add a note. For now at least they are going to be disallowed, for the same reasons as not being able to run user defined functions. The program in which the macro is actually running will not have access to types from the users code.

[jakemac53]

Updated the title, I was just starting to go through and update the proposal for enhanced enums and I realized enum value arguments are very similar to metadata annotations. Users may want to dig into the values provided to the constructor.

[jakemac53]

@johnniwinther As I start working on this, I am wondering if the lazy constant evaluation of some platforms might make it not possible to allow macro authors to get the actual const values of annotations?

[jakemac53]

Also related, @johnniwinther @scheglov how would you feel about a general API for evaluating ExpressionCode instances? It would produce an error if the expression wasn't a valid const expression, or if there were any references to things which didn't yet exist (or didn't yet have a body/initializer).

The result would still always be a DartObject style object so that we could represent objects from user code that don't exist in the macro program.

[jakemac53]

Yet another hiccup I just realized if we do allow const evaluation - const variables might actually be augmented? Or at least we don't block that today. That means the value could change in phase 3 compared to the value previously.

I am somewhat leaning towards just blocking augmenting const variables at all? This would mean any const expression can always be evaluated in any phase safely, as long as all the references to variables within it have been defined (I would still allow generating new const variable declarations within augmentations).

cc @munificent

[jakemac53]

For anybody following along, I have the first PR of two sent out here https://dart-review.googlesource.com/c/sdk/+/313640. This just gives the very basic level of introspection, without const evaluation. Basically enough to know "what kind of metadata is this?". I also chose not to try and give direct access to the arguments, because I think the fields on the evaluated object will be more useful.

Const evaluation will be in a followup.

[GregoryConrad]

I also chose not to try and give direct access to the arguments

Const evaluation will be in a followup.

@jakemac53 Just want to check on something based on what you said here; I recently filed #3210, and was wondering if that would be compatible with the work you're doing here. It'd be nice to be able to re-emit the const object (or in this case function) that is being used as the annotation:

void foo(){}

@myMacro
void bar(
  @func int foobar,
) {}

// myMacro will be able to re-emit the `func` from the `@func` in its generated code

Thanks.

[jakemac53]

Yes - that would be somewhat supported already with the first PR I just sent out yesterday. The MetadataAnnotation class has two subtypes, IdentifierMetadataAnnotation and ConstructorMetadataAnnotation, this would just be an IdentifierMetadataAnnotation so you would get an identifier for the referenced function and could emit that in Code objects (and invoke it, etc).

The main thing missing right now would be the ability to figure out what the type of that identifier is, (ie: make sure it's a function). This is a more general hole I think, we don't have the ability to go from an identifier to its declaration for anything except types. It will take a bit of thought just because we really don't want to allow doing that until phase 3 (not all declarations exist until then), but I will add an issue for it.

[johnniwinther]

Supporting const evaluation of arbitrary metadata for the introspection is a can of worms that I'd rather that we avoided if possible. Yes, it does conflict with lazy evaluation of constants (for the fromEnvironment constants) but it'll probably also add (even more) complicated dependencies between declarations.

For instance, is this a circular dependency?

@patch
class Object {}
class Patch extends Object {
  const Patch();
}
const Patch patch = const Patch();

Or this?

class Class {
  final int value;
  const Class(@Class(0) int value) : value = value + 42;
}

[jakemac53]

@johnniwinther What if we only allowed evaluating constants from libraries not in the current strongly connected component?

There are strong use cases for this, many Builders today use annotations to configure things. But almost always those annotations are defined in the same package as the Builder which should avoid most of these issues.

[jakemac53]

See #3408 which handles this by allowing general purpose const evaluation with the following constraints:

• No identifier in code may refer to a constant which refers to any
  system environment variable, Dart define, or other configuration which is not
  otherwise visible to macros.
• All identifiers in code must be defined outside of the current
  strongly connected component (that is, the strongly connected component
  which triggered the current macro expansion).

Hopefully the further restrictions on environment variables and Dart defines also helps here.

I need to do a bit more work for the enum values portion though.

[jakemac53]

I looked at enum values a bit today, but given that we allow you to fill in the definition of an enum value in the definition phase, I don't think we should give access to them directly in the EnumValue API.

We do still need to add a way of getting the arguments for a metadata annotation.

It will also still be possible to get the constant value of an enum through const evaluation (if the enum exists in a different strongly connected component).

[davidmorgan]

Closing in favour of #3847 as this falls into the broader topic of macro metadata.