-
Notifications
You must be signed in to change notification settings - Fork 214
Consider generalizing macros over applications and annotations #3873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
A macro can sort of do this already on a single library... although you end up needing an extra annotation. A macro can run on an entire library, and then introspect to find the annotations it wants to generate code for, share code/work as desired, and alter anything within the library.
Within a single library, probably it would be fine. I wouldn't want to generalize this to all libraries in an app though, because then there is the (I would argue significant) advantage that it encourages writing code which is deterministic and not reliant upon the specifics of how a given compiler works, or whole world versus incremental versus modular compiles etc. As a concrete example, consider this issue which was recently filed. A macro author with access to the entire world at once is likely to want to write macros like this, which require global knowledge, and either just won't work in certain modes (hot reload) or will be slow, non-deterministic, etc. The current model where each application is treated in isolation helps macro authors stay on the rails and write macros that will work consistently, as well as avoiding anti-patterns such as global queries. |
It's easy to offer libraries on top that make it look like execution is in isolation: And these can then do optimization behind the scenes, leaving the option to do something more complex only for those who understand it / want it. I think that then combines the advantages of both approaches :) |
I do believe #3854 is a very important scenario. I myself have a similar need in many occasions. For instance, I want to generate DB binding for a model .... which includes a schema migration. Generating one function per model/library would be unrealistic, because it'd be very error prone (easy to forget to invoke one library's function). I think that's fine if the generated code is often invalidated and slower to generate. It's important to be able to do it at all. |
Thanks Remi. Are you saying you want one macro application to do all that work--or is it okay to have one macro applications per data model class? I think the latter can work nicely: each model class application can output a function for that model class, then a whole program application can output code that aggregates--calls them all. That won't be super fast but you only need one in your program so rerunning it all the time should be fast enough. |
I don't care too much about the exact details. Personally, I'd have one annotation per model ... and maybe one on the "main" (that would look-up every files in the package). But if a different pattern enables implementing this, I don't mind. On that topic, is there a certain library order when dealing with generation over multiple libraries? For instance, given: // lib/main.dart
import 'src/model.dart';
@macro
void main( ){}
// lib/src/model.dart
@anotherMacro
class Foo {} Would the macro on Since macros can use IO a bit, I was wondering if |
Execution order mostly depends on imports: as long as there are no imports from model.dart onto main.dart, model.dart macros can execute and complete before main.dart macro run at all. If there is a cycle of imports (a "library cycle") they have to run "in parallel" in some sense. |
Any chance we could have a mechanism that gives macros a fixed order, even when libraries are not related through an import? One way could be to order files based on the path depth. Such that So |
Dart requires everything to be findable via imports from an entrypoint, so that would require a change independent of macros. |
It would be findable via imports from the entrypoint. @macro
void main() {}
// ...
import 'src/model.dart' as prefix0;
import 'src/model2.dart' as prefix1;
augment main() {
prefix0.Model.migrate();
prefix1.Model2.migrate();
...
} |
I don't think that's possible. But anyway it's unrelated to this issue, which is for a specific discussion about the work in progress. |
Moved to the "breaking changes" milestone with the expectation that we close this discussion as part of finalizing the host<->macro protocol, not that we do it. |
I do think we shouldn't try to do this right now, but we could investigate whether it could be done later on. So, whether we just close it or move it into a different bucket either is fine with me. |
I think this will naturally come with the switch to A macro's initial query, whether explicitly or implicitly made, probably includes the class the annotation is applied to, then everything reachable from that in a particular way. (e.g. types mentioned in fields). It's not obvious why a macro would want to limit that to just the one application, because that would split the same work over more requests+responses, and in some cases introduce duplicate work. Instead it might ask for: all the applications in one file; all applications that are being applied right now; all applications in the program. I think it's not so much about implementing new stuff as about skipping the limitation that once macro application has to match up with one round trip on the wire. It's easy to get that simplicity again if you want it, example code that splits the data and delegates for each application. |
I think that this is actually quite a complicated feature. One problem for example is ordering, and introspection loops. If a macro can be applied at different "layers" - lets say a top level function or a method - those don't run at the same time. The method should fully run first. As far as introspection loops, it greatly increases the chances of one happening if the granularity of macro applications is larger. You could have a DAG in terms of the actual dependencies of two classes, but since macro applications on each are now grouped together, you get a cycle where one didn't exist previously. |
Yes, that's fair; maybe it only makes sense in combination with "rounds". Which I think also come under "only do this if it turns out to be easier" :) |
A difference that cropped up naturally in the
dart_model
exploration, and seems worth considering independently of that work.Generalize over applications:
Consider having one macro "runtime instance" handle multiple applications.
There isn't any particular advantage to macro code executing in the context one of application: rather, all applications of a macro
Foo
are runs of the same code that in the worst case can certainly manage not to interfere with itself, and in the best case can benefit from sharing work.For example, suppose that the
Foo
macro must introspect and make decisions about all the types used as a return type in the classFoo
is applied to.Then, there is a high chance this work will be duplicated across
Foo
applications, because different classes usingFoo
are very likely to have some of the same return types. Sharing the work across applications can lead to significantly better scalability.This can also lead to opportunities to batch requests and responses for better serialization throughput.
Generalize over annotations:
Consider having once macro "instance" handle, if so configured, multiple annotations.
There is also no particular advantage to tying macros to a single annotation. One macro package might offer annotations
Foo
andBar
that care about each other: different aspects of the same generated code. Then, there is no particular reason to force everyFoo
to redo the work done byBar
, when it could simply be shared. So: support using the same macro code for bothFoo
andBar
and running it on allFoo
and allBar
.Merging
Foo
andBar
into a single macro in this way also simplifies the job of the host, because they are no longer separate macro applications each of which might produce output visible to the other; they share information internally and must produce one unified set of augmentations, leaving no more work for the host.Generalize fully?
Probably not. To generalize fully in this direction would mean:
And this seems too much:
#1
and#3
are too surprising.A reasonable compromise seems to be to restrict
#1
and#2
back to focusing on annotations:What about modular builds?
The host should communicate which applications are "read only": i.e. the augmentation has already been written in an earlier build step; the macro does not have to produce the augmentation again but is free to introspect that application if it wants.
The text was updated successfully, but these errors were encountered: