Use the frontend server to compile pub executables

[jakemac53]

Opening for discussion, I wanted to try this out to see if we can speed up the rebuilds of pub executables - it does appear to be much faster (basically immediate if there is no change). Surprisingly, it is almost twice as fast on the initial precompile as well for me. I think that probably means that the vm snapshot functionality is doing something additional - I don't know what the implications of using kernel instead might be.

Specifically this:

• Creates a new dir .dart_tool/pub/incremental where it stores the previous dill outputs for each executable.
• Uses the frontend server to compile to that directory (which will use those files automatically to initialize from).
• Copies the output from there to the actual executable path if successful.

Previous snapshot behavior:

New fe server behavior, first build:

New behavior, pub get and run again (no changes):

Note that in the rebuild case with no changes, it didn't actually log the time at all which I believe means it was <1 second.

cc @mit-mit

[jakemac53]

Possible resolution to #2962 as well - in the case of no changes the precompile step is now much faster.

[jakemac53]

I will look more into the windows failures and do additional cleanup if we want to push forward with this change.

[jakemac53]

cc @a-siva would you be able to speak to the implications of running from kernel versus snapshot?

[a-siva]

Could you clarify what kind of 'snapshot' you are referring to here?

Usage: dart compile <subcommand> [arguments]

Available subcommands:
  aot-snapshot   Compile Dart to an AOT snapshot.
  exe            Compile Dart to a self-contained executable.
  jit-snapshot   Compile Dart to a JIT snapshot.
  js             Compile Dart to JavaScript.
  kernel         Compile Dart to a kernel snapshot.

(aot-snapshot, jit-snapshot or just plain kernel which is also sometimes referred to as a snapshot).

[jakemac53]

Could you clarify what kind of 'snapshot' you are referring to here?

It is just --snapshot - that must be an alias for some other listed flag?

pub/lib/src/dart.dart

Line 55 in 018c965

'--snapshot=$snapshotPath',

[a-siva]

if you are just using the --snapshot option and not passing in the --snapshot-kind option then the default is kernel

--snapshot-kind=<snapshot_kind>
--snapshot=<file_name>
  These snapshot options are used to generate a snapshot of the loaded
  Dart script:
    <snapshot-kind> controls the kind of snapshot, it could be
                    kernel(default) or app-jit
    <file_name> specifies the file into which the snapshot is written

which means you are already only producing a kernel file in this step.

[jakemac53]

Ok thanks @a-siva so there should be no difference in terms of the result then.

I do see some slight difference in output, the md5 hash is different and the new result is a bit smaller - (18016104 bytes versus 18633360 bytes).

[a-siva]

Is it possible that the version of CFE used in the two cases is different.

[jakemac53]

@jonasfj @sigurdm anything you are waiting for me on here? If I can get a general affirmative I will spend some more time cleaning things up (get tests passing, delete the old snapshot function, etc)

[jonasfj]

@jakemac53 looks great to me.. I'm not familiar with FrontendServerClient, but I assume it incremental compilation infrastructure...

Only concern is how this will interact when users switch SDK versions forwards and backwards... But that might not be so bad.. also it might affect global packages too in the same way.

[jakemac53]

@jakemac53 looks great to me.. I'm not familiar with FrontendServerClient, but I assume it incremental compilation infrastructure...

Yes, this is the same tool that flutter uses for incremental recompilation.

edit: just to clarify flutter does not use package:frontend_server_client, it has its own wrapper code. But they both talk to the same frontend_server binary which ships with the sdk and does the actual compilation.

Only concern is how this will interact when users switch SDK versions forwards and backwards... But that might not be so bad.. also it might affect global packages too in the same way.

The way I have implemented this it should all just work. I haven't changed the invalidation logic around the actual runnable executables, so they still get invalidated (deleted) in the same way as before.

When running from a cached dill file, the frontend server automatically throws it away if its kernel version doesn't match afaik. At least in some way it does handle this case and does a full recompile.

[jakemac53]

@jonasfj the remaining test failure is here https://github.com/dart-lang/pub/pull/2968/checks?check_run_id=2442925564#step:7:415 (same failing test on the 3 bots).

SocketException: Failed host lookup: 'arg.com' (OS Error: Temporary failure in name resolution, errno = -3)

Do you have any more details/info on this test?

[jakemac53]

This is ready for review now @jonasfj @sigurdm

[jakemac53]

@jonasfj This is ready to land from my perspective, but I will let you hit the button unless you want me to :D.