[native_assets_cli] fetch and offline mode

[dcharkes]

@reidbaker Suggested we should have fetch hook/flag, and an offline flag.

I've spend some thought on what would be a good way of doing this.

One design goal should be that a build hook can wrap an existing native build system or native package manager (#189). This leads to the following design constraints:

• The directory between fetch and build should be shared. Otherwise the fetch step invoking the fetch step in the native tool would download dependencies to a place where the build step cannot access them.

Another design goal should be that after the fetch step, the build step wouldn't need any internet access. In other words, it would succeed offline. This leads to the following design constraints:

• The fetch step needs the same configuration as the build step. This includes build mode, link mode preference, OS, architecture, minimum API versions for NDK/iOS/MacOS. Any of these configuration parts could lead to different assets as output. If libraries are prebuilt and downloaded in the fetch step, they fetch step needs to know these configuration options.
  • (As a side note, this means an implementation for Flutter for invoking the fetch step would require roundtripping through the Flutter app Gradle/XCodeProject to get some of this info. flutter run doesn't have target architectures available for example, only flutter assemble does.)

I'm not sure if the best design would be to have a hook/fetch.dart which would get a FetchConfig (most likely no output, just exitCode of the process for success or not). Or if BuildConfig should get a bool fetchOnly.

Pros for fetch as flag on hook/build.dart

• If fetchOnly is false, it's a single process invocation for fetch and build, instead of two.
• Hook writers that use helper packages for a native build system, can simply forward to a helper package which will do the right thing (package:gradle), fetch on fetch, build on build, offline on offline. If we have a hook/fetch.dart and a hook/build.dart there'd be two hooks which forward with the same configuration.
• It's 100% obvious that fetch and build hooks have access to the same configuration, and the same output directory.

Cons for fetch as flag on hook/build.dart

• It's less explicitly split.

For offline, I think a bool on BuildConfig is the right place.

And offline and fetch are related in the sense that after fetch the build hook should not need any internet access. Which passing the offline flag would enforce.

@HosseinYousefi @mosuem @liamappelbe @brianquinlan In your use cases, do you wrap existing native build systems / native package managers that need internet access to fetch dependencies? (To produce dylibs/jars/...) And if yes, do those have a separate fetch step and an offline mode?

Any other concerns we should take into account for a fetch step/flag and offline flag?

cc @mkustermann @stuartmorgan

[mosuem]

To be had in a separate discussion probably:

If fetchOnly is false, it's a single process invocation for fetch and build, instead of two.

That we do many process invocations for hooks is a problem in itself, which I believe should be addressed at some point (Maybe collect all hooks into a single Dart file?).

[brianquinlan]

In your use cases, do you wrap existing native build systems / native package managers that need internet access to fetch dependencies? (To produce dylibs/jars/...) And if yes, do those have a separate fetch step and an offline mode?

Not in any use case that I've encountered so far.

[dcharkes]

Notes from a discussion with @mkustermann regarding support for working offline and sandboxing.

• Flutter lazily downloads artifacts. We'd like to do something similar in the Dart SDK. The way to ensure all the elements are there is to run the Flutter (or Dart) command once with internet access. After that the cache should be hit and internet access should not be needed.
  • Hooks that download their binaries from the web should behave the same way. They should check if the binary is already on disk (and has the right hash), and only try to access the internet if it needs to be redownloaded. Running the same Dart or Flutter command should always result in running the hook with the same code config.
  • Hooks that wrap other build systems need to rely on such other build system doing the right thing when being reinvoked.
• If we want to do sandboxing at some point, we should do so on the OS level so that subprocess invocations cannot escape it.
  • The internet access should then go through some Dart program that allowlists certain URIs, and only allows downloads.
  • The disk access should then be read-only of the whole disk except for allowlisted directories (output directory, possibly a native package manager cache directory).

[stuartmorgan-g]

• The way to ensure all the elements are there is to run the Flutter (or Dart) command once with internet access.

There is also a specific flutter command to pre-cache things, precisely to allow someone to run it once with internet access without actually doing an entire build. As I've previously said, one of the reasons I strongly believe that fetch should be a separate step from everything else is to allow for a similar command to exist for native assets.

[HosseinYousefi]

In your use cases, do you wrap existing native build systems / native package managers that need internet access to fetch dependencies? (To produce dylibs/jars/...) And if yes, do those have a separate fetch step and an offline mode?

Yes, in the case of jnigen this is useful for downloading jars from maven/gradle/... The same hook could be used to generate bindings from the said jars.

However, for jnigen, it's important that this hook is run while development and not while building, because the user would want to use the downloaded jars and their generated bindings in their code and they shouldn't be forced to compile and run the entire app once in order to have access to this.

[dcharkes]

• The way to ensure all the elements are there is to run the Flutter (or Dart) command once with internet access.

There is also a specific flutter command to pre-cache things, precisely to allow someone to run it once with internet access without actually doing an entire build. As I've previously said, one of the reasons I strongly believe that fetch should be a separate step from everything else is to allow for a similar command to exist for native assets.

TIL flutter precache! Thanks @stuartmorgan! It doesn't precache anything app-specific as far as I can see. E.g. it doesn't run gradle dependencies for the Android app. Have we thought about doing something like that? @reidbaker

[reidbaker]

Yeah we have but keeping gradle working has been the higher priority. FWIW I think do we actually download gradle but just not the builds dependencies. Gradle shares a gradle cache across the machine by default so for some users their dependencies are already there.

[dcharkes]

I think do we actually download gradle but just not the builds dependencies.

I didn't mean precaching Gradle itself, but the dependencies that are in an app's build.gradle. Such that the next invocation of Gradle doesn't need to download any JARs from the internet.

(Edit: Though Gradle itself of course should as well be cached. In the same way that for example the Rust toolchain should be cached if a build hook wants to use Rust to build a dylib.)

Gradle shares a gradle cache across the machine by default so for some users their dependencies are already there.

That is is similar to what @mkustermann and I mentioned above. On the second run of a flutter build/run all dependencies that are downloaded during the build should be already there. (Not machine-wide, instead pub-workspace-wide.)

[stuartmorgan-g]

TIL flutter precache! Thanks @stuartmorgan! It doesn't precache anything app-specific as far as I can see. E.g. it doesn't run gradle dependencies for the Android app.

This is true, but also as discussed previously, there are gradle commands that can be used to precache dependencies.

Currently, flutter/plugins is able to drive almost all downloading steps from a separate CI step, so that we can isolate network-based failures from actual test failures. We do this via a combination of things like pub, gradle, and cocoapods commands.

If native assets become commonplace without similar isolation being possible, it will be a regression for our packages CI.

(And to capture another important point from previous discussions: I'm fully aware that it is already possible for packages to do arbitrary downloads during builds; there is a huge difference, in practice, between "it is possible for packages that don't follow standard practice to cause downloads to only be driven by doing a full build" and "the standard (or only) way of doing downloads can only be driven by doing a full build".)

[reidbaker]

As an example here is a build where we have a gradle cache step where we download all the gradle dependencies.

It is just not part of the flutter tool. https://ci.chromium.org/ui/p/flutter/builders/staging/Linux_android_emu%20flutter_driver_android_test/2810/overview

[dcharkes]

Currently, flutter/plugins is able to drive almost all downloading steps from a separate CI step, so that we can isolate network-based failures from actual test failures. We do this via a combination of things like pub, gradle, and cocoapods commands.

Ah, I see. So we'd want to have something like flutter pub get && flutter native-assets precache && gradle dependencies && ...

(Thanks for being patient with Dart-only folks, we basically only have dart pub get on our CI. And pub is pretty reliable.)

[dcharkes]

For a prefetch hook:

• @goderbauer: flutter hooks prefetch could have the same logic as flutter run for deciding for what OS/arch to pre-cache. And it should have arguments for os/arch for prefetch code assets to explicitly state the target OS and architectures.
• After some more thinking:
  • The contract for the hooks would be: The fetch hook might or might not get invoked before a the build hook. If the fetch gets invoked by the SDK, then the build hook should not access the internet.
  • The contract for the SDK: The fetch hook should get the same configuration as the build hook (e.g. asset types, and asset-type specific configuration). Otherwise a fetch cannot be done. For code assets this includes a target architecture and target OS. And for code assets this might need to include a OS API level etc. (The latter is currently not available in flutter tools, only in the native build.)
  • The HookConfig for both hooks should be identical.
  • The fetch hook has no access to metadata.
  • The output directory between the fetch hook and build is shared. (sharedOutputDirectory)
  • The fetch hook has no output.json. (Only an exit code.)

so that we can isolate network-based failures from actual test failures

RE: Recognizing network failures: An alternative approach without a prefetch hook could be:

• hook/build.dart outputs a status that can be success, build_failure, or infra_failure. 🙏 @HosseinYousefi
• flutter_tools outputs a status that can be build_failure or infra_failure.
• LUCI gets support for conditionally making a step being reported as a build failure or infra failure. 🙏 @goderbauer