package:native_toolchain_cmake

[dcharkes]

We already have native_toolchain_c in this repo, https://github.com/irondash/native_toolchain_rust, and https://pub.dev/packages/native_toolchain_go.

It would be cool to have a class CMakeBuilder in package:native_toolchain_cmake that is to be used in the hook/build.dart hook in a similar way to the CBuilder.

Looking for contributors to own this package! (The package doesn't have to live on this repo.)

(Side note, also looking for an owner for Bazel #1260)

[dcharkes]

I think a cmake implementation could be quite easy and maybe even support web if this package would run in the web case too. Being able to trigger the emscripten build from dart so that the files are prepped to land for another process to interface with (like flutter pubspec) would be super helpful to devs looking to support web.

[...]

I think to enable cmake with the new format I would simply have to have a process runner that runs the cmake generation and build commands in the specified directory with user provided args? As a stretch goal the builder could have default settings that provide platform specific args for easy cross-build. Any thoughts?

Originally posted by @MichealReed in #129757

Let's continue the conversation here.

I think to enable cmake with the new format I would simply have to have a process runner that runs the cmake generation and build commands in the specified directory with user provided args? As a stretch goal the builder could have default settings that provide platform specific args for easy cross-build. Any thoughts?

The builder should get it's targetArchitecture and targetOS from package:native_assets_cli's CodeConfig for non-web builds.

Once we add support for #156 (in this repo, in Dart, and in Flutter), package:native_toolchain_cmake can be extended to cover the web.

[Levi-Lesches]

What would be needed for such a package? I'm not exactly an expert in writing CMake files, but I've had to get more familiar with the process of building complex CMake-based SDKs for Dart FFI, such as OpenCV, Intel's RealSense, etc., so I could be of help here.

[dcharkes]

My very simple idea would to start with

import 'dart:io';

import 'package:logging/logging.dart';
import 'package:native_assets_cli/native_assets_cli.dart';
import 'package:native_toolchain_cmake/native_toolchain_cmake.dart';

void main(List<String> arguments) async {
  await build(arguments, (input, output) async {
    final builder = CMakeBuilder(
      cmakeLists: 'src/CMakeLists.txt',
    );
    await builder.run(
      input: input,
      output: output,
      logger:
          Logger('')
            ..level = Level.ALL
            ..onRecord.listen((record) {
              print(record.message);
            }),
    );
  });
}

And see if we can wire everything up from the CMake build to the hook:

• output:
  • The list of dynamic / static libraries produced to be added to output.assets.code.
  • The list of source files and include directories to be added to the output.dependencies.
• input:
  • cross compile to the right architecture and os for input.config.code.targetOS/Arch.
• Fail the build hook if the CMake build fails

And probably start with the simplest of CMake builds: With a single C file. (Possibly the one from flutter create --template=plugin_ffi.)

[MichealReed]

Once we add support for #156 (in this repo, in Dart, and in Flutter), package:native_toolchain_cmake can be extended to cover the web.

Not sure I understand the need to wait? Having build orchestration on web, even if not linked by dart is still immensely helpful. Although generated bindings would be nice, the pattern of using a simple loader with non-generated bindings is still solid for web if we can build the libraries.

https://github.com/MichealReed/minigpu/blob/master/minigpu_web/lib/web/minigpu_web.loader.js

cross compile to the right architecture and os for input.config.code.targetOS/Arch.

Is this set up to call the builder multiple times if a platform demands multiple architectures like when building an APK?

What would be needed for such a package? I'm not exactly an expert in writing CMake files, but I've had to get more familiar with the process of building complex CMake-based SDKs for Dart FFI, such as OpenCV, Intel's RealSense, etc., so I could be of help here.

Happy to collaborate on this or leave it to you. I've recently noticed the lack of CMake integration with iOS/MacOS for native libraries, so I might work on this instead of battling cocoapods.

[dcharkes]

Not sure I understand the need to wait? Having build orchestration on web, even if not linked by dart is still immensely helpful.

I see, you want to invoke the hook/build.dart from your own build orchestration. 👍 Yes we should support that! The hook system is explicitly designed to work with frameworks that have their own bundling logic in mind!

You should then use package:native_assets_builder in your build orchestration code to invoke all the hooks for you. And you'd pass buildAssetTypes a 'wasm' asset type. And pick this asset type up in package:native_toolchain_cmake and emit a WasmAsset.

[MichealReed]

Not sure I understand the need to wait? Having build orchestration on web, even if not linked by dart is still immensely helpful.

I see, you want to invoke the hook/build.dart from your own build orchestration. 👍 Yes we should support that! The hook system is explicitly designed to work with frameworks that have their own bundling logic in mind!

You should then use package:native_assets_builder in your build orchestration code to invoke all the hooks for you. And you'd pass buildAssetTypes a 'wasm' asset type. And pick this asset type up in package:native_toolchain_cmake and emit a WasmAsset.

More like, I would want the hook/build.dart to invoke my build orchestration. I see on pub.dev native_assets_builder does not show web support, can I use it regardless? Linking into dart matters less for web/wasm but it is still useful to trigger a subbuild when the dart is building so the expected files can be in place when dart expects them.

[dcharkes]

The high level idea is as follows. There are two layers:

1. Dart without any assets (basically dart path/to/some.dart and dart compile)
2. Frameworks that care about assets and know how to bundle assets
   a. Flutter
   b. dart build exe and dart run my_package:my_package
   c. Web frameworks
   d. Custom Dart embedders

dart compile js and dart compile wasm are in category 1. So they will not invoke any hooks. If they would invoke hooks, they would not know where to place the assets.

The idea is that custom web frameworks (such as Jasper, Angular and friends) would invoke package:native_assets_builder and consume the assets that are output to bundle them correctly.

More like, I would want the hook/build.dart to invoke my build orchestration.

If you have your own web framework, it should invoke hooks instead of be written in a hook. And most likely if you have your own web framework your bundling logic should be in a bin/my_webframework.dart so that other packages can use the command line.

If you have a wasm asset that should be consumed by web frameworks, those web frameworks should start invoking the build hook. You can then indeed write your build script that builds into wasm in the hook. Currently there are no frameworks that consume wasm files and bundle them correctly, so such a hook would never be invoked.

I don't know much about how web development goes with Dart. What is your setup? Do you use a framework that is responsible for bundling assets?

[MichealReed]

dart compile js and dart compile wasm are in category 1. So they will not invoke any hooks. If they would invoke hooks, they would not know where to place the assets.

They do not need to know where to place assets, even a static expected path is sufficient. With CMake and other build platforms the outputs can easily be found and placed wherever expected. Currently Flutter only triggers CMake when building for Windows, Linux, and Android.

With iOS and MacOS I must prebuild via CMake with s.prepare_command or s.script_phases in the podspec so that the framework is available to be linked. Flutter only triggers via cocoapods.

With web, I must include the libraries, there is not a way to trigger a library build at all to my knowledge.

I don't know much about how web development goes with Dart. What is your setup? Do you use a framework that is responsible for bundling assets?

Most WASM libraries are built using emscripten via emcmake. Emscripten will output the wasm and the JS bindings. These JS bindings can easily be loaded with a simple script like linked above. Once the bindings are loaded, the library simply works from dart, no additional linking needed. This would be the same in Flutter and Dart.

At the simplest, plugins would benefit from a basic pre-build step that allows them to invoke and wait for an external builder before building the dart. Even on Windows I need to manually place files where Flutter expects

if(MSVC)
    add_custom_command(TARGET ${MAIN_LIB} POST_BUILD
        COMMAND ${CMAKE_COMMAND} -E copy_if_different
                ${DAWN_BUILD_DIR}/$<CONFIG>/webgpu_dawn.dll
                ${RUNNER_FOLDER}/$<CONFIG>/webgpu_dawn.dll
        COMMENT "Copying webgpu_dawn.dll to runner folder"
    )
endif()

Most external builders can easily place the files where dart expects. For simple native libraries, yes we can recreate the build process via dart, but for a complex library like Google Dawn, we must use their existing build process in tangent with dart/flutter and use post-build macros to find and place the files where expected by dart/flutter.

If my package could automatically run a pre-build script, I can put the libraries wherever needed for dynamic loading.

Like in the web case where I need to have my assets built and available for the flutter pubspec so that my loader can load the JS bindings for the WASM

  assets:
    - packages/minigpu_web/web/minigpu_web.loader.js
    - packages/minigpu_web/web/minigpu_web.wasm
    - packages/minigpu_web/web/minigpu_web.js

[dcharkes]

Even on Windows I need to manually place files where Flutter expects

Like in the web case where I need to have my assets built and available for the flutter pubspec so that my loader can load the JS bindings for the WASM

This should be solved as part of adding support for JS and WASM assets to Flutter. We should not be outputting these assets into a path which we are also predefining in the pubspec, that is duplicating information in two places. The design for the assets system is to access the assets by asset id and have Flutter/Dart/frameworks/embedders take care of the mapping from asset-id to loading the asset.

The hook does not know what the right place is. In Flutter it might be the web/ or the assets/ directory. But if you use the JS or WASM assets in another framework they might need to be in a different place. We don't want to bake Flutter/Dart/framework/embedder knowledge into the hooks. That is not a scalable solution. So that's an explicit non-goal for the build hooks.

Have you played around with flutter create --template=package-ffi? There you can see how CodeAssets are accessed in Dart code by asset-id. And how the hook output.json is not putting the assets in the right place but Flutter is.

So the end goal is to support JS and WASM assets in Flutter/Dart/frameworks/embedders. Have the hook produce and report them, and have Flutter/Dart/frameworks/embedders consume the list of assets, put them in the right place and provide the lookup mapping from asset-id to where they can be loaded from.

I understand that your current setup does not involve a framework, and you just want to automate running a script so that one step is not manual. I don't know a way to automate such step for your use case. My best suggestion is to run your app via a tool/build.dart that invokes Flutter/Dart and invokes CMake, and run from that. (Of course this doesn't work for users of a package, for them it should be then bin/setup.dart and your documentation needs to mention they should run that.) Those scripts can then call the hook/build.dart or call package:native_toolchain_cmake directly.

With web, I must include the libraries, there is not a way to trigger a library build at all to my knowledge.

I think this is true.

And as I have explained above, triggering a build automatically but not consuming the assets in Flutter/Dart/framework is a non-goal for hook/build.dart.

tool/build.dart or bin/setup.dart would be the right place to have a library build manually. Sorry that I cannot be of more help. 🤓

[MichealReed]

The examples here are very helpful
https://github.com/dart-lang/native/tree/084c10727ac1037cff7f12a32606d6ab88746977/pkgs/native_assets_cli/example

I see a path to a CMakeBuilder, will ping you when I get a draft commit up.

Still not understanding why Flutter web cannot/does not trigger the hook currently. The cli package is still useful there to download or build additional assets. The lack of CBuilder web support does not seem like a blocker for support on web. User created bindings will work if the hook outputs the accessible asset and the current approach of the library telling apps to add the loader script to the template html still seems viable with native assets. Same as the native_assets_test.dll flows to the runner folder, the native_assets_test.js, native_assets_test.wasm, and native_assets_test_loader.js could flow to be loaded as a script in the flutter index.html making the web library available for use.

[dcharkes]

Still not understanding why Flutter web cannot/does not trigger the hook currently.

It could, but since it doesn't consume any of the existing asset types it wouldn't do anything.

Most likely, when support for the data-asset types lands, Flutter for web will run build hooks flutter/flutter#164094 (cc @mosuem).

I see a path to a CMakeBuilder, will ping you when I get a draft commit up.

👍

[mosuem]

Most likely, when support for the data-asset types lands, Flutter for web will run build hooks

Yes, flutter/flutter#164094 will enable Flutter web running hooks

[MichealReed]

It seems @rainyl already has a working package published, great job!

https://pub.dev/packages/native_toolchain_cmake

Maybe this issue can now be closed and feedback/ideas can be opened there?

https://github.com/rainyl/native_toolchain_cmake

[rainyl]

It seems @rainyl already has a working package published, great job!

https://pub.dev/packages/native_toolchain_cmake

Maybe this issue can now be closed and feedback/ideas can be opened there?

https://github.com/rainyl/native_toolchain_cmake

aha, I am busy recently and didn't notice this issue.

Thanks to native_toolchain_c, native_toolchain_cmake was born from there, it now works but still under development and wasm is not supported yet. Welcome to talk there.