Use `jnigen` and `ffigen` to call Java/Kotlin/Swift/ObjC natively without the `MethodChannels`

[marandaneto]

Description

https://dart.dev/guides/libraries/java-interop and https://dart.dev/guides/libraries/objective-c-interop
Depends on Dart 3, still experimental.
Can probably use gen packages (ffigen) for Linux, Windows, etc.
Need to figure out how to call Web without MethodChannels

[ueman]

For web you don't need any method channels or similar since Dart has JS interopt built-in: https://dart.dev/web/js-interop

[vaind]

1. We may need to do this to enable profiling. Starting a profiler via a method channel is an async operation (even though the actual profiler startup call in objective-c/swift is synchronous). However, Hub.startTransaction() is synchronous so we cannot wait for the profiler to be actually started before returning from this function.
2. I don't see why this would be limited to Dart 3, considering it seems to use standard FFI which has been around forever. Do you remember the source for that SDK version limitation @marandaneto?

[ueman]

ffigen for Swift was introduced in Dart 2.18 which shipped with Flutter 3.3, see https://medium.com/flutter/announcing-flutter-3-3-at-flutter-vikings-6f213e068793

Then there's dart:ffi (which is part of the Dart SDK), and package:ffi (distributed via pub). ffigen makes use of a specific version of package:ffi. ffigen is also the tool where most of the Swift binding magic and its improvments happens.

In addition to that, Swift ffi is considered experimental, however cupertino_http (a package using those Swfit/objc bindings) is considered stable.

[vaind]

Looking at the wording in the Dart 2.18 announcement, it looks like it's just a documentation issue (and the availability of a generator), not that it didn't work with an older version of the SDK. The changelog for 2.18 doesn't mention anything (or I'm blind) so that would indicate my assumption may be true.
I may be wrong but I think we wouldn't even need to bump the minimum Dart version in pubspec. Anyway, I'll likely give this a quick try, considering the need for profiling I've mentioned above.

(FYI, I am pretty familiar with FFI: https://github.com/objectbox/objectbox-dart/graphs/contributors - the whole thing is based on a C(++) library)

[marandaneto]

@vaind I read somewhere about min Dart 3 but I don't recall where exactly, it could be a docs issue so go ahead and try it out.

[vaind]

I've evaluated this (in the experiment PR #1622) so let me summarize the findings.

• both Cocoa SDK and JNI are usable in the versions available at the time of writing (see pubspec.yaml)
• both generators (ffigen & jnigen) seem to be actively developed in areas important to us (objective-c & java/android support)
• generated interfaces look pretty usable (not very cumbersome to use)
• we should see a smaller call overhead with FFI compared to MethodChannels - I don't have any benchmarks at hand, just based on my understanding of MethodChannels and how they work (e.g. that they require data serialization for all invocations).
• we would be able to call native functions without async, further simplifying the API (and improving performance).
• as is, the generated code doesn't require raising the SDK dependency to Dart 3, although the just released (an hour ago) version of jnigen has just raised this from 2.18 to 3.1 for no obvious reason. I think it's safe to say they'd try to address new SDK so it may not be a great fit for us. For cocoa SDK, the generator (ffigen) is more mature and trying more to keep the compatibility with older SDKs so even though the generated code has some incompatible code, it's currently limited to adding the final specifier to classes which can be easily replaced locally after code generation ran.

Risks

• JNI generation is fairly new so there may be more risks involved (the generated code may be more buggy)
• the previous point applies to some extent to ffigen too because even though that's quite a mature package, the new code addressing objective-c specifically hasn't been out for so long
• on the other hand - it's possible to just stick to the generator version that "works for us" and ignore new versions - even have a custom fork with cherry-picked changes if we really wanted to. For example to keep compatibility with older Dart SDK version (2.*). However, that comes with its own drawbacks so we may want to wait out till we raise the SDK dependency bar.

Next steps

• switching could be done either separately for Cocoa and for JNI, with some changes how those are currently being initialized (distinguish in the NativeSdkIntegration class), or do them both together, saving some code complexity of not having to maintain method-channel integrations at all
• there should be a decision on whether, when and to what extent we want to make use of FFI
• we can even do an incremental approach - implement only the new code via FFI

Notes on code generation updates

Since this involves code generation, updating the SDK would have an additional step (this should be part of upgrade script that's triggered by the updater, e.g. flutter/scripts/update-cocoa.sh ): dart run ffigen --config ffi-cocoa.yaml (or the alternative for JNI).