[tfjs-tflite] Can't initialize model, Encountered unresolved custom op

[mattrossman]

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 12.4 ARM
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
• TensorFlow.js installed from (npm or script link): both
• TensorFlow.js version (use command below): 3.18.0
• Browser version: Version 103.0.5060.53 (Official Build) (arm64)
• Tensorflow.js Converter Version: N/A

Describe the current behavior

Loading MediaPipe's face_landmark_with_attention.tflite fails with:

Error: Failed to create TFLiteWebModelRunner: INVALID_ARGUMENT: Can't initialize model

Several warnings appear mentioning an unresolved custom op Landmarks2TransformMatrix and a link to this guide for registering custom operators via C++.

It's unclear how this operator should be registered for this JS library as there is no C++ in this package's source files. The tflite_web_api_cc file generating the warning is mentioned in wasm/BUILD.bazel, and it appears to be downloaded pre-built rather than built from source.

Describe the expected behavior

• The model should successfully load
• OR some mechanism for automatically loading custom ops should be provided
• OR instructions for building tflite_web_api_cc with custom ops should be provided

Standalone code to reproduce the issue

Here is a JSFiddle that uses the README sample to load these two face landmarks TFLite models (regular, with attention)

https://jsfiddle.net/mattrossman/o0pbqgyr/3/

The regular model loads successfully, while the model with attention fails to load.

Other info / logs

[AleCaste]

We are getting these errors when loading many other tflite models.
If we compile tflite ourselves and produce our own custom WASM file, then we can load all tflite files without problems.
But we would like to use the pre-built WASM files used by TFLite Web.
Any ideas about how to overcome this problem?

[shmishra99]

Hi @mattrossman ,
Apologies for the delay in my response. I was able to replicate the issue you reported using the latest version of @tensorflow/tfjs@4.4.0, and I encountered the same error.
I have included the code below :

const mediaPipeFaceLandmarkTfLiteUrl = "https://rawcdn.githack.com/google/mediapipe/c688862570156908cd95303afb06beb0935e5cc6/mediapipe/modules/face_landmark/face_landmark.tflite"
const mediaPipeFaceLandmarkAttentionTfLiteUrl = "https://rawcdn.githack.com/google/mediapipe/c688862570156908cd95303afb06beb0935e5cc6/mediapipe/modules/face_landmark/face_landmark_with_attention.tflite"


tflite.loadTFLiteModel(mediaPipeFaceLandmarkTfLiteUrl)
	.then((tfliteModel) => console.log('Loaded regular face landmarks model:', tfliteModel))
  .catch((err) => console.error('Failed to load regular face landmarks model:', err))

tflite.loadTFLiteModel(mediaPipeFaceLandmarkAttentionTfLiteUrl)
	.then((tfliteModel) => console.log('Loaded face landmarks model with attention:', tfliteModel))
  .catch((err) => console.error('Failed to load face landmarks model with attention:', err))

Error :

Thank you for bringing this issue to our attention. We will investigate it further and get back to you.
CC: @jinjingforever