Description
System information
-
Have I written custom code (as opposed to using a stock example script
provided in TensorFlow): No -
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): aarch64 linux
-
TensorFlow installed from (source or binary): binary
-
TensorFlow version (use command below): tf 2.15
-
Python version: python3.10.9
-
Exact command to reproduce:
linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10
Describe the problem
I am using an aarch64 device like raspberry pi. I created the gpt2 autocomplete.tflite model using the official colab tutorial https://colab.research.google.com/github/tensorflow/codelabs/blob/main/KerasNLP/io2023_workshop.ipynb#scrollTo=uLsz2IcN46eb
I was able to create both the quantized and unquantized tflite model. I then tried to benchmark these models on the aarch64 device using the official nightly tflite benchmark model linux_aarch64 using tf 2.15 sourced from https://www.tensorflow.org/lite/performance/measurement#native_benchmark_binary
On running the benchmark model using the command: linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10
I get benchmarking failure errors. I am running the model on CPU itself but it seems the ops are unsupported. I get same benchmarking failure on running the unquantized version of it as well. The logs are below.
root@user:~# ./linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Num threads: [4]
INFO: Graph: [autocomplete.tflite]
INFO: #threads used for CPU inference: [4]
INFO: Loaded model autocomplete.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: Select TensorFlow op(s), included in the given model, is(are) not support ed by this interpreter. Make sure you apply/link the Flex delegate before infere nce. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-li te-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/ guide/ops_select
ERROR: Node number 2 (FlexMutableHashTableV2) failed to prepare.
ERROR: Select TensorFlow op(s), included in the given model, is(are) not support ed by this interpreter. Make sure you apply/link the Flex delegate before infere nce. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-li te-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/ guide/ops_select
ERROR: Node number 2 (FlexMutableHashTableV2) failed to prepare.
ERROR: Failed to allocate tensors!
ERROR: Benchmarking failed.
How do I benchmark this model correctly? Since its a tflite model I should be able to benchmark it using tflite benchmark model.
On trying the model with flex delegate benchmark model, it doesn't seem like benchmarking is failing but it does give a lot of error messages along with the INFO last line giving some avg values and exits soon after. the logs are below:
root@user:~# linux_aarch64_benchmark_model_plus_flex --graph=autocomplete.tflite --num_threads=1
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Num threads: [1]
INFO: Graph: [autocomplete.tflite]
INFO: #threads used for CPU inference: [1]
INFO: Loaded model autocomplete.tflite
INFO: Created TensorFlow Lite delegate for select TF ops.
INFO: TfLiteFlexDelegate delegate: 29 nodes delegated out of 1139 nodes with 14 partitions.ERROR: Op type not registered 'RegexSplitWithOffsets' in binary running on device. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib (e.g.
tf.contrib.resampler
), accessing should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
ERROR: Op type not registered 'RegexSplitWithOffsets' in binary running on device. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib (e.g.tf.contrib.resampler
), accessing should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
WARNING: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#243 is a dynamic-sized tensor).
INFO: The input model file size (MB): 129.674
INFO: Initialized session in 131.484ms.
INFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
INFO: count=113 first=216495 curr=60356 min=22 max=216495 avg=4851.66 std=26505