Skip to content

TFLITE: Benchmarking failure on GPT2 quantized autocomplete.tflite  #62506

Closed as not planned
@suyash-narain

Description

@suyash-narain

System information

  • Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)
    : No

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): aarch64 linux

  • TensorFlow installed from (source or binary): binary

  • TensorFlow version (use command below): tf 2.15

  • Python version: python3.10.9

  • Exact command to reproduce: linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10

Describe the problem

I am using an aarch64 device like raspberry pi. I created the gpt2 autocomplete.tflite model using the official colab tutorial https://colab.research.google.com/github/tensorflow/codelabs/blob/main/KerasNLP/io2023_workshop.ipynb#scrollTo=uLsz2IcN46eb

I was able to create both the quantized and unquantized tflite model. I then tried to benchmark these models on the aarch64 device using the official nightly tflite benchmark model linux_aarch64 using tf 2.15 sourced from https://www.tensorflow.org/lite/performance/measurement#native_benchmark_binary

On running the benchmark model using the command: linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10

I get benchmarking failure errors. I am running the model on CPU itself but it seems the ops are unsupported. I get same benchmarking failure on running the unquantized version of it as well. The logs are below.

root@user:~# ./linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Num threads: [4]
INFO: Graph: [autocomplete.tflite]
INFO: #threads used for CPU inference: [4]
INFO: Loaded model autocomplete.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: Select TensorFlow op(s), included in the given model, is(are) not support ed by this interpreter. Make sure you apply/link the Flex delegate before infere nce. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-li te-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/ guide/ops_select
ERROR: Node number 2 (FlexMutableHashTableV2) failed to prepare.
ERROR: Select TensorFlow op(s), included in the given model, is(are) not support ed by this interpreter. Make sure you apply/link the Flex delegate before infere nce. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-li te-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/ guide/ops_select
ERROR: Node number 2 (FlexMutableHashTableV2) failed to prepare.
ERROR: Failed to allocate tensors!
ERROR: Benchmarking failed.

How do I benchmark this model correctly? Since its a tflite model I should be able to benchmark it using tflite benchmark model.

On trying the model with flex delegate benchmark model, it doesn't seem like benchmarking is failing but it does give a lot of error messages along with the INFO last line giving some avg values and exits soon after. the logs are below:

root@user:~# linux_aarch64_benchmark_model_plus_flex --graph=autocomplete.tflite --num_threads=1
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Num threads: [1]
INFO: Graph: [autocomplete.tflite]
INFO: #threads used for CPU inference: [1]
INFO: Loaded model autocomplete.tflite
INFO: Created TensorFlow Lite delegate for select TF ops.
INFO: TfLiteFlexDelegate delegate: 29 nodes delegated out of 1139 nodes with 14 partitions.

ERROR: Op type not registered 'RegexSplitWithOffsets' in binary running on device. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib (e.g. tf.contrib.resampler), accessing should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
ERROR: Op type not registered 'RegexSplitWithOffsets' in binary running on device. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib (e.g. tf.contrib.resampler), accessing should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
WARNING: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#243 is a dynamic-sized tensor).
INFO: The input model file size (MB): 129.674
INFO: Initialized session in 131.484ms.
INFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized

ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
INFO: count=113 first=216495 curr=60356 min=22 max=216495 avg=4851.66 std=26505

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions