TFLITE: Benchmarking failure on GPT2 quantized autocomplete.tflite

[suyash-narain]

System information

• Have I written custom code (as opposed to using a stock example script
  provided in TensorFlow): No

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): aarch64 linux

• TensorFlow installed from (source or binary): binary

• TensorFlow version (use command below): tf 2.15

• Python version: python3.10.9

• Exact command to reproduce: linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10

Describe the problem

I am using an aarch64 device like raspberry pi. I created the gpt2 autocomplete.tflite model using the official colab tutorial https://colab.research.google.com/github/tensorflow/codelabs/blob/main/KerasNLP/io2023_workshop.ipynb#scrollTo=uLsz2IcN46eb

I was able to create both the quantized and unquantized tflite model. I then tried to benchmark these models on the aarch64 device using the official nightly tflite benchmark model linux_aarch64 using tf 2.15 sourced from https://www.tensorflow.org/lite/performance/measurement#native_benchmark_binary

On running the benchmark model using the command: linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10

I get benchmarking failure errors. I am running the model on CPU itself but it seems the ops are unsupported. I get same benchmarking failure on running the unquantized version of it as well. The logs are below.

root@user:~# ./linux_aarch64_benchmark_model --graph=autocomplete.tflite --num_threads=1 --num_runs=10
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Num threads: [4]
INFO: Graph: [autocomplete.tflite]
INFO: #threads used for CPU inference: [4]
INFO: Loaded model autocomplete.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: Select TensorFlow op(s), included in the given model, is(are) not support ed by this interpreter. Make sure you apply/link the Flex delegate before infere nce. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-li te-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/ guide/ops_select
ERROR: Node number 2 (FlexMutableHashTableV2) failed to prepare.
ERROR: Select TensorFlow op(s), included in the given model, is(are) not support ed by this interpreter. Make sure you apply/link the Flex delegate before infere nce. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-li te-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/ guide/ops_select
ERROR: Node number 2 (FlexMutableHashTableV2) failed to prepare.
ERROR: Failed to allocate tensors!
ERROR: Benchmarking failed.

How do I benchmark this model correctly? Since its a tflite model I should be able to benchmark it using tflite benchmark model.

On trying the model with flex delegate benchmark model, it doesn't seem like benchmarking is failing but it does give a lot of error messages along with the INFO last line giving some avg values and exits soon after. the logs are below:

root@user:~# linux_aarch64_benchmark_model_plus_flex --graph=autocomplete.tflite --num_threads=1
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Num threads: [1]
INFO: Graph: [autocomplete.tflite]
INFO: #threads used for CPU inference: [1]
INFO: Loaded model autocomplete.tflite
INFO: Created TensorFlow Lite delegate for select TF ops.
INFO: TfLiteFlexDelegate delegate: 29 nodes delegated out of 1139 nodes with 14 partitions.

ERROR: Op type not registered 'RegexSplitWithOffsets' in binary running on device. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib (e.g. tf.contrib.resampler), accessing should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
ERROR: Op type not registered 'RegexSplitWithOffsets' in binary running on device. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib (e.g. tf.contrib.resampler), accessing should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
WARNING: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#243 is a dynamic-sized tensor).
INFO: The input model file size (MB): 129.674
INFO: Initialized session in 131.484ms.
INFO: Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized

ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
ERROR: Delegate kernel was not initialized
ERROR: Node number 1139 (TfLiteFlexDelegate) failed to prepare.
INFO: count=113 first=216495 curr=60356 min=22 max=216495 avg=4851.66 std=26505

[sushreebarsa]

@suyash-narain Could you please make sure you have included the necessary libraries for the "RegexSplitWithOffsets" op, such as the TFLite Text library and try to rebuild the runtime with appropriate flag enabled?
Also try to disable XNNPACK, If you don't need the XNNPACK delegate for performance reasons. Thank you!

[suyash-narain]

Hi @sushreebarsa, RegexSplitWithOffsets op error comes into picture when I use benchmark model with flex delegate. Is using flex delegate a necessity with autocomplete.tflite model? Why is general tflite nightly benchmark model not able to execute this tflite model?
Do i need to enable tflite task library? i thought thats enabled by default when we build runtime?

does the prebuilt benchmark model from tflite not have support for tensorflow text libraries?

which flag do i need to enable while building tflite runtime for tflite text library as i cannot find any references on either tensorflow github or tflite documentation.

[sushreebarsa]

@suyash-narain Using the Flex delegate is not strictly necessary with the autocomplete.tflite model, but it is highly recommended for two primary reasons such as for unsupported operations and high performance. Due to some missing dependencies or incompatible versions. General TFLite nightly benchmark model not able to execute the tflite model due to missing dependencies.

You don't need to explicitly enable the TensorFlow Lite Task Library in most cases. It is automatically included in the TFLite runtime and utilizes the same inference infrastructure as the standard TFLite API.

Unfortunately, the prebuilt benchmark model from TFLite currently does not have native support for TensorFlow Text libraries.

Thank you!

[suyash-narain]

Hi @sushreebarsa, thanks for your reply.
I still don't understand. Building tflite runtime by default should have the Task Library enabled in most cases. But tflite benchmark model provided as default (even the one with flex delegate) cannot benchmark autocomplete.tflite gpt2 model.
How do I benchmark it then?
if i build my benchmark model, which flags do i need to enable, if you can let me know? or would building default tflite benchmark model will work for tf2.15?