Unsupported Scalar Type 5? -- Portable/optimized ops don't consistently support half/bfloat16

[bluejack]

🐛 Describe the bug

After exporting a model to pte form and running it through executor_runner, I get:

E 00:00:02.220756 executorch:inputs_portable.cpp:45] Unsupported scalar type 5

I believe this is the "Half" type, or float16

Does that simply mean executor_runner does not support float16? Or does the whole framework not support float16?

Noted that when I run some investigation on the file using a python script, I get as far as sending it my float16 tensors, but it still fails to execute with a similar error:

[op_native_layer_norm.cpp:169] In function operator()(), assert failed (false): Unhandled dtype Half for native_layer_norm.out

I'm including the versions below, but note that this is using executorch built from head, rather than the last release. Should I expect the framework to support float16? And look to my own code for the error?

Versions

PyTorch version: 2.6.0.dev20250104
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.6.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.4)
CMake version: version 3.31.4
Libc version: N/A

Python version: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 08:22:19) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.6.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Pro

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+cd0e584
[pip3] flake8==7.0.0
[pip3] mypy==1.11.2
[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.0.0
[pip3] numpydoc==1.7.0
[pip3] torch==2.6.0.dev20250104
[pip3] torchao==0.8.0+git2e032c6b
[pip3] torchaudio==2.6.0.dev20250104
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20250104
[conda] executorch 0.6.0a0+cd0e584 pypi_0 pypi
[conda] numpy 2.0.0 pypi_0 pypi
[conda] numpydoc 1.7.0 py312hca03da5_0
[conda] torch 2.6.0.dev20250104 pypi_0 pypi
[conda] torchao 0.8.0+git2e032c6b pypi_0 pypi
[conda] torchaudio 2.6.0.dev20250104 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250104 pypi_0 pypi

cc @larryliu0820 @manuelcandales

[swolchok]

Does that simply mean executor_runner does not support float16?

It looks like this particular function does not support float16. I've just sent #7750 to fix it.

Or does the whole framework not support float16?

We are capable of supporting it, but it looks like portable ops coverage is spotty. I'll send a fix for native_layer_norm and as many other places as I can find.

[swolchok]

By the way, if you're running on your Mac, you might want to enable the XNNPACK delegate when exporting; there's a good chance you will get both better performance and a workaround for the remaining instance of this issue I haven't got PRs out for yet (though I don't know whether XNNPACK has layer norm off the top of my head).

[bluejack]

By the way, if you're running on your Mac, you might want to enable the XNNPACK delegate when exporting; there's a good chance you will get both better performance and a workaround for the remaining instance of this issue I haven't got PRs out for yet (though I don't know whether XNNPACK has layer norm off the top of my head).

Ok, I will look at this option, thanks for the tip.

[bluejack]

To enable the XNNPACK delegate on export is it anything more than this:

edge_program = to_edge_transform_and_lower(
        exported_text,
        partitioner=[XnnpackPartitioner()],
        compile_config=EdgeCompileConfig(
            _check_ir_validity=True,
            _skip_dim_order=True,
        ),
    )

I replaced my basic to_edge call with this one, which I pulled from the xnnpack examples... but it does not seem to make any difference. Not sure if that's an indication that I am not actually doing the delegation properly, or whether it genuinely doesn't make a difference.

[kimishpatel]

I think you will probably want to apply recipe similar to llama stuff here?
Like for quantization it has to first do this https://github.com/pytorch/executorch/blob/main/examples/models/llama/export_llama_lib.py#L1041 and only 4bit quant option (8da4w works right now)
And then do XNNPACK "lowering" following code similar to https://github.com/pytorch/executorch/blob/main/examples/models/llama/export_llama_lib.py#L685.

SO in the above there are really two steps

1. 4bit quant that requires some code from https://github.com/pytorch/executorch/blob/main/examples/models/llama/source_transformation/quantize.py

And "lowering" that then identifies appropriate portions of the graph and leverages XNNPACK to execute them. That is the second link.

If you run into issues, which I expect you may, please post here and if you can paste the appropriate graph snippets/model from each stage it would help. @mcr229 and @digantdesai know a ton on this

[kimishpatel]

Also note that the code pointers above have largely been validated with llama3+ models, so if your model is similar that should likely enable using those utils.

Another thing worth mentioning is that, for language models 4-bit optimization works the best for now and is well supported which I linked above. But if thats not the case for your model and some variant of 8-bit quantization works better than I am gonna ask @mcr229 to point you to some code snippets to enable quantization and lowering for you.

[bluejack]

Thanks for these tips. Ours is a vision model, so eventual quality might suggest more than 4 bits, but at the moment we are just trying to get a proof of concept going. I'll dig in to these resources, thanks.

[swolchok]

For anyone else who wants to jump in on op support, here is how I'm identifying ops to look at:

$ cd kernels/portable/cpu
$  rg --files-without-match 'Half|HALF|ALL|HBF16|HBBF16|ufunc'  -g '*.cpp' -g !test | sort

Still have to spot-check, but this gives an initial list.

[swolchok]

closing because most things should support half/bfloat16 now. (norm ops are the exception per #7846; hoping to get to code sharing with PyTorch and then solve accuracy issues that way)