QNN: wav2letter doesn't export

[guangy10]

🐛 Describe the bug

python -m examples.qualcomm.scripts.wav2letter -b cmake-out -m SM8450 --compile_only -p ./states_fused.pth

fails to export. See stacktrace:

QNN_SDK_ROOT=/home/guangyang/fbsource/third-party/qualcomm/qnn/qnn-2.26
Traceback (most recent call last):
  File "/data/users/guangyang/executorch/examples/qualcomm/scripts/wav2letter.py", line 220, in <module>
    main(args)
  File "/data/users/guangyang/executorch/examples/qualcomm/scripts/wav2letter.py", line 131, in main
    inputs, targets, input_list = get_dataset(
  File "/data/users/guangyang/executorch/examples/qualcomm/scripts/wav2letter.py", line 69, in get_dataset
    for wave, label in data_loader:
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 708, in __next__
    data = self._next_data()
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 764, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torchaudio/datasets/librispeech.py", line 170, in __getitem__
    waveform = _load_waveform(self._archive, metadata[0], metadata[1])
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torchaudio/datasets/utils.py", line 51, in _load_waveform
    waveform, sample_rate = torchaudio.load(path)
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torchaudio/_backend/utils.py", line 204, in load
    backend = dispatcher(uri, format, backend)
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/site-packages/torchaudio/_backend/utils.py", line 116, in dispatcher
    raise RuntimeError(f"Couldn't find appropriate backend to handle uri {uri} and format {format}.")
RuntimeError: Couldn't find appropriate backend to handle uri ./wav2letter/LibriSpeech/test-clean/2094/142345/2094-142345-0042.flac and format None.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/guangyang/.conda/envs/executorch/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/users/guangyang/executorch/examples/qualcomm/scripts/wav2letter.py", line 226, in <module>
    raise Exception(e)
Exception: Couldn't find appropriate backend to handle uri ./wav2letter/LibriSpeech/test-clean/2094/142345/2094-142345-0042.flac and format None.

where states_fused.pth is downloaded from https://github.com/nipponjo/wav2letter-ctc-pytorch/tree/main?tab=readme-ov-file#wav2letter-ctc-pytorch

cc: @cccclai @cbilgin

Versions

latest main

cc @cccclai @winskuo-quic @shewu-quic

[cccclai]

Is there any error message?

[guangy10]

Updated with stacktrace. cc: @cccclai

[winskuo-quic]

Hi @guangy10,
I am trying to reproduce this error, but everything is working fine on my side.
I am currently using torchaudio 2.6.0.dev20250131+cpu.

[guangy10]

@winskuo-quic If you think the model is working fine, should we just enable it in the CI, the setup there can be the source of truth for future reference. Here is the QNN models we are currently running on CI: https://github.com/pytorch/executorch/blob/main/.github/workflows/trunk.yml#L305-L329

Can you add wav2letter to it? The CI only need to test it in --compile_only mode, to ensure there is no complication issue

[winskuo-quic]

@guangy10 Could you please share where I can place my model's weight file? I can't use wget to retrieve the weights, so I think I might need a location to place the weights where CI can also have access to.

[guangy10]

@guangy10 Could you please share where I can place my model's weight file? I can't use wget to retrieve the weights, so I think I might need a location to place the weights where CI can also have access to.

Can we use random weights instead, for compile_only, since the model is not calibrated nor evaluating the accuracy. In the CI the goal is to just ensure it's functioning end-to-end with low latency. Users can always rerun these models locally by disabling the compile_only so that they will get the high quality model easily. wdyt @winskuo-quic

[cccclai]

random weights

Random weights possibly have a different latency number than the actual weights. If it will be used on dashboard, it's probably better to use real weights

[guangy10]

random weights

Random weights possibly have a different latency number than the actual weights. If it will be used on dashboard, it's probably better to use real weights

@cccclai Curious how different the numbers are—can you share them? Or are you assuming a hypothetical scenario?

If the weights are public, one solution is to cache the weights in the docker image for qnn. It can avoid redownloading the weights again and again. And we already have that docker.

[cccclai]

@cccclai Curious how different the numbers are—can you share them? Or are you assuming a hypothetical scenario?

I've observed a 10-15% perf difference for a model we plan to ship to production. It took me a while to debug the perf gap and it turns out that the difference is random weight/real weight

[winskuo-quic]

@guangy10, @cccclai,
For wav2letter, we are using public weights, however, for mobilebert, we are training our own weights in the script.
For mobilebert, I would like to know if we can save our own trained weights in the docker?
Also, could you share how I can save the weights in the docker image? Thanks

[cccclai]

how I can save the weights in the docker image

cc @kirklandsign and @huydhn for this question

[kirklandsign]

Save weight in s3 maybe?

Docker is really for env set up right?