M1 support

[OpenSource03]

Hello there!

While using the app, I experienced the following issue on my M1 Max (Mac Studio).

sudo whisper test.mp4 --language Serbian --model medium --device mps
loc("mps_multiply"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/b594a083-3050-11ed-abe0-ca9c01dc79de/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":221:0)): error: input types 'tensor<1x1024x3000xf16>' and 'tensor<1xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
zsh: abort      sudo whisper test.mp4 --language Serbian --model medium --device mps

The app seems to be using exclusively CPU, not even 1% of GPU. While both PyTorch and Python are fully optimised for M1, I don't see any reason for this app to stay behind.

Hope to see some news soon!

Regards

[tstm]

I did get it to (partially) run. I installed the nightly build of torch which has the best mps support.

And then run it with CPU fallback and fp16 disabled

$ PYTORCH_ENABLE_MPS_FALLBACK=1 whisper sample.mp3 --model small.en --device mps --fp16 False

But it would still crash after some time - but with some correct output first - with the following error:

RuntimeError: dst_.nbytes() >= dst_byte_offset INTERNAL ASSERT FAILED at "/Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/operations/Copy.mm":200, please report a bug to PyTorch.

[silvacarl2]

i am sure this will work: https://github.com/ggerganov/whisper.cpp

[nclr]

This did the trick, Thank You

[LazaroGranado]

did you get this to work with m1 gpu by chance? or cpu?

[oldcai]

It's by CPU currently, but it's much faster.

[eddiedunn]

The smaller models seem to work pretty well on my M1 mac mini with CPU. In general I hope someone comes along soon and gives some Apple Silicon GPU love to torch.

[dickwu]

I used the last version of PyTorch, but the problem is still

[liorkesos]

The problem still persists even after I installed a conda environment with nightly pytorch as described here https://towardsdatascience.com/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1

[mayeaux]

I get the same complaint:

The operator 'aten::repeat_interleave.self_int' is not currently supported on the MPS backend

You can see it's still uncompleted on the to-do list: pytorch/pytorch#77764 (comment)

[kklem0]

pytorch/pytorch@fe0c7fb

It's merged.

[mayeaux]

@kklem0 Has anyone tested with the latest pytorch ? I wonder if it spits out a new error or where it gets stuck.

[Bernhard-Steindl]

I tried openai-whisper in version 20230314 together with torch version 2.0.0 and --device mps. But the error is:

NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'SparseMPS' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build).

[nclr]

I confirm this problem persists.

[slhck]

There's a C++ version that runs natively under macOS with Apple Silicon: https://github.com/ggerganov/whisper.cpp

[LazaroGranado]

can this also be used with m1 gpu or only cpu?

[slhck]

This runs on the CPU but it's heavily optimized for the M1 chip.

[rben01]

You can compile a version that runs on the GPU on M1 devices. See instructions: https://github.com/ggerganov/whisper.cpp?tab=readme-ov-file#core-ml-support

[0x2447196]

I removed the two .to_sparse() calls in model.py and it works on M1 (setting pytorch backend to MPS), and I can see the GPU is being used. However, it is very slow. Is it just because sparse operations here are faster?

[rben01]

I would assume sparse operations save a lot of memory. Densifying a matrix will likely cause memory to balloon, which causes poor performance for a number of reasons.

[changeling]

Same experience as @0x2447196. Removing the couple of .to_sparse() calls in model.py returns whisper to working, but with a drastically reduced speed.

[changeling]

Has there been any progress on this issue? It's long, long standing.

[tstm]

There's a port that does M1, and is much, much faster. https://github.com/ggerganov/whisper.cpp

No real need to get this fixed at this point, as it's also not being optimised as such.

[mutonby]

some news to gpu use in m1? thx

[zphj1987]

mark

[cszmzh]

i have the same problem when using mps:
NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'SparseMPS' backend.

[RazeBerry]

Same here. What is the workaround if there is any?

[Hyprnx]

here for any kinds of update

[marcmp]

The so called MPS ( Metal Performance Shaders ) are there to be used by pytorch. But, as far as I can tell, nobody has been able to use them within this project.

Bare in mind that running Whisper locally goes against OpenAI interests, therefore I would not expect any time soon to see support for Apple Silicon GPU by any of the commiters of the project. In a fork someday? maybe...

In the meantime, I encourage you to try whisper.cpp it works like a charm in Apple Silicon using the GPU as a first class citizen.

[NebulusIO]

I would also suggest checking out https://github.com/ml-explore/mlx