Is CUDA Toolkit a Prerequisite?

[ievgeniilarin]

I can't make LlamaSharp work with my GPU. Is CUDA Toolkit a prerequisite? It's not mentioned anywhere.

[hswlab]

I'm using LLamasharp with CPU on a very old notebook with simple AVX support^^'. It is very slow ~ 1Token/sec but it works fine. LLamasharp is able to select an installed backend automatically, but you can also select what ever backend you like with the following call. It need to be called before you are using any other LLamasharp method.

Example:
NativeLibraryConfig.All
.WithAvx(LLama.Native.AvxLevel.Avx)
.WithCuda(false)
.SkipCheck(true)
.WithAutoFallback(false)

You also need to install the same version of the backend as the version of llamasharp you are using. If you want to use only CPU, then you don't need to install the cuda backend, just the CPU backend.

My notebook is able to use cuda8, but currently you can only download cuda 11 or 12 as a nuget package for Llamasharp.
I also tested LLamasharp on a CPU with AVX2 support, it is much faster. I can live with that speed, but if I could I would use a GPU with cuda support :)

[iejeecee]

I am using it with Vulkan which uses my GPU