GPU is detected but Koboldcpp's compiling kernel fails.

[tncfdp]

Hello, 1st of all thanks for this great software which just works for most use cases. I bring an edge case in the hopes of helping extend the frontier further, namely, older GPUs.

in summary, koboldcpp detects the card AND the available memory correctly (with a bit of a delay though, after it already output "no gpu detected" which is curious) but when I click launch, the compilation fails claiming the "type half can only be used directly When the cl_khr_fp16 extension is enabled (unsupported)" or something similar to that (I'm paraphrasing)

A few potentially important details:

-- I tried all the "older cpus" options that shows the correct GPU (Pitcairn) below it, basically the openblast ones, it can even autoset the gpulayers after i select the model;

-- i tried --usecublas running from the cli directly, but that delay I mentioned doesn´t seem to allow enough time for detection when running this way, because it fails before getting to the compilation phase;

-- I'm running Archlinux, and those libraries are installed from the AUR repo, that is, manually installed and this was quite some time ago, however, I did update the rest of the system and, most importantly, Blender works (a version that is known to work with this GPU), and I can see the cycles renderer outputting to the terminal it's progress on (several) kernels compilations and running them to produce a (faster than cpu) render. Which leads me to believe that it's just a matter of the opencl version not supporting that "half type", either by the hardware or by my installation of the libraries. Also, I can also send my own kernels to compile and run, although I forget now how to do it, I can potentially help dev/debug koboldcpp.

Is there a distro more adequate, perhaps with the drivers pre-installed, to run koboldcpp on? which one is it developed in? Debian?

I'll be happy to provide any further details to help solve this, thanks.
TF

[LostRuins]

This looks like an error from clblast instead of cublas. Could you kindly share your terminal output? This will contain the launch parameters as well as any outputs or errors encountered. You can simply run it in terminal and copy/paste the output here.

[0cc4m]

Then it should be working.

What do you mean? What I meant to say is that I included the 2 available Vulkan options in with all the others that don't work 'cause they produce the same output "Illegal instruction"

What does vulkaninfo --summary show?

I don't have vulkaninfo installed, I do have clinfo though, do you want that? In fact, I don't think that GPU supports it, IIRC AMD was at the time transitioning from magma to vulkan (assuming im not confusing stuff, i don't see any mention in the box nor manual), I was still seeing marketing in regards to that.. anyways, clinfo does have some output that seem to settle this, and it relates to LR question:

I'm telling you Pitcairn is Sea Islands and as long as you use the amdgpu driver and not the old radeon, it will have full Vulkan support and can run llama.cpp-based applications like koboldcpp just fine, with Vulkan on Linux. I've run it myself in the past with an AMD FirePro S10000, which is a slightly larger Sea Islands chip than Pitcairn. If you don't have vulkaninfo, just install it.. (vulkan-tools on Arch)

What CPU do you have?

The CPU(?) on that machine is an old Athlon 64 3000+ :) Speaking of CPU flags though, clinfo has a field called "Device Extensions", with stuff that look A LOT like the "cl_khr_f16" error I get and, sure enough, that one isn´t there.

If we trust the name "Device" extensions, then it won't be fixed once I reinstall to a newer OS hoping to get a new opencl (and wherever clinfo comes from) version.

So unless you guys wanna take me by the hand to try and port this to this more limited opencl (oldestest cpu option? :)), let me know. As I searched for the error, it seems that "half" type was already there but only as a "storage" primitive... so maybe not that many changes.

Either way, I let you know if anything changes after installing a new OS. (btw, I did build and reinstall the new clblast version from AUR before this reply, but that didn't change anything for that error)

That is a very old CPU and means you need to build the program yourself natively, instead of using distributed executables. "Illegal instruction" means it tried to use something your CPU doesn't support. It should not happen if you just compile it by yourself.

[LostRuins]

Yup occam is right - don't build with LLAMA_PORTABLE flag and it will use native by default, which should work.

[tncfdp]

hi, some good and some bad news:

the good news is that vulkaninfo worked practically OOTB after I installed Debian (AMD seems to like .deb packages, though they release more specifically for ubuntu), I just had to add the kernel flags.
Another good news is that koboldcpp detects more than simply "pitcairn", the vulkan driver "RADV" and the particular card model too.
Even better news is that the original error is gone and now even clblast gives illegal instruction, going so far as beginning to load the model in vram! So, if occam is right, the compilation solution may work for both vulkan AND opencl.

Before I do that though, I'd like to consult with you guys about that RADV vulkan driver. Vulkaninfo says right at the top that "radv is not a conformant implementation and it's for testing only", also, archlinux https://wiki.archlinux.org/title/Vulkan makes it sound really easy to simply change that implementation though I could not find much instruction to do it in Debian.

For amdvlk, amd instructions sound simple enough https://github.com/GPUOpen-Drivers/AMDVLK?tab=readme-ov-file#runtime-settings and since their download page link directly to that part it gives the impression I could ignore the (more complicated) stuff above it (scroll up to take a look on those instructions, those apply only for someone building the thing, right?)

So I'll try this before anything else, doesn't look like it might break anything. The other implementation however, vulkan-amdgpu-pro , would probably be much more involved and replace the card drivers (instead of only the vulkan driver) and what not.

So the current plan is:

1. try to install amdvlk on Debian, see if anything changes;
2. Depending on your feedback, try the same with vulkan-amdgpu-pro, they seem to have made a script to both install and uninstall,so maybe not so bad?
3. try to change between them, as per the archwiki instructions, hopefully only need to change the prefix variable when running a program
4. go for compilation of koboldcpp.

Any ideas? Thanks.

[0cc4m]

Just use RADV, it is the only driver you can use and the best one available on Linux. Ignore the warning, it is only about being Vulkan-certified, for which some minor things may be missing, depending on the version, but which doesn't concern you at all.

[tncfdp]

well.. okay. May I ask why the other drivers cannot be used with this card?
Anyways, I'll accept your suggestion and skip directly to the compilation then.

To be able to follow LostRuins advice and disable the LLAMA_PORTABLE flag, it seems I'll have to follow the "manual Method" as the "conda method" doesn't seem to allow setting variables, correct?

apparently just run make LLAMA_CLBLAST=1 LLAMA_VULKAN=1 LLAMA_PORTABLE=0 after cloning the repo.
I'll be back..
after I get around to it :)

[0cc4m]

Sure, I'll try to summarize it:

radv is a driver developed by the open source community as part of the Mesa 3D library, which provides the default drivers for OpenGL, Vulkan, OpenCL, etc on Linux for AMD and Intel GPUs, and many more. They also provide Nvidia drivers, but due to the closed source nature of Nvidia, this is more of the struggle and those often don't perform well.
Since this is a community effort, financial interests are less of a factor and as a result there are people who maintain support for very old GPUs, like yours.

amdvlk is AMD's open source reference Vulkan driver. It is only developed by AMD and shares a lot of code with the Pro driver and the Windows driver. AFAIK it performs worse in games than RADV, so there is rarely a reason to use it. It may support very new hardware better while the Mesa developers catch up.
Support for old GPUs is dropped relatively quickly, I think currently it supports only RDNA GPUs. Polaris, Vega and other GCN generations (your GPU is a first generation GCN card) have been dropped and won't work with it.

vulkan-amdgpu-pro is AMD's closed source Linux driver, it mainly exists for enterprise customers that need a certified driver, there is no reason whatsoever to use it on a consumer machine, basically.

[tncfdp]

Thanks, sounds reasonable.

I compiled the thing! Though it didn't fix the issue, it confirmed the suspicion the cpu couldn't be the whole story, because the last option "Failsafe Mode (older cpu)" works flawlessly. And it can´t be (somehow) the model either (I'm using a variant of the one placed in the docker image) since it talks :)

As the other CPU options also give "Illegal instruction", there is hope the problem is indeed not the GPU (like you said except not at compile time but runtime), is there a way to run the same command used for the failsafe mode, except with a vulkan/clblast backend instead? What is it in that failsafe command that is different from the other CPU options, and can we remove the extras when calling those other backends?
Or, if you still think it's at compile time, what other flags like LLAMA_PORTABLE should be disabled?

[0cc4m]

You should only be compiling with LLAMA_VULKAN=1 LLAMA_CLBLAST=1, nothing else.