Eval bug: gemma-4-31B-it-UD-IQ3_XXS.gguf (unsloth) produces gibberish on blackwell (RTX 5080 tested) with GGML_CUDA=ON compilation VS normal correct output with BLAS/CPU compilations

[xyehya]

Name and Version

llama.cpp version: b8645 (57ace0d61) (+ fixed tokenzier for gemma-4)

Operating systems

Linux

GGML backends

CUDA

Hardware

Core Ultra 9 285K + RTX 5080

Models

gemma-4-31B-it-UD-IQ3_XXS.gguf (unsloth)

Problem description & steps to reproduce

Title

Gemma 4 UD-IQ3_XXS gives gibberish on RTX 5080 with default CUDA build, but works on CPU and with FORCE_CUBLAS flag when compiling

Environment

• llama.cpp version: b8645, commit 57ace0d
• Gemma 4 tokenizer fix applied locally from PR fix: Gemma4 tokenizer #21343 / commit b069b10
• OS: Fedora Linux 43
• Compiler: GNU 15.2.1
• CUDA toolkit: 13.2.51
• NVIDIA driver: 595.58.03
• GPU: NVIDIA GeForce RTX 5080, 16 GB
• Model: unsloth/gemma-4-31B-it-GGUF
• GGUF: gemma-4-31B-it-UD-IQ3_XXS.gguf

Summary

After applying the Gemma 4 tokenizer fix, this model still produces gibberish on GPU with a normal CUDA build on RTX 5080.

This does not seem to be a tokenizer problem anymore.

What I checked:

• tokenizer.ggml.model is gemma4
• runtime reports vocab type = BPE
• merges are present
• basic tokenization looks correct

The problem looks tied to the CUDA execution path for this quant on Blackwell.

built flags with gibberish output

This is the build that gives bad output:

• GGML_CUDA=ON
• CMAKE_CUDA_ARCHITECTURES=120a
• GGML_NATIVE=ON
• GGML_LTO=ON
• GGML_CUDA_GRAPHS=ON
• GGML_CUDA_FA_ALL_QUANTS=ON

Reproducer

./build/bin/llama-cli \
  -m /path/to/gemma-4-31B-it-UD-IQ3_XXS.gguf \
  --jinja \
  -p "Hello" \
  -n 12 \
  --temp 0.0 --top-p 1.0 --top-k 0 \
  --ctx-size 512 \
  -fa off \
  --no-mmap

Example bad output:

B laeL la laHH la la single lasH- deL抹茶-H laist// la

I can also reproduce the same behavior through llama-server with chat completions.

The model returns gibberish on GPU with the default CUDA build above.

What makes this look like a CUDA path issue

1. CPU only works

Running the same model on CPU only gives coherent output:

CUDA_VISIBLE_DEVICES="" ./build/bin/llama-cli \
  -m /path/to/gemma-4-31B-it-UD-IQ3_XXS.gguf \
  --jinja  \
  -p "Hello" \
  -n 12 \
  --temp 0.0 --top-p 1.0 --top-k 0 \
  --ctx-size 512 \
  --no-mmap

Example output:

[Start thinking]

"Hello"
Greeting.
Friendly,

2. A conservative CUDA build also works

I built a second CUDA variant with:

• GGML_CUDA=ON
• CMAKE_CUDA_ARCHITECTURES=120a
• GGML_CUDA_FORCE_CUBLAS=ON
• GGML_NATIVE=OFF
• GGML_LTO=OFF
• GGML_CUDA_GRAPHS=OFF
• GGML_CUDA_FA_ALL_QUANTS=OFF

Then I ran:

./build-cublas/bin/llama-cli \
  -m /path/to/gemma-4-31B-it-UD-IQ3_XXS.gguf \
  --jinja  \
  -p "Hello" \
  -n 12 \
  --temp 0.0 --top-p 1.0 --top-k 0 \
  --ctx-size 512 \
  -fa off \
  --no-mmap \
  --fit on

This produced coherent output on the same RTX 5080.

What is ruled out

• Not stale binaries. I did a full clean rebuild of all binaries, not just llama-server.
• Not chat UI formatting only. I can reproduce it in llama-cli.
• Not sampling noise as it still happen with greedy decoding.
• Not Flash Attention alone as it still happens with -fa off.
• Not missing Gemma 4 tokenizer fix. That fix is present and the tokenizer behavior matches it.

Workaround

The practical workaround for me is to use the cuBLAS flag in CUDA build:

-DGGML_CUDA_FORCE_CUBLAS=ON

also disabled:

-DGGML_NATIVE=OFF
-DGGML_LTO=OFF
-DGGML_CUDA_GRAPHS=OFF
-DGGML_CUDA_FA_ALL_QUANTS=OFF

probable root cause

I think the default CUDA kernel path for this IQ quant is wrong on Blackwell, and forcing cuBLAS avoids the bad path.

• same GGUF fails on the default GPU build
• same GGUF works on CPU
• same GGUF works on a conservative GPU build with FORCE_CUBLAS

So at the moment this looks more like a CUDA backend or kernel selection problem than a bad GGUF.

About re-quantizing or regenerating the imatrix

Im nt sure if regenerating the imatrix with the latest release could fix this issue.

First Bad Commit

No response

Relevant log output

./build/bin/llama-cli
-m /home/yk/Data/lmstudio/models/unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-IQ3_XXS.gguf
--jinja -st
-p "Hello"
-n 24
--temp 0.7 --top-p 0.95 --top-k 64
--ctx-size 4096
-fa on
--no-mmap

Generated text:
B la singularHL deHLH singular ideSing de personP

Failing GPU run with greedy decoding and Flash Attention disabled:

Command:
./build/bin/llama-cli
-m /home/yk/Data/lmstudio/models/unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-IQ3_XXS.gguf
--jinja -st
-p "Hello"
-n 12
--temp 0.0 --top-p 1.0 --top-k 0
--ctx-size 512
-fa off
--no-mmap

Generated text:
B laeL la laHH la la single lasH- deL抹茶-H laist// la

Working CPU-only run from the same patched tree:

Command:
CUDA_VISIBLE_DEVICES="" ./build/bin/llama-cli
-m /home/yk/Data/lmstudio/models/unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-IQ3_XXS.gguf
--jinja -st
-p "Hello"
-n 12
--temp 0.0 --top-p 1.0 --top-k 0
--ctx-size 512
--no-mmap

Generated text:
[Start thinking]

"Hello"
Greeting.
Friendly,

Working GPU run with conservative CUDA build:

Command:
./build-cublas/bin/llama-cli
-m /home/yk/Data/lmstudio/models/unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-IQ3_XXS.gguf
--jinja -st
-p "Hello"
-n 12
--temp 0.0 --top-p 1.0 --top-k 0
--ctx-size 512
-fa off
--no-mmap
--fit on

Generated text:
[Start thinking]

"Hello"
Greeting.
Friendly,

Working llama-server smoke test with conservative CUDA build:

Command:
./build-cublas/bin/llama-server
-m /home/yk/Data/lmstudio/models/unsloth/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-IQ3_XXS.gguf
--ctx-size 4096
--alias gemma-4-31b-it
--parallel 1
--temp 0.0
--top-p 1.0
--top-k 0
-fa off
--host 127.0.0.1
--port 8890
--no-mmap
--fit on
--jinja

Request:
curl -s http://127.0.0.1:8890/v1/chat/completions
-H "Content-Type: application/json"
-d '{"model":"gemma-4-31b-it","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Response:
{"choices":[{"finish_reason":"length","index":0,"message":
{"role":"assistant","content":"","reasoning_content":"\n"Hello"\nGreeting.\nFriendly, helpful,
and open.\n\n * Acknowledge the greeting.\n * Offer"}}], ...}

[xyehya]

@danielhanchen

[theo77186]

Possibly related issue: #21255
The quantized gguf are probably fine.

[xyehya]

@theo77186 @danielhanchen The quants are 100% fine. The issue is with CUDA 13.2 - I recompiled latest b8648 on a cuda12.8 container running on the same machine and tested it and it works flawlessly.

CUDA issue confirmed 100%

[danielhanchen]

Hey - let me check as well - it seems like CUDA 13.2 is causing these issues maybe? Let me repor

[xyehya]

@danielhanchen yes, I used your pre-compiled latest cuda linux llama.cpp binaries from your fork and it worked perfectly.

[danielhanchen]

@xyehya Oh ye I think we used CUDA 13.0

[anskumar01]

Next public release of Cuda (13.3) should have the fix for this issue

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.