Skip to content

Eval bug: Gibberish output when K/V cache use identical quantization types (q4_0/q4_0 or q8_0/q8_0) on RTX 5060 Ti (Blackwell) #23717

Description

@gulf1324

Name and Version

version: 9333 (35c9b1f)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

Intel 12400F + RTX 5060Ti 16GB

Models

Qwen3.6-27B-UD-Q4_K_XL.gguf
Source: https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/tree/main

Problem description & steps to reproduce

Command line

Example:

llama-server.exe ^
  -m models\Qwen3.6-27B-UD-Q4_K_XL.gguf ^
  --ctx-size 32768 ^
  --parallel 1 ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --jinja ^
  --host 127.0.0.1 ^
  --port 8080 ^
  --temp 0.6 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --presence-penalty 1.0 ^
  --reasoning off ^
  --cache-ram 0

Observed behavior

The model initially generates coherent text, but after roughly 100-150 generated tokens it consistently degrades into gibberish / token soup.

Example prompt 1:

Write a Python function to reverse a string.

Example output 1:

def main_module_example_rebind_to_context_freeze_use_main_question_order_3_buggy_in_fact_oa_watrdut_numberic_table_qreserdoc_hack_threading_list_demo_request_code_sign_question_numbe...

Example prompt 2:

write an essay about python

Example output 2:

# The Python Paradigm: A Developer’s Journey Through Syntax and Mindset

In the world of software development...

...X.Cupperborndreolinkesunocakemovergizolderdecompressorotsumber...

Reproduction matrix

Works correctly

K cache V cache
bf16 bf16
q8_0 bf16
bf16 q8_0
q4_0 bf16
bf16 q4_0
q8_0 q4_0
q4_0 q8_0
q5_0 q5_0

Produces gibberish

K cache V cache
q4_0 q4_0
q8_0 q8_0

First Bad Commit

No response

Relevant log output

Logs
# q4_0 / q4_0, nothing seems wrong on CLI side

0.09.840.807 I srv          init: init: chat template, thinking = 0
0.09.840.858 I srv  llama_server: model loaded
0.09.840.861 I srv  llama_server: server is listening on http://127.0.0.1:8080
0.09.840.868 I srv  update_slots: all slots are idle
0.18.862.653 I srv  params_from_: Chat format: peg-native
0.18.862.816 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.18.862.863 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.19.622.951 I slot create_check: id  0 | task 0 | created context checkpoint 1 of 32 (pos_min = 12, pos_max = 12, n_tokens = 13, size = 149.626 MiB)
0.34.560.597 I slot print_timing: id  0 | task 0 | n_decoded =    100, tg =   6.81 t/s
0.37.566.326 I slot print_timing: id  0 | task 0 | n_decoded =    120, tg =   6.78 t/s
0.40.687.946 I slot print_timing: id  0 | task 0 | n_decoded =    141, tg =   6.78 t/s
0.43.707.438 I slot print_timing: id  0 | task 0 | n_decoded =    161, tg =   6.76 t/s
0.45.808.080 W srv          stop: cancel task, id_task = 0
0.45.963.426 I slot      release: id  0 | task 0 | stop processing: n_tokens = 192, truncated = 0
0.45.963.439 I srv  update_slots: all slots are idle

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions