Severe prompt processing slowdown on CUDA backend after NVIDIA driver 610.62 (RTX 5070 Ti, llama.cpp backend)

[Frusciante79]

LM Studio: 0.4.17 (Build 3)
operating system: Windows 11

After updating to NVIDIA driver 610.62, I observe a significant regression in prompt processing performance in LM Studio using the CUDA backend (llama.cpp).

The model executes correctly and GPU is properly detected. There is no CPU fallback or crash. However, prompt processing (prefill phase) is significantly slower compared to previous behavior.

Generation speed remains relatively stable after prompt processing completes, indicating the issue is isolated to the prefill/prompt processing stage rather than full inference throughput.

A key observation is that GPU utilization remains active, but efficiency during prompt processing appears degraded, suggesting a possible regression in CUDA kernel execution or KV cache handling.

Steps to reproduce the behavior:

Open LM Studio
Load model: nex-agi_Nex-N2-mini-Q4_K_M.gguf
Select CUDA or CUDA12 backend (llama.cpp)
Use a long context prompt (4K–8K+ tokens)
Start generation
Observe prompt processing timing and token throughput
Expected behavior

Consistent prompt processing performance similar to prior behavior (~29–30 tokens/s stable), without degradation over increasing context size.

Actual behavior
Prompt processing starts relatively fast (~150 tokens/s initial estimate)
Performance degrades progressively during longer prompts (~154 → 147 → 134 tokens/s)
Noticeable slowdown compared to previous expected baseline (~29–30 tok/s stable in prior conditions)
GPU remains active throughout execution
No CPU fallback observed
Screenshots

Logs
slot prompt_clear: clearing prompt with 0 tokens
slot process_sing: saving idle slot to prompt cache
slot print_timing: prompt processing, n_tokens = 2048 → 4096 → 6144
performance trend: 154 → 147 → 134 tokens/s (degrading)

Additional context
GPU: NVIDIA RTX 5070 Ti
Driver: 610.62
Backend: CUDA (llama.cpp)
Same model shows significantly better prompt processing behavior prior to driver update
Issue is reproducible across multiple runs and sessions
Hypothesis

Possible regression in NVIDIA 610.62 affecting CUDA execution behavior in llama.cpp backend, specifically:

KV cache reuse efficiency
CUDA memory allocation patterns
kernel scheduling / prefill optimization path
potential loss of CUDA graph optimizations during prompt processing
Request

Could you please confirm whether recent NVIDIA driver changes (610.x branch) may impact prompt processing performance in CUDA backend (llama.cpp), and whether any LM Studio configuration changes are required for optimal performance?

[will-lms]

Hello @Frusciante79 . Thank you for the report!

I am not fully following. Your report is that the baseline is "~29–30 tok/s" but then you report "~150 tokens/s" with the new version. Is prompt processing really 5x faster? That sounds inconsistent with the rest of your report.

Can you please double check those numbers? While you're at it, I'd appreciate if you could try something for me

1. Go Settings > Developer and disable "Use LM Studio Engine Protocol"
2. Load the model and note both the Prompt Processing TPS and the Token Generation TPS at varying context lengths.
3. Go back to Settings > Developer and enable "Use LM Studio Engine Protocol"
4. Load the same model and run the same tests as step 2.

This will help me understand if a bug in the LM Studio Engine Protocol specifically is contributing to the difference in behavior you're reporting.

Thank you!

[Frusciante79]

Hello @will-lms ,

Thank you for your help and for asking me to re-check the measurements.

I re-ran the tests using the same model and input and I now have consistent data for both configurations (Engine Protocol ON/OFF). I can confirm that there is no meaningful performance regression between the two modes:

Prompt processing is stable (~2.6k tokens/s range depending on context size)
Token generation is stable (~67–68 tokens/s)
Differences between ON/OFF appear negligible and within normal variance

So I agree that my initial comparison was incorrect due to mixing different metrics and non-comparable prompt sizes.

However, I would like to report a separate issue that I am now consistently observing after recent updates:

Some models that previously worked without issues are now noticeably slower to load or behave inconsistently during prompt execution. This is not related to throughput alone, but also to stability during initialization and execution.

In particular, I tested the model:

google/gemma-4-26b-a4b-qat

and observed the following behavior.

I am attaching the log from a request using this model:

2026-06-19 23:46:09 [DEBUG]
0.00.042.332 I srv init: using 31 threads for HTTP server
0.00.042.340 I srv init: The UI is disabled
0.00.042.341 I srv init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
0.00.042.423 I srv start: binding port with default address family
2026-06-19 23:46:09 [DEBUG]
0.00.045.482 I srv llama_server: loading model
0.00.045.496 I srv load_model: loading model 'D:\AI\LMStudio_Models\lmstudio-community\gemma-4-26B-A4B-it-QAT-GGUF\gemma-4-26B-A4B-it-QAT-Q4_0.gguf'
2026-06-19 23:46:11 [DEBUG]
0.02.039.949 I srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 1286.72 MiB
0.02.039.964 I common_init_result: fitting params to device memory ...
0.02.039.965 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
2026-06-19 23:46:15 [DEBUG]
0.05.981.354 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 23, abort
2026-06-19 23:46:16 [DEBUG]
0.07.307.506 W load: control-looking token: 50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
2026-06-19 23:46:16 [DEBUG]
0.07.307.887 W load: control-looking token: 212 '' was not control-type; this is probably a bug in the model. its type will be overridden
2026-06-19 23:46:16 [DEBUG]
0.07.356.527 W load: special_eog_ids contains '<|tool_response>', removing '' token from EOG list
2026-06-19 23:46:24 [DEBUG]
0.14.639.826 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
2026-06-19 23:46:29 [DEBUG]
0.19.409.806 I srv load_model: loaded multimodal model, 'D:/AI/LMStudio_Models/lmstudio-community/gemma-4-26B-A4B-it-QAT-GGUF/mmproj-gemma-4-26B-A4B-it-QAT-BF16.gguf'
0.19.409.814 I srv load_model: initializing slots, n_slots = 4
2026-06-19 23:46:29 [DEBUG]
0.19.558.906 W common_speculative_init: no implementations specified for speculative decoding
2026-06-19 23:46:29 [DEBUG]
0.19.559.059 I slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
0.19.559.065 I slot load_model: id 1 | task -1 | new slot, n_ctx = 262144
0.19.559.065 I slot load_model: id 2 | task -1 | new slot, n_ctx = 262144
0.19.559.066 I slot load_model: id 3 | task -1 | new slot, n_ctx = 262144
0.19.559.122 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
0.19.559.124 I srv load_model: use --cache-ram 0 to disable the prompt cache
0.19.559.127 I srv load_model: for more info see ggml-org/llama.cpp#16391
0.19.559.128 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
2026-06-19 23:46:29 [DEBUG]
0.19.559.500 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
2026-06-19 23:46:29 [DEBUG]
0.19.566.068 I init: chat template, example_format: '<|turn>system
<|think|>
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
'
2026-06-19 23:46:29 [DEBUG]
0.19.566.922 I srv init: init: chat template, thinking = 1
2026-06-19 23:46:29 [DEBUG]
0.19.567.483 I srv llama_server: model loaded
0.19.567.488 I srv llama_server: server is listening on http://127.0.0.1:58099
2026-06-19 23:46:29 [DEBUG]
0.19.567.768 I srv update_slots: all slots are idle
2026-06-19 23:46:38 [DEBUG]
0.28.750.489 I srv params_from_: Chat format: peg-gemma4
2026-06-19 23:46:38 [DEBUG]
0.28.751.857 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.28.751.860 I srv get_availabl: updating prompt cache
2026-06-19 23:46:38 [DEBUG]
0.28.752.126 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.28.752.130 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
0.28.752.131 I srv get_availabl: prompt cache update took 0.27 ms
0.28.752.156 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.28.752.156 I slot process_sing: id 0 | task -1 | saving idle slot to prompt cache
0.28.752.157 I slot prompt_clear: id 0 | task -1 | clearing prompt with 0 tokens
0.28.752.287 I slot process_sing: id 1 | task -1 | saving idle slot to prompt cache
0.28.752.290 I slot prompt_clear: id 1 | task -1 | clearing prompt with 0 tokens
0.28.752.412 I slot process_sing: id 2 | task -1 | saving idle slot to prompt cache
0.28.752.414 I slot prompt_clear: id 2 | task -1 | clearing prompt with 0 tokens

This behavior includes:

slower or inconsistent model loading
temporary stalls during prompt processing
variation in behavior across different models (some stable, others degraded)

This started appearing after recent updates and may be related to CUDA driver/runtime changes or LM Studio backend behavior.

I am currently using CUDA runtime v. 2.22.0

Thanks