`
Summary
Running GLM-5.2 (glm-dsa, unsloth GLM-5.2-GGUF UD-IQ2_M) in CPU+GPU hybrid mode (--cpu-moe, routed experts in host RAM, non-expert layers on a 20 GB RTX A4500), any partial offload of the non-expert layers (-ngl below n_layer) causes generation to degenerate into token salad once the conversation/context grows past a depth threshold:
- build
f96eadd (2026-06-26): corruption appears at roughly 9–13K tokens of context - build
bbc7de4 (HEAD, 2026-07-03): coherent noticeably deeper (a real multi-turn agentic session stayed clean through ~17K) but corrupts by ~20K (observed at n_past ≈ 20,430)
Full non-expert offload (-ngl 99 --cpu-moe) is clean on the same prompts/workload to at least ~16K on both builds (agentic sessions completed correctly end-to-end).
The corruption is nondeterministic per request — a garbled generation can be followed by a coherent one in the same conversation (with the garbage still in context). When the garbage contains invalid UTF-8, /completion///v1/chat/completions returns HTTP 500 after generating; the server does not crash.
Bisection (each cell = a real multi-turn agentic workload, 10.6K-token system prompt growing
to ~16K; PASS = task completed with coherent output end-to-end, ❌ = output degenerated into token salad mid-session)
Config (build f96eadd) | Result
-- | --
-ngl 99 --cpu-moe, -ub 512, f16 KV | ✅ clean
-ngl 99 --cpu-moe, -ub 512, q4_1 KV | ✅ clean
-ngl 75 --cpu-moe, -ub 512, q4_1 KV | ❌ garbage
-ngl 75 --cpu-moe, -ub 4096, q4_1 KV | ❌ garbage
-ngl 70 --cpu-moe, -ub 6144, q4_1 KV | ❌ garbage
Failing command line (example)
GGML_CUDA_NO_PINNED=1 GGML_CUDA_DISABLE_GRAPHS=1 ./build/bin/llama-server \
-m GLM-5.2-UD-IQ2_M-00001-of-00006.gguf \
-c 32768 -t 64 -b 2048 -ub 512 \
-ngl 75 --cpu-moe -ctk q4_1 -ctv q4_1 -amb 512 \
--temp 1.0 --top-p 0.95 --min-p 0.01 --jinja \
--host 0.0.0.0 --port 8080
(Same command with -ngl 99 is clean. GGML_CUDA_DISABLE_GRAPHS=1 was set in all tests on both builds, for an unrelated reason — see note at the end. -fa/-mla at defaults; --dsa not used on HEAD.)
Reproduction
Any multi-turn /v1/chat/completions conversation whose context exceeds the threshold reproduces it within a few turns. What we used:
- A large varied system prompt (several thousand tokens of distinct runbook-style sentences — not one repeated sentence) + 5 user turns with short factual/code questions,
chat_template_kwargs: {"enable_thinking": false}, temp 1.0. - Or, most reliably: an agentic coding harness (OpenCode) session — its ~10.6K-token system prompt puts every turn past the f96eadd threshold immediately; output degenerates after ~3 coherent tool turns.
Symptom example (mid-conversation, previously coherent):
holtSoside asm confirm
d92eside
'tside
eff80form
"form4fa,ike.fa,4form,.4---0,f-,.
Environment
- Threadripper PRO 3995WX (64c/128t, Zen2/AVX2), 256 GB DDR4-3200 (8-ch), RTX A4500 20 GB
- Driver 580.173.02 / CUDA 13.0 toolkit,
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_NATIVE=ON - Debian 12 (privileged LXC on Proxmox, kernel 7.0.2-6-pve)
- Model:
unsloth/GLM-5.2-GGUF UD-IQ2_M (glm-dsa 744B.A40B, 2.7 bpw, ~220 GiB, 6 shards)
Note: unrelated CUDA-graph cache growth (why GGML_CUDA_DISABLE_GRAPHS=1 was set)
While investigating we also found that the CUDA-graph cache (ggml_backend_cuda_context::cuda_graphs) is keyed by the raw pointer of cgraph->nodes[0] and is only ever cleared in the model-reload path (src/llama-reload.cpp), so long-running servers accumulate captured graphs (VRAM) as request shapes vary until an allocator OOM wedges the server. Happy to file that separately if useful — the corruption above reproduces with graphs disabled, so the two issues are independent.
`
`
Summary
Running GLM-5.2 (
glm-dsa, unslothGLM-5.2-GGUFUD-IQ2_M) in CPU+GPU hybrid mode (--cpu-moe, routed experts in host RAM, non-expert layers on a 20 GB RTX A4500), any partial offload of the non-expert layers (-nglbelow n_layer) causes generation to degenerate into token salad once the conversation/context grows past a depth threshold:f96eadd(2026-06-26): corruption appears at roughly 9–13K tokens of contextbbc7de4(HEAD, 2026-07-03): coherent noticeably deeper (a real multi-turn agentic session stayed clean through ~17K) but corrupts by ~20K (observed at n_past ≈ 20,430)Full non-expert offload (
-ngl 99 --cpu-moe) is clean on the same prompts/workload to at least ~16K on both builds (agentic sessions completed correctly end-to-end).The corruption is nondeterministic per request — a garbled generation can be followed by a coherent one in the same conversation (with the garbage still in context). When the garbage contains invalid UTF-8,
/completion///v1/chat/completionsreturns HTTP 500 after generating; the server does not crash.Bisection (each cell = a real multi-turn agentic workload, 10.6K-token system prompt growing
to ~16K; PASS = task completed with coherent output end-to-end, ❌ = output degenerated into token salad mid-session)
Config (build f96eadd) | Result -- | -- -ngl 99 --cpu-moe, -ub 512, f16 KV | ✅ clean -ngl 99 --cpu-moe, -ub 512, q4_1 KV | ✅ clean -ngl 75 --cpu-moe, -ub 512, q4_1 KV | ❌ garbage -ngl 75 --cpu-moe, -ub 4096, q4_1 KV | ❌ garbage -ngl 70 --cpu-moe, -ub 6144, q4_1 KV | ❌ garbageFailing command line (example)
(Same command with
-ngl 99is clean.GGML_CUDA_DISABLE_GRAPHS=1was set in all tests on both builds, for an unrelated reason — see note at the end.-fa/-mlaat defaults;--dsanot used on HEAD.)Reproduction
Any multi-turn
/v1/chat/completionsconversation whose context exceeds the threshold reproduces it within a few turns. What we used:chat_template_kwargs: {"enable_thinking": false}, temp 1.0.Symptom example (mid-conversation, previously coherent):
Environment
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_NATIVE=ONunsloth/GLM-5.2-GGUFUD-IQ2_M(glm-dsa 744B.A40B, 2.7 bpw, ~220 GiB, 6 shards)Note: unrelated CUDA-graph cache growth (why GGML_CUDA_DISABLE_GRAPHS=1 was set)
While investigating we also found that the CUDA-graph cache (
`ggml_backend_cuda_context::cuda_graphs) is keyed by the raw pointer ofcgraph->nodes[0]and is only ever cleared in the model-reload path (src/llama-reload.cpp), so long-running servers accumulate captured graphs (VRAM) as request shapes vary until an allocator OOM wedges the server. Happy to file that separately if useful — the corruption above reproduces with graphs disabled, so the two issues are independent.