CRITICAL: found TWO root causes for PPL=165 TheTom#30

1. V cache returns rotated-space values (cosine=0.02 vs correct 0.987)
2. dynamic_cast to llama_kv_cache_context fails for MoE models
   (uses llama_memory_hybrid_context, not kv_cache_context)
   → Q rotation and V inverse rotation NEVER executed

Fix: store rotation tensors in llm_graph_context, not KV cache.
Or access through hybrid memory interface.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: 2 people

docs/quality-benchmarks.md

@@ -138,3 +138,72 @@ The block size 32 change + MSE-only + pre-rotate-queries may have introduced a b
 4. The non-vec flash attention instantiation might use wrong nl parameter
 
 MUST FIX before claiming any quality results.
+
+## Quality Bisection Log
+
+Goal: find which change broke perplexity (165.6 vs 6.1 baseline).
+
+Changes applied (in order):
+1. Pre-rotate-queries: rotation moved from dequant to Q in attention graph
+2. MSE-only: dropped QJL, 3-bit centroids (2-bit qs + 1-bit signs)
+3. Block size 32: storage blocks shrunk from 128 to 32 elements
+
+Bisection plan:
+- [ ] Test A: revert block 32 → 128, keep MSE-only + pre-rotate
+- [ ] Test B: revert MSE-only → 2bit+QJL, keep block 128 + pre-rotate
+- [ ] Test C: revert pre-rotate → rotation in dequant, keep block 128 + 2bit+QJL
+- [ ] If A/B/C all still bad: check the rotation matrix consistency
+
+## Bisection Results
+
+### Test A: Disable Q rotation → PPL 165.6 (STILL BAD)
+Pre-rotate-queries is NOT the problem. Q rotation has no effect on perplexity.
+
+### Root Cause Found: V CACHE IN ROTATED SPACE
+
+The bug: both K and V caches go through the same turbo3 quantize (which rotates)
+and dequant (which returns rotated values without inverse rotation).
+
+For K cache: this is correct because Q is also rotated → <R*q, R*k> = <q, k> ✓
+For V cache: this is WRONG because the attention output = attn_weights @ V
+needs to be in ORIGINAL space, not rotated space.
+
+The fix: V cache must NOT be rotated during quantize, or the attention output
+must be inverse-rotated after the V multiplication.
+
+Options:
+1. Skip rotation for V in quantize — only rotate K
+2. Add inverse rotation after attention output: out = R^T @ (attn @ R*V) = attn @ V ✓
+3. Use f16 for V cache (only compress K) — mixed types need support
+
+Option 1 is simplest: in the SET_ROWS kernel, check if this is K or V and
+skip rotation for V. The dequant already skips rotation.
+Problem: the SET_ROWS kernel doesn't know if it's writing K or V.
+
+Option 2 is cleanest: add R^T multiplication to attention output in the graph.
+This is one ggml_mul_mat per layer (same as Q rotation). Cost is negligible.
+
+### Root Cause #1: V cache dequant returns rotated-space values
+Python verified: cosine(input, dequant_output) = 0.02 (garbage)
+After inverse rotation: cosine = 0.987 (correct)
+
+### Root Cause #2: dynamic_cast to llama_kv_cache_context FAILS
+The Qwen 3.5 MoE uses llama_memory_hybrid_context (not llama_kv_cache_context).
+The dynamic_cast returns null, so the V inverse rotation never executes.
+This also means the Q pre-rotation never executed — explaining why the
+chat server seemed to work (no rotation applied = raw quantize/dequant, which
+happens to produce plausible-looking text at low context but garbage perplexity).
+
+### Fix needed
+1. Store turbo rotation tensors in llm_graph_context (not KV cache)
+   OR access them through the hybrid memory interface
+2. Apply V inverse rotation through a non-cast path
+3. Verify Q rotation also executes (it was also using the same cast)
+
+### The "coherent text" mystery explained
+The model produced grammatical text because:
+- Without Q rotation, attention scores are computed in wrong space but still produce
+  some attention pattern (just not the right one)
+- Without V inverse rotation, the output is in rotated space which is orthogonal to
+  the correct space — but each layer compounds the error
+- Short conversations look plausible but perplexity reveals the content is wrong

src/llama-kv-cache.cpp

@@ -224,11 +224,13 @@ llama_kv_cache::llama_kv_cache(
 
         layers.push_back({ il, k, v, k_stream, v_stream, });
 
-        // TurboQuant: create rotation matrix tensor (once, shared across layers)
+        // TurboQuant: create rotation matrix tensors (once, shared across layers)
         if (turbo_rotation == nullptr &&
             (type_k == GGML_TYPE_TURBO3_0 || type_k == GGML_TYPE_TURBO4_0)) {
             turbo_rotation = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 128, 128);
-            ggml_format_name(turbo_rotation, "turbo_rotation");
+            ggml_format_name(turbo_rotation, "turbo_rotation");  // R^T
+            turbo_rotation_inv = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 128, 128);
+            ggml_format_name(turbo_rotation_inv, "turbo_rotation_inv");  // R
         }
     }
 
@@ -275,11 +277,12 @@ llama_kv_cache::llama_kv_cache(
 
         ggml_backend_buffer_clear(buf, 0);
 
-        // Fill turbo rotation matrix AFTER buffer clear (clear zeroes everything)
+        // Fill turbo rotation matrices AFTER buffer clear (clear zeroes everything)
         if (turbo_rotation != nullptr && turbo_rotation->buffer != nullptr && !model.hparams.no_alloc) {
             #include "turbo-rotation-data.h"
-            ggml_backend_tensor_set(turbo_rotation, TURBO_ROTATION_MATRIX, 0, 128 * 128 * sizeof(float));
-            LLAMA_LOG_INFO("%s: TurboQuant rotation matrix initialized (128x128)\n", __func__);
+            ggml_backend_tensor_set(turbo_rotation, TURBO_ROTATION_RT, 0, 128 * 128 * sizeof(float));
+            ggml_backend_tensor_set(turbo_rotation_inv, TURBO_ROTATION_R, 0, 128 * 128 * sizeof(float));
+            LLAMA_LOG_INFO("%s: TurboQuant rotation matrices initialized (128x128)\n", __func__);
         }
         ctxs_bufs.emplace_back(std::move(ctx), buf);
     }
@@ -2469,6 +2472,10 @@ ggml_tensor * llama_kv_cache_context::get_turbo_rotation() const {
     return kv->get_turbo_rotation();
 }
 
+ggml_tensor * llama_kv_cache_context::get_turbo_rotation_inv() const {
+    return kv->get_turbo_rotation_inv();
+}
+
 ggml_tensor * llama_kv_cache_context::cpy_k(ggml_context * ctx, ggml_tensor * k_cur, ggml_tensor * k_idxs, int32_t il) const {
     return kv->cpy_k(ctx, k_cur, k_idxs, il, sinfos[i_cur]);
 }

src/llama-kv-cache.h

@@ -165,8 +165,11 @@ class llama_kv_cache : public llama_memory_i {
     ggml_tensor * get_k(ggml_context * ctx, int32_t il, uint32_t n_kv, const slot_info & sinfo) const;
     ggml_tensor * get_v(ggml_context * ctx, int32_t il, uint32_t n_kv, const slot_info & sinfo) const;
 
-    // TurboQuant: get rotation matrix for pre-rotate-queries
+    // TurboQuant: get rotation matrices
+    // turbo_rotation = R^T (for Q forward rotation via ggml_mul_mat)
+    // turbo_rotation_inv = R (for output inverse rotation via ggml_mul_mat)
     ggml_tensor * get_turbo_rotation() const { return turbo_rotation; }
+    ggml_tensor * get_turbo_rotation_inv() const { return turbo_rotation_inv; }
 
     // store k_cur and v_cur in the cache based on the provided head location
     ggml_tensor * cpy_k(ggml_context * ctx, ggml_tensor * k_cur, ggml_tensor * k_idxs, int32_t il, const slot_info & sinfo) const;
@@ -273,10 +276,9 @@ class llama_kv_cache : public llama_memory_i {
 
     std::vector<kv_layer> layers;
 
-    // TurboQuant pre-rotate-queries: rotation matrix tensor (128×128)
-    // Created when type_k is TURBO3_0 or TURBO4_0
-    // Used in build_attn_mha to rotate Q before flash attention
-    ggml_tensor * turbo_rotation = nullptr;
+    // TurboQuant rotation matrices (128×128)
+    ggml_tensor * turbo_rotation = nullptr;      // R^T (for Q forward rotation)
+    ggml_tensor * turbo_rotation_inv = nullptr;   // R (for output inverse rotation)
 
     // model layer id -> KV cache layer id
     std::unordered_map<int32_t, int32_t> map_layer_ids;
@@ -365,8 +367,9 @@ class llama_kv_cache_context : public llama_memory_context_i {
     ggml_tensor * get_k(ggml_context * ctx, int32_t il) const;
     ggml_tensor * get_v(ggml_context * ctx, int32_t il) const;
 
-    // TurboQuant: get rotation matrix for pre-rotate-queries optimization
+    // TurboQuant rotation accessors
     ggml_tensor * get_turbo_rotation() const;
+    ggml_tensor * get_turbo_rotation_inv() const;
 
     // store k_cur and v_cur in the cache based on the provided head location
     // note: the heads in k_cur and v_cur should be laid out contiguously in memory