Even better Q4_0 KV cache (CPU)

[ikawrakow]

Cont #1547, which was CUDA only.

This PR does the same for the CPU back-end. Once at it, I also simdified Q4_0 quantization on AVX2 for a minor performance benefit when using Q4_0 KV cache.

[ubergarm]

If anyone is quantizing Kimi-K2.5 and new Kimi-K2.6 from bf16 GGUF to the modified Q4_X format in an attempt to best match their llm-compressor settings, it takes two lines to patch now. Thanks @jukofyork for the heads up before I missed this important detail!

$ git diff
diff --git a/ggml/src/ggml-quants.c b/ggml/src/ggml-quants.c
index 6f9007e0..883b892b 100644
--- a/ggml/src/ggml-quants.c
+++ b/ggml/src/ggml-quants.c
@@ -689,7 +689,7 @@ void quantize_row_q4_0_ref(const float * restrict x, block_q4_0 * restrict y, in
             }
         }

-        const float d  = max / -8;
+        const float d  = max / -7;
         const float id = d ? 1.0f/d : 0.0f;

         y[i].d = GGML_FP32_TO_FP16(d);
@@ -708,8 +708,8 @@ void quantize_row_q4_0_ref(const float * restrict x, block_q4_0 * restrict y, in
 }

 void quantize_row_q4_0(const float * restrict x, void * restrict y, int64_t k) {
-    iqk_quantize_q4_0(x, y, k);
-    //quantize_row_q4_0_ref(x, y, k);
+    //iqk_quantize_q4_0(x, y, k);
+    quantize_row_q4_0_ref(x, y, k);
 }

This is for running llama-quantize only, and not needed on clients after the GGUFs have already been made.