feat(kv): activation rotation + packed 4-bit KV cache (#61)

[geisten]

Summary

Adds activation rotation and a packed 4-bit KV cache to the transformer KV path, following llama.cpp#21038 / QuaRot. Closes #61.

Two new opt-in, per-session KV features (INT8 stays the default — nothing changes unless you ask):

• GEIST_KV_ROT=1 — Hadamard (Fast Walsh–Hadamard) rotation of Q/K/V before quantization, output rotated back. Suppresses activation outliers so low-bit KV quantizes better. Backend-agnostic, O(n·log n), self-inverse (one transform serves forward + backward), dot-preserving (QK scores unchanged).
• GEIST_KV_INT4=1 (new GEIST_KV_INT4 mode) — packed symmetric 4-bit KV cache: two values/byte, half the INT8 data footprint, per-token per-head scale. Rides the INT8 storage path (same buffer slots allocated half-size, same scales + ctx wiring) with a packing append branch and an unpacking attention kernel. Rotation is default-on for this mode (INT4-without-rotation is a quality cliff and the rotation is ~free / a net win at long context); GEIST_KV_ROT=0 opts out. Plain INT8 keeps rotation opt-in.

Why (measured, not asserted)

The whole change is data-driven; every claim below is from bench_kv_quality (extended here with a KL-divergence vs FP32 metric) and bench_session_throughput on a quiesced Pi 5 (Cortex-A76, mean-of-5).

Quality — mean KL(P_fp32 ‖ P_mode), nats, BitNet-2b4t (trustworthy absolute scale; Gemma inflated by un-applied logit softcap but same ordering):

mode	KL
INT8	0.00197
INT4	0.01275
INT4+ROT	0.00861
INT2	1.635
INT2+ROT	0.915
KIVI (2-bit asym)	0.262

• Rotation's benefit scales inversely with bit-width: ~nil at 8-bit (matches the PR's q8_0 row), recovers the 4-bit penalty to near-INT8, recovers most of the 2-bit collapse.
• INT4+ROT is near-lossless and ~30× closer to FP32 than KIVI — but at 4 bits, not 2. Symmetric 2-bit + ROT does not beat asymmetric KIVI; KIVI stays the right choice at 2-bit.

Cost (A76 decode, BitNet, ms/tok, min-of-3) — depends on context length. At short context decode is weight-bandwidth-bound so the halved KV reads are invisible and only the unpack shows; as KV length grows it comes to dominate decode bandwidth and INT4 crosses over to faster than INT8:

kv_len	INT8	INT4	INT4+ROT	INT4 vs INT8
~256	65.96	66.57	67.41	+0.9%
~1024	101.84	100.14	101.30	−1.7%
~2048	150.45	144.52	145.83	−3.9%
~3072	202.91	187.92	190.15	−7.4%

The crossover sits between 256 and 1024 tokens; by ~3072 packed INT4 is 7.4% faster than INT8 and even INT4+ROT (rotation included) is 6.3% faster.

So: in the regime packed INT4 is for — long chats, RAG, agents — it delivers half the KV footprint, near-lossless quality, AND faster decode at once; INT4+ROT is the sweet spot. Attractive on the 4 GB Pi. These are different operating points from KIVI, not competitors → kept as an explicit mode (INT8 default), no per-environment auto-arbitration.

What's in here

File	Change
fwht.{c,h}	orthonormal FWHT (+ unit test)
int4_kv.h	4-bit nibble pack/unpack (+ unit test, sign-extension)
kv_store.c	rotation in append + attention; packed-INT4 append branch
attention.c	attention_int4_via_buffers unpacking kernel
arch_state.{c,h}	mode resolution, GEIST_KV_ROT / GEIST_KV_INT4 flags, half-size buffer alloc
geist.h	GEIST_KV_INT4 public mode
bench_kv_quality.c	full mode sweep + KL-divergence metric
ARCHITECTURE.md	note the new modes

Verification

• New unit tests green (test_fwht_unit, test_int4_kv_unit); packed-INT4 KL is bit-identical to the INT8-container simulation, confirming pack/unpack correctness.
• INT8 + KIVI paths unregressed; coherent generation in every mode.
• Builds clean under -Werror on macOS (clang) and Pi 5 (gcc).

Follow-ups (not in this PR)

• Packed INT4 keeps an fp32 scale per token-head, shared with the INT8 path. At the real head_dims here (128–512) that is only 0.06–0.25 bits/element on top of 4 → INT4 lands at ~0.51× INT8, so the halving holds. An fp16 or grouped scale would only matter for a small-head model (hd ≤ 64); low priority.
• No GPU/Metal path (CPU experiment).

🤖 Generated with Claude Code

[geisten]

Long-context decode: packed INT4 crosses over and beats INT8

Follow-up measurement (the PR notes flagged this as open). Decode ms/tok on the Pi 5 (A76, BitNet-2b4t, GEIST_BENCH_TG=64, min-of-3, quiesced) across growing KV length:

kv_len	INT8	INT4	INT4+ROT	INT4 vs INT8
~256	65.96	66.57	67.41	+0.9% (slower)
~1024	101.84	100.14	101.30	−1.7% (faster)
~2048	150.45	144.52	145.83	−3.9%
~3072	202.91	187.92	190.15	−7.4%

The crossover sits between kv_len 256 and 1024, and the gap widens with context. By ~3072, packed INT4 is 7.4% faster than INT8 — and even INT4+ROT (rotation cost included) is 6.3% faster than plain INT8 (190.2 vs 202.9).

Mechanism, confirmed: at short context decode is weight-bandwidth-bound, so the halved KV reads are invisible and only the unpack compute shows (+0.9%). As kv_len grows, KV-cache streaming comes to dominate decode bandwidth; halving it then outweighs the unpack, which overlaps the memory stalls.

Takeaway: the earlier "+1–3% decode cost" is the short-context worst case. In the regime packed INT4 is actually for — long chats, RAG, agents — it delivers half the KV memory, near-lossless quality, AND faster decode simultaneously. INT4+ROT is the sweet spot at long context.