docs: fix VeriCache misattribution, drop false novelty wedge, add cited gap audit

VeriCache (2605.17613) is real but doesn't claim perplexity-insensitivity; re-cite Pitfalls (2510.00231) and frame 'perplexity is blind' as a hypothesis under test. Drop the contradicted 'nobody measures functional correctness' wedge (KVPress et al. already do); narrow to executable-code + tool-schema conformance. Flag StreamingLLM no-re-roping and head-averaged SnapKV as known fidelity limitations. Add docs/RESEARCH.md (full adversarially-verified audit). Docs only; no code changed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: pjdurden

README.md

@@ -6,18 +6,25 @@
 
 **A canary for *silent* KV-cache-compression failures.**
 
-Lossy KV-cache compression (quantization, token eviction) is benchmarked almost entirely on
-**speed and memory** — and on token-level quality (perplexity). But token-level metrics can stay
-flat while **functional** outputs silently break: generated code stops passing its tests, tool
-calls stop being valid JSON. kv-canary measures that gap.
-
-> The wedge: everyone measures how *fast* KV compression is. Almost nobody measures whether it
-> silently breaks your code and tool calls while perplexity says everything is fine.
-
-This is a measurement harness + methodology, motivated by VeriCache
-([arXiv:2605.17613](https://arxiv.org/html/2605.17613)), which showed lossy-KV bias accumulates
-linearly and functional tasks fail while token-level metrics look unchanged. kv-canary makes that
-effect reproducible and quantifiable across compression methods.
+Lossy KV-cache compression (quantization, token eviction) silently breaks **functional** outputs:
+generated code stops passing its tests, tool calls stop being valid JSON. VeriCache
+([arXiv:2605.17613](https://arxiv.org/abs/2605.17613)) shows these methods are "inherently
+lossy … their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded,
+which leads to catastrophic failures in code generation and tool calling." *The Pitfalls of KV Cache
+Compression* ([arXiv:2510.00231](https://arxiv.org/abs/2510.00231), ACL 2026) shows aggregate
+benchmark metrics hide these per-instruction failures.
+
+> The question kv-canary tests: when KV compression breaks your code and tool calls, does the cheap
+> token-level metric (perplexity) **warn you, or stay silent?** This is a *hypothesis under test* —
+> no paper has shown perplexity is blind to KV-compression damage; kv-canary measures whether it is.
+
+**Scope & prior art (honest).** Functional/downstream degradation under KV compression is *already*
+measured by [NVIDIA KVPress](https://github.com/NVIDIA/kvpress), Pitfalls (IFEval), and
+[arXiv:2512.12008](https://arxiv.org/abs/2512.12008) (8 reasoning benchmarks). What those don't score
+is **executable code (pass@k) and JSON/tool-call schema conformance** specifically — that narrow
+target, contrasted against perplexity, is kv-canary's niche. See
+[`docs/RESEARCH.md`](docs/RESEARCH.md) for the full cited gap audit, including known implementation
+limitations (StreamingLLM re-roping, per-head SnapKV) that a faithful run must address.
 
 ---
 
@@ -100,19 +107,26 @@ configs/*.yaml ─▶ runner ─▶ results/raw/*.jsonl ─▶ aggregate ─▶
 
 ## Status & honest caveats
 
-**v1 is the harness.** 16 build steps, ~38 tests, green CI. The real run is documented in
-[`docs/RUNBOOK.md`](docs/RUNBOOK.md) and needs a single 24GB GPU.
-
-- **The headline finding is not in this repo yet.** One piece is intentionally deferred to the GPU
-  run: wiring the eviction selection into HuggingFace's live KV cache (a custom `DynamicCache`
-  subclass). Until that lands, `HFBackend` records *how much* KV each method would retain but runs
-  normal generation — so **all methods produce identical output and the divergence is NULL.** This
-  is called out in the `HFBackend` docstring and the runbook. Any numbers you see in a freshly
-  generated `report/` from the CPU smoke are pipeline-shape placeholders, **not** a result.
-- The gap kv-canary targets rests on a single recent paper (VeriCache); the contribution here is
-  making it *measurable and reproducible*, not claiming novelty of the phenomenon.
-- The wedge is **functional-vs-token-level divergence**, not "reproducible benchmarking" — local
-  inference benchmarking is fragmented (a coverage gap), but that is a different problem.
+**v1 is the harness, wired end-to-end.** Compression is applied into HF's live KV cache
+(`CompressedCache`), `score()` measures perplexity under the compressed context, and the divergence
+is real (not NULL). The headline GPU run on a real instruct model is documented in
+[`docs/RUNBOOK.md`](docs/RUNBOOK.md); run it before trusting any numbers. Full cited audit in
+[`docs/RESEARCH.md`](docs/RESEARCH.md). Known limitations a faithful run must weigh:
+
+- **The method implementations are simplified and a faithful reviewer would flag two of them.**
+  StreamingLLM here evicts from a *post-RoPE* HF cache **without re-roping** kept keys to contiguous
+  cache positions — the real method caches keys pre-RoPE and re-indexes positions, so eviction here
+  is degraded partly *for the wrong reason* (see RESEARCH.md Q3). SnapKV here **averages attention
+  over heads** with no pooling, whereas real SnapKV selects **per-head with a max-pool clustering
+  step** (Q4). Treat these as "StreamingLLM-/SnapKV-style" eviction, not exact reproductions.
+- **"Perplexity is blind" is a hypothesis under test, not a cited result.** VeriCache shows the
+  functional failures; no paper shows perplexity fails to warn. kv-canary measures whether it does.
+- **Prior art exists.** KVPress / Pitfalls / arXiv:2512.12008 already measure functional/downstream
+  degradation under KV compression; kv-canary's narrow niche is executable-code + tool-schema
+  conformance specifically. The originality is that corner, not the phenomenon.
+- **Design scope.** Compression is applied to the prompt once (no decode-time rolling); VeriCache's
+  failures accrue over long *decode*, so this design under-exercises that regime (RESEARCH.md §Deeper
+  problems). Quant is per-tensor absmax and `bits/16` ignores scale/zero-point overhead.
 
 ## Layout
 

docs/RESEARCH.md

@@ -0,0 +1,119 @@
+# kv-canary — foundations & gap audit (deep research, 2026-06-12)
+
+A multi-source, adversarially-verified review of the claims and method implementations behind
+kv-canary. 23 sources fetched, 99 claims extracted, 25 verified by 3-vote adversarial checking
+(24 confirmed, 1 killed). Load-bearing arXiv IDs were independently re-fetched. Findings that
+change the project are recorded here so the repo is honest about what it does and doesn't establish.
+
+## TL;DR verdict
+
+- The **thesis is sound and correctly grounded**: lossy KV-cache compression really does cause
+  catastrophic functional failures in code generation and tool calling (VeriCache, real).
+- The **originality wedge is false**: functional/downstream degradation under KV compression is
+  already measured by NVIDIA KVPress and multiple 2025–2026 papers. Only a *narrow* corner survives.
+- **Two method implementations are wrong**, not merely simplified: StreamingLLM without re-roping,
+  and head-averaged SnapKV without pooling. A knowledgeable reviewer would dismiss those results.
+- The **prompt-only design may be confounded** for the long-decode failure mode it targets.
+
+## Q1 — Citation check
+
+**VeriCache is real.** `arXiv:2605.17613` = *"VeriCache: Turning Lossy KV Cache into Lossless LLM
+Inference"* (Yao, Shen, Du, Feng, Seo, Zhang, Huang, Liu, Lu, Jiang; May 17 2026). Abstract:
+> "almost all of these methods are inherently lossy—despite minimal accuracy degradation for short
+> outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded,
+> which leads to catastrophic failures in code generation and tool calling."
+
+**But VeriCache does NOT make the "perplexity is blind" argument** (verified, 0-3 — the one killed
+claim). It never mentions perplexity; it frames short-output-small vs long-output-large divergence.
+Citing VeriCache for "token-level metrics stay flat" is a **misattribution**.
+
+The closest real support for "aggregate/token-level metrics hide functional damage" is
+***The Pitfalls of KV Cache Compression*** (`arXiv:2510.00231`, Chen, Geh, Grover, Van den Broeck,
+Israel; UCLA; ACL 2026):
+> "certain instructions degrade much more rapidly with compression, effectively causing them to be
+> completely ignored by the LLM."
+
+It evaluates the exact methods kv-canary models (StreamingLLM, SnapKV, +TOVA/H2O/K-Norm) on
+Llama3.1-8B and Qwen2.5-14B — but measures **IFEval instruction-following, not perplexity**. So
+"perplexity fails to warn" is kv-canary's **hypothesis to test**, not an established result. Source:
+https://arxiv.org/abs/2605.17613 · https://arxiv.org/abs/2510.00231
+
+## Q2 — Is the wedge novel? No (broadly)
+
+"Everyone measures speed/memory, nobody measures functional correctness under KV compression" is
+**contradicted**:
+- **NVIDIA KVPress** (https://github.com/NVIDIA/kvpress, maintained) — accuracy CLI over RULER,
+  LongBench/-v2, Loogle, InfiniteBench, Zero-Scrolls, Needle-in-a-Haystack.
+- **Pitfalls** (2510.00231) — IFEval instruction-following.
+- **Hold Onto That Thought** (`arXiv:2512.12008`, Dec 2025) — 8 reasoning benchmarks (FOLIO, DROP,
+  GSM8K, MATH-500, ReClor, StrategyQA, CommonSenseQA, OpenBookQA), built on the kvpress library.
+
+**Narrow survivor:** none of these score **executable-code-passing-unit-tests + valid-JSON/tool-call
+schema conformance** specifically (they do QA / retrieval / reasoning / instruction-following
+accuracy). That narrow corner is kv-canary's only remaining originality. Source:
+https://github.com/NVIDIA/kvpress · https://arxiv.org/abs/2512.12008
+
+## Q3 — StreamingLLM re-roping (CONFIRMED BUG, 3-0)
+
+StreamingLLM **requires** caching keys *pre*-RoPE and assigning positions by **cache index**, not
+original token position. Paper (`arXiv:2309.17453`, ICLR 2024):
+> "When determining the relative distance and adding positional information to tokens, StreamingLLM
+> focuses on positions within the cache rather than those in the original text." … "For encoding like
+> RoPE, we cache the Keys of tokens prior to introducing the rotary transformation."
+
+Worked example: cache `[0,1,2,3,6,7,8]` decoding token 9 uses positions `[0,1,2,3,4,5,6,7]`.
+HF transformers issue #35350 confirms post-RoPE eviction yields "confused position"; KVPress ships a
+`KeyRerotationPress` to correct it. **kv-canary evicts from a post-RoPE HuggingFace DynamicCache
+without re-roping → degraded-for-the-wrong-reason.** Source: https://arxiv.org/abs/2309.17453 ·
+https://github.com/huggingface/transformers/issues/35350 · https://github.com/NVIDIA/kvpress
+
+## Q4 — SnapKV per-head + pooling (CONFIRMED MISREPRESENTATION, 3-0)
+
+SnapKV (`arXiv:2404.14469`, NeurIPS 2024) selects "clustered important KV positions **for each
+attention head**", preserving the head dimension throughout, and a **1D max-pooling clustering step
+is mandatory** (ablated as essential: "with the pooling … performs significantly better"). Observation
+window = last segment of the prompt (~16–64 tokens) used to vote; top-k per head after pooling.
+**kv-canary averages attention over all heads into one per-layer keep set with no pooling — that is
+not SnapKV.** Source: https://arxiv.org/abs/2404.14469 · https://github.com/FasterDecoding/SnapKV
+
+## Q5 — Perplexity protocol & quant memory (medium confidence)
+
+- A single **0.5 context/continuation split** is legitimate but **non-standard**; the HF/NLP
+  convention is **sliding-window (strided) perplexity**. A single split underestimates variance and
+  is cherry-pickable. (https://huggingface.co/docs/transformers/perplexity)
+- **`kv_bytes_retained = bits/16`** ignores per-tensor/group **scale + zero-point** overhead, so true
+  retained memory is higher than reported — compression is modestly **overstated**. Per-tensor absmax
+  quant is also cruder than the per-token/per-channel group-wise quant used in real KV-quant (KIVI,
+  KVQuant), so kv-canary's quant results are likely **pessimistic**.
+
+## Deeper problems (matter more than the bugs)
+
+1. **Design vs. failure mode.** VeriCache's functional failures arise during **long decode**;
+   kv-canary compresses the **prompt once, no decode-time rolling**. `2512.12008` finds decode-time
+   compression is what governs reasoning correctness. The prompt-only design may be **confounded for
+   the very code/tool-over-long-decode failures it targets.**
+2. **"Why not just use KVPress?"** Fixing Q3+Q4 properly ≈ reimplementing KVPress (worse). NVIDIA
+   maintains it.
+
+## Recommended direction
+
+Reframe the contribution as **executable / structured-output correctness (code-exec pass@k +
+JSON/tool-call schema validity) under *decode-time* KV compression**, ideally **built on top of
+NVIDIA KVPress** rather than a solo reimplementation — inheriting KVPress's correct StreamingLLM/
+SnapKV (Q3/Q4 gone), targeting the one gap KVPress's accuracy CLI doesn't cover, and landing in a
+high-visibility NVIDIA repo. If kept standalone: drop the broad wedge, frame "perplexity warns?" as a
+hypothesis, fix re-roping + per-head-SnapKV-with-pooling, and add decode-time compression.
+
+## Open questions
+
+- Is there ANY existing tool scoring executable-code + tool-schema conformance under KV compression?
+  If not, that narrow wedge is the whole contribution.
+- Authoritative magnitudes for Q5 (strided-perplexity doc; KIVI/KVQuant overhead bytes).
+- Can a prompt-only design generalize to decode-time functional failures at all?
+
+## Caveats on this audit
+
+Fast-moving subfield (key papers Oct/Dec 2025, May 2026; KVPress v0.3.0 June 2026) — the prior-art
+picture can shift in months. Q1–Q4 rest on primary sources with unanimous/near-unanimous votes and
+are robust; Q5 is medium confidence (lives inside a synthesizing verifier's note, not a separately
+primary-cited claim).

docs/index.html

@@ -41,13 +41,16 @@ <h1>kv-<span class="canary">canary</span></h1>
     <a href="https://github.com/pjdurden/kv-canary/blob/main/docs/RUNBOOK.md">Runbook</a>
   </p>
 
-  <p>Lossy KV-cache compression (quantization, token eviction) is benchmarked almost entirely on
-  <strong>speed and memory</strong>, and on token-level quality (perplexity). But token-level
-  metrics can stay flat while <strong>functional</strong> outputs silently break: generated code
-  stops passing its tests, tool calls stop being valid JSON. kv-canary measures that gap.</p>
+  <p>Lossy KV-cache compression (quantization, token eviction) silently breaks <strong>functional</strong>
+  outputs: generated code stops passing its tests, tool calls stop being valid JSON.
+  <a href="https://arxiv.org/abs/2605.17613">VeriCache</a> shows these methods' outputs "increasingly
+  diverge from full-KV-cache outputs as more tokens are decoded, [leading] to catastrophic failures in
+  code generation and tool calling"; <a href="https://arxiv.org/abs/2510.00231">The Pitfalls of KV
+  Cache Compression</a> shows aggregate metrics hide those per-instruction failures.</p>
 
-  <p class="pitch">Everyone measures how <em>fast</em> KV compression is. Almost nobody measures
-  whether it silently breaks your code and tool calls while perplexity says everything is fine.</p>
+  <p class="pitch">The question kv-canary tests: when KV compression breaks your code and tool calls,
+  does the cheap token-level metric (perplexity) warn you, or stay silent? That is a <em>hypothesis
+  under test</em> — no paper has shown perplexity is blind to KV-compression damage.</p>
 
   <h2>The finding: divergence</h2>
   <p>For each compression method, kv-canary plots two lines against <strong>KV memory retained</strong>:
@@ -88,12 +91,15 @@ <h2>How it works</h2>
   finished cells — so a killed spot-GPU run restarts cheaply.</p>
 
   <div class="note">
-    <strong>Status — honest caveat.</strong> This is the measurement harness. The headline finding
-    is not in the repo yet: one piece is intentionally deferred to the GPU run — wiring the eviction
-    selection into HuggingFace's live KV cache. Until that lands, all methods produce identical output
-    and the divergence is NULL. Any numbers in a freshly generated report from the CPU smoke are
-    pipeline-shape placeholders, not a result. See the
-    <a href="https://github.com/pjdurden/kv-canary/blob/main/docs/RUNBOOK.md">runbook</a>.
+    <strong>Status &amp; honest caveats.</strong> The harness is wired end-to-end — compression is
+    applied into HF's live KV cache and perplexity is measured under it, so the divergence is real
+    (run the GPU matrix on a real instruct model before trusting numbers). Two method implementations
+    are simplified and a faithful reviewer would flag them: StreamingLLM evicts from a post-RoPE cache
+    <em>without re-roping</em>, and SnapKV is head-averaged without pooling — so treat these as
+    "-style" eviction, not exact reproductions. Functional-degradation-under-KV-compression is also
+    already measured by NVIDIA KVPress and others; the narrow niche here is executable-code +
+    tool-schema conformance vs. perplexity. Full cited audit:
+    <a href="https://github.com/pjdurden/kv-canary/blob/main/docs/RESEARCH.md">RESEARCH.md</a>.
   </div>
 
   <footer>