docs: README; v1 GPU config, report script, runbook; store raw_output; widen CI lint

Author: pjdurden

.github/workflows/ci.yml

@@ -8,7 +8,7 @@ jobs:
       - uses: actions/setup-python@v5
         with: { python-version: "3.11" }
       - run: pip install -e ".[dev]"
-      - run: ruff check kvcanary
+      - run: ruff check kvcanary tests scripts
       - run: pytest -q
       - run: pip install -e ".[dev,ml]"
       - run: pytest -q -m slow      # CPU end-to-end smoke (downloads distilgpt2)

README.md

@@ -0,0 +1,120 @@
+# kv-canary
+
+**A canary for *silent* KV-cache-compression failures.**
+
+Lossy KV-cache compression (quantization, token eviction) is benchmarked almost entirely on
+**speed and memory** — and on token-level quality (perplexity). But token-level metrics can stay
+flat while **functional** outputs silently break: generated code stops passing its tests, tool
+calls stop being valid JSON. kv-canary measures that gap.
+
+> The wedge: everyone measures how *fast* KV compression is. Almost nobody measures whether it
+> silently breaks your code and tool calls while perplexity says everything is fine.
+
+This is a measurement harness + methodology, motivated by VeriCache
+([arXiv:2605.17613](https://arxiv.org/html/2605.17613)), which showed lossy-KV bias accumulates
+linearly and functional tasks fail while token-level metrics look unchanged. kv-canary makes that
+effect reproducible and quantifiable across compression methods.
+
+---
+
+## The idea in one chart
+
+For each compression method, kv-canary plots two lines against **KV memory retained**:
+
+- **functional accuracy** (solid) — does generated code pass its tests / is the tool call correct?
+- **perplexity-implied quality** (dashed) — the token-level metric, normalized to the full cache.
+
+The finding is the **divergence**: functional accuracy cliffing while the perplexity line hugs 1.0.
+That visual gap is the whole point — the token metric tells you nothing is wrong while real
+functional quality collapses.
+
+To make it a single number, kv-canary defines the **Silent Degradation Score (SDS)**:
+
+```
+SDS = (relative functional drop) / (relative perplexity rise)
+```
+
+High SDS = functional accuracy craters while perplexity barely moves (the metric "lies"). SDS ≈ 1
+is graceful degradation (perplexity warns you in step). Perplexity *improving* while function
+breaks is the most deceptive case of all, and SDS reports it as maximally silent — by design.
+
+## What it measures
+
+| Compression family | Methods | How `budget` maps |
+|---|---|---|
+| baseline | full (fp16) | 1.0 |
+| quantization | int8 / int4 / int2 KV | bits / 16 |
+| token eviction | StreamingLLM, SnapKV | fraction of tokens kept |
+
+Against two objective, KV-eviction-sensitive functional tasks — **code execution** (pass@1) and
+**tool/JSON calling** (valid + correct-function + arg-match) — contrasted with **perplexity** on a
+held-out set. All on a shared *KV-memory-retained* x-axis, so quantization and eviction are directly
+comparable (and the fact that equal memory budgets degrade them differently is the finding).
+
+## Quickstart (CPU, no GPU)
+
+Verifies the whole pipeline end-to-end on a tiny model:
+
+```bash
+pip install -e ".[dev,ml]"
+python -m kvcanary run configs/smoke.yaml --out results/raw/smoke.jsonl
+python scripts/aggregate_and_report.py results/raw/smoke.jsonl
+# -> report/divergence.png, report/RESULTS.md
+```
+
+> The smoke run uses `distilgpt2` and proves the harness runs. See the note on the v1 deferral below
+> for why the *finding* itself needs a GPU.
+
+Run the fast test suite:
+
+```bash
+pip install -e ".[dev]"
+pytest -q          # model-free unit tests
+pytest -q -m slow  # + end-to-end smoke on distilgpt2 (downloads weights)
+```
+
+## How it works
+
+Three small, independently-tested seams compose into a resumable experiment runner:
+
+- **`KVCompressor`** — `full`, `quantized`, `streamingllm`, `snapkv` behind one `budget` knob; the
+  eviction selection logic is a pure function of attention scores (unit-tested on synthetic tensors).
+- **`Backend`** — `HFBackend` (HuggingFace causal LM: greedy generation + teacher-forced
+  perplexity) and `FakeBackend` (a deterministic double, so the runner and tasks test with no model).
+- **`Task`** — `CodeExecTask` (sandboxed subprocess, pass@1), `ToolCallTask` (JSON parse + function
+  + arg match), `PerplexityTask`.
+
+The runner sweeps `models × compressors × tasks`, writes one JSON line per sample, and **resumes**
+by skipping already-completed cells — so a killed spot-GPU run restarts cheaply. `metrics/` and
+`report/` turn the JSONL into the divergence chart and the SDS table.
+
+```
+configs/*.yaml ─▶ runner ─▶ results/raw/*.jsonl ─▶ aggregate ─▶ report/{divergence.png, RESULTS.md}
+                    │
+        KVCompressor · Backend · Task
+```
+
+## Status & honest caveats
+
+**v1 is the harness.** 16 build steps, ~38 tests, green CI. The real run is documented in
+[`docs/RUNBOOK.md`](docs/RUNBOOK.md) and needs a single 24GB GPU.
+
+- **The headline finding is not in this repo yet.** One piece is intentionally deferred to the GPU
+  run: wiring the eviction selection into HuggingFace's live KV cache (a custom `DynamicCache`
+  subclass). Until that lands, `HFBackend` records *how much* KV each method would retain but runs
+  normal generation — so **all methods produce identical output and the divergence is NULL.** This
+  is called out in the `HFBackend` docstring and the runbook. Any numbers you see in a freshly
+  generated `report/` from the CPU smoke are pipeline-shape placeholders, **not** a result.
+- The gap kv-canary targets rests on a single recent paper (VeriCache); the contribution here is
+  making it *measurable and reproducible*, not claiming novelty of the phenomenon.
+- The wedge is **functional-vs-token-level divergence**, not "reproducible benchmarking" — local
+  inference benchmarking is fragmented (a coverage gap), but that is a different problem.
+
+## Layout
+
+```
+kvcanary/        compressors/ · backends/ · tasks/ · runner/ · metrics/ · report/
+configs/         smoke.yaml (CPU) · v1.yaml (GPU matrix)
+scripts/         aggregate_and_report.py
+docs/            RUNBOOK.md · superpowers/specs/ (design) · superpowers/plans/ (build plan)
+```

configs/v1.yaml

@@ -0,0 +1,16 @@
+# configs/v1.yaml  — full experiment matrix (run on a rented 24GB spot GPU)
+models: [Qwen/Qwen2.5-7B-Instruct]
+tasks: [code, tool, perplexity]
+compressors:
+  - {method: full,         budget: 1.0}
+  - {method: quantized,    budget: 0.5}
+  - {method: quantized,    budget: 0.25}
+  - {method: quantized,    budget: 0.125}
+  - {method: streamingllm, budget: 0.75}
+  - {method: streamingllm, budget: 0.5}
+  - {method: streamingllm, budget: 0.25}
+  - {method: streamingllm, budget: 0.125}
+  - {method: snapkv,       budget: 0.75}
+  - {method: snapkv,       budget: 0.5}
+  - {method: snapkv,       budget: 0.25}
+  - {method: snapkv,       budget: 0.125}

docs/RUNBOOK.md

@@ -0,0 +1,28 @@
+# kv-canary — GPU run runbook
+
+The CPU smoke (`configs/smoke.yaml`, distilgpt2) proves the pipeline end-to-end. The real finding
+needs a GPU. Steps:
+
+1. **Rent a 24GB spot GPU** (4090 / A10 / L4). `pip install -e ".[dev,ml]"`.
+2. **Wire real eviction into HF's KV cache.** This is the one deferred piece. `HFBackend` currently
+   records `kv_bytes_retained` but does NOT yet apply eviction/quantization during the forward pass
+   (see the class docstring in `kvcanary/backends/hf.py`): until this lands, ALL compressors produce
+   identical text/perplexity and the divergence will be NULL. Implement a custom `DynamicCache`
+   subclass whose `update()` calls `compressor.kept_indices(...)` (the pure selection logic is
+   already tested in Tasks 4–6) and pass it into `model.generate(..., past_key_values=...)`.
+   Validate on `Qwen2.5-0.5B` first.
+3. **Run the matrix** (resumable — safe to re-run after a spot kill):
+   `python -m kvcanary run configs/v1.yaml --out results/raw/v1.jsonl`
+4. **Aggregate + report:** `python scripts/aggregate_and_report.py results/raw/v1.jsonl`
+5. **Inspect `report/divergence.png`:** the eviction methods' functional (solid) lines should cliff
+   while their perplexity-quality (dashed) lines hug 1.0 — that gap is the finding. The SDS column in
+   `report/RESULTS.md` quantifies it per (method, budget).
+6. **CONTINUE** if the divergence reproduces on >=1 eviction method + task (independently confirming
+   the VeriCache effect). **KILL / pivot** to the benchmark-coverage harness if functional
+   degradation tracks perplexity 1:1 (no divergence) — see spec §8.
+
+## Caveats (carry into the write-up)
+- The wedge is functional-vs-token-level DIVERGENCE, not "reproducible benchmarking" (that gap is
+  coverage, not reproducibility — see spec §9).
+- Quant and eviction share a memory-retained x-axis but are different mechanisms; the differing
+  degradation is the point.

kvcanary/runner/runner.py

@@ -36,17 +36,17 @@ def run_cells(cells, backend, compressor_factory, tasks, out_path: str) -> int:
                 if task.needs_generation:
                     gen = backend.generate(sample.prompt, compressor=comp)
                     res = task.evaluate(sample, output=gen.text)
-                    n_tok, kv = gen.n_tokens, gen.kv_bytes_retained
+                    raw, n_tok, kv = gen.text, gen.n_tokens, gen.kv_bytes_retained
                 else:
                     ppl = backend.score(sample.prompt, compressor=comp)
                     res = task.evaluate(sample, output="", ppl=ppl)
                     full_bytes = getattr(backend, "full_bytes", 1000.0)
-                    n_tok, kv = 0, comp.kv_bytes_retained(full_bytes)
+                    raw, n_tok, kv = "", 0, comp.kv_bytes_retained(full_bytes)
                 row = CellResult(
                     spec=spec,
                     sample_id=sample.id,
                     scores=res.scores,
-                    raw_output="",
+                    raw_output=raw,
                     n_tokens=n_tok,
                     latency_s=time.perf_counter() - t0,
                     kv_bytes_retained=kv,

scripts/aggregate_and_report.py

@@ -0,0 +1,16 @@
+import json
+import os
+import sys
+
+from kvcanary.metrics.aggregate import aggregate_rows
+from kvcanary.report.chart import render_divergence_chart
+from kvcanary.report.markdown import render_results_md
+
+with open(sys.argv[1]) as f:
+    rows = [json.loads(line) for line in f if line.strip()]
+agg = aggregate_rows(rows)
+os.makedirs("report", exist_ok=True)
+render_divergence_chart(agg, "report/divergence.png")
+with open("report/RESULTS.md", "w") as f:
+    f.write(render_results_md(agg))
+print("wrote report/divergence.png and report/RESULTS.md")

tests/test_runner.py

@@ -27,6 +27,7 @@ def test_runner_writes_rows_and_resumes(tmp_path):
     assert n1 == 2                                   # 2 cells x 1 sample
     rows = [json.loads(line) for line in out.read_text().splitlines()]
     assert {r["cell_id"] for r in rows} == {"toy|full|1.0|tool", "toy|streamingllm|0.5|tool"}
+    assert all(r["raw_output"] == '{"name": "f", "arguments": {}}' for r in rows)  # generated text stored
     n2 = run_cells(cells, backend, _factory, tasks, str(out))   # resume
     assert n2 == 0                                   # everything already done
     assert len(out.read_text().splitlines()) == 2    # no duplicate rows