diff --git a/README.md b/README.md index 2e401c4..c421f0b 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Bulkhead is a small Solana program that holds a protocol's token accounts on the **[The attack](#what-it-does-to-an-attack)** · **[How it works](#how-it-works)** · **[Verify it](#verify-it-yourself)** · **[Devnet](#live-on-devnet)** · **[FAQ](docs/FAQ.md)** · **[Docs](#documentation)** -[![CI](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml/badge.svg)](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml) [![devnet](https://img.shields.io/badge/devnet-live-57606A?style=flat-square)](#live-on-devnet) [![Kani](https://img.shields.io/badge/Kani_proofs-15%2F15-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![tests](https://img.shields.io/badge/program_tests-123%2F123-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![audit](https://img.shields.io/badge/audit-not_yet,_not_for_real_funds-E5481B?style=flat-square)](SECURITY.md) [![license](https://img.shields.io/badge/license-Apache--2.0-57606A?style=flat-square)](LICENSE) +[![CI](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml/badge.svg)](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml) [![devnet](https://img.shields.io/badge/devnet-live-57606A?style=flat-square)](#live-on-devnet) [![Kani](https://img.shields.io/badge/Kani_proofs-17%2F17-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![tests](https://img.shields.io/badge/program_tests-133%2F133-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![audit](https://img.shields.io/badge/audit-not_yet,_not_for_real_funds-E5481B?style=flat-square)](SECURITY.md) [![license](https://img.shields.io/badge/license-Apache--2.0-57606A?style=flat-square)](LICENSE)
@@ -144,7 +144,7 @@ scripts/check-programs.sh # the programs on the real runtime: needs the E2E=1 scripts/check-programs.sh # ...plus the attacker-vs-guardian run and a deploy rehearsal, on local validators scripts/prove.sh # Kani bounded proofs (~6 min) scripts/mutants.sh # mutation testing (~45 min) -scripts/fuzz.sh # coverage-guided fuzzing, 8 targets (SECONDS_PER=600 for longer) +scripts/fuzz.sh # coverage-guided fuzzing, 9 targets (SECONDS_PER=600 for longer) scripts/deny.sh # supply chain: RustSec advisories, licences, crate sources python3 tools/floor_check.py # who can upgrade any mainnet program; is it ready for SIMD-0500? (--rpc for another cluster) scripts/floor-demo.sh # Floor on that program's real bytes, locally: register, stop, restore byte-identical @@ -165,12 +165,12 @@ These scripts install nothing. The core has zero dependencies, and Cloud and the | Check | Result (latest run of each) | |---|---| -| **Programs on the real runtime**
LiteSVM, mainnet feature set, mainnet token program | **123/123**: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction but the two read-only views (6,182 swaps refused, 10 Oct) | -| **Bounded proofs**
Kani, and exhaustive checks | **15/15 harnesses verified** on 8 Oct (12 for every value of their types), 354 s. 6 exhaustive tests over ~584M inputs | -| **Differential**
an independent Python reference vs the Rust core | **6,943,561 lines identical** (6 seeds × 200k cases, 27 Sep); 31 commands now, 300k cases on every pull request and 3M weekly in CI | +| **Programs on the real runtime**
LiteSVM, mainnet feature set, mainnet token program | **133/133**: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction but the two read-only views (6,182 swaps refused, 10 Oct) | +| **Bounded proofs**
Kani, and exhaustive checks | **17/17 harnesses verified** on 14 Oct (14 for every value of their types), 227 s of verification ([`09 §5`](docs/09-VERIFICATION.md#5-bounded-proofs-kani-and-exhaustive-checks)). 6 exhaustive tests over ~584M inputs | +| **Differential**
an independent Python reference vs the Rust core | **6,943,561 lines identical** (6 seeds × 200k cases, 27 Sep); 33 commands now, 300k cases on every pull request and 3M weekly in CI | | **Program vs model** | Random steps on-chain, every outcome equal to the core's prediction: 12,000 (Vault), 300,000 with random limits and every brake (Vault), 200,000 (Floor) | | **Mutation testing**
cargo-mutants, 669 injected bugs, in CI on every change to the core | **Every non-equivalent mutant killed**; 22 equivalent mutants, each with the reason it can't change behavior ([`09 §6`](docs/09-VERIFICATION.md#6-mutation-testing)) | -| **Coverage-guided fuzzing**
libFuzzer, 8 targets: the byte parsers and the full-width limiter and ledger | 36M–277M executions per target in the 6 Oct run (10 min each), no program bug; every parser at full line coverage ([`09 §2`](docs/09-VERIFICATION.md#2-layers-and-current-results-10-oct-2026)) | +| **Coverage-guided fuzzing**
libFuzzer, 9 targets: the byte parsers and the full-width limiter and ledger | 36M–277M executions per target in the 6 Oct run (10 min each), no program bug; every parser fuzzed in that run at full line coverage. That run had 8 targets: the guardian-set parser's, added 14 Oct, has no recorded 10-minute run or coverage measurement ([`09 §2`](docs/09-VERIFICATION.md#2-layers-and-current-results-10-oct-2026)) | | **Property tests**
G1–G5, fail closed, tightening, the SIMD-0500 gate, snapshot identity, ejection | 26/26, with coverage counters so a vacuous pass fails | | **Golden vectors**
every worked example in the docs, exact boundaries, the abort binaries' bytes, loader layouts | 119/119 identical, Rust = Python | | **End to end** | [Above](#what-it-does-to-an-attack) | @@ -226,6 +226,7 @@ programs/tests/ LiteSVM: scenarios, negative tests, compute, the program cloud/ zero-dependency JavaScript: client, guardian bot, status page, end-to-end attack, deployment and live-cluster checks tools/floor_check.py read-only report on any upgradeable program +tools/claims.py the counts README and docs/ state, checked against the tree (scripts/check.sh) reference/ independent Python reference, differential runner, golden vectors, fixtures, Drift model evidence/ governance census (script + committed snapshot), model output docs/ the full design (below) diff --git a/SECURITY.md b/SECURITY.md index b7fdd82..efeb43e 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,7 +6,7 @@ | Component | State | Checked by | |---|---|---| -| `bulkhead-core` (the math) | Built | An independent Python reference with a differential runner, golden vectors, property tests, exhaustive checks, 15 Kani harnesses, mutation testing (`docs/09-VERIFICATION.md`) | +| `bulkhead-core` (the math) | Built | An independent Python reference with a differential runner, golden vectors, property tests, exhaustive checks, 17 Kani harnesses, mutation testing (`docs/09-VERIFICATION.md`) | | Programs: Floor, Vault, and Harbor (a demo protocol) | Built for sBPF v3 | LiteSVM on the real agave runtime and loader (scenarios, negative tests, compute), a program-vs-model differential, and an end-to-end run on a local validator | | Bulkhead Cloud (guardian bot, status page, client) | Built | Unit tests, a JavaScript-vs-Python differential, the end-to-end run | | Devnet deployment | Deployed 28 Sep 2026; upgraded in place 29 Sep, 6 Oct, 7 Oct, twice on 8 Oct, six times on 9 Oct, once on 10 Oct, once on 11 Oct and once on 15 Oct (below) | All three programs run the canonical build (Linux x86_64; CI publishes it) of `main` at PR #85 (Harbor unchanged since 6 Oct), matched byte for byte with the upgrade authority checked | diff --git a/docs/04-THREAT-MODEL.md b/docs/04-THREAT-MODEL.md index 52a4067..f2d9ad7 100644 --- a/docs/04-THREAT-MODEL.md +++ b/docs/04-THREAT-MODEL.md @@ -71,7 +71,7 @@ A security product with a bug is worse than no product. This file lists every ad | I12 | No token leaves a vault except release, settle, lifeboat | program-vs-model differential (vault balance equals the model after every step) | | I13 | Releases fail before the seal | program `nothing_can_be_released_before_the_seal…` | | I14 | `vault.queued_amount == Σ pending tickets` | program-vs-model differential, after every step | -| I15 | Floor: no new binary before 72h unless a quorum approved that exact frozen buffer during a restore, or the snapshot is restored | 41 Floor tests on the real loader (`programs/tests/tests/floor.rs`) | +| I15 | Floor: no new binary before 72h unless a quorum approved that exact frozen buffer during a restore, or the snapshot is restored | 45 Floor tests on the real loader (`programs/tests/tests/floor.rs`) | | I16 | Python reference and Rust core agree on every input | diff (6,943,561 identical lines in the final 6-seed run; 36 fixtures with expected outputs) | ## 4. The #1 risk: a bug in Bulkhead itself diff --git a/docs/05-PROGRAM-SPEC.md b/docs/05-PROGRAM-SPEC.md index 00baf77..f4ba51c 100644 --- a/docs/05-PROGRAM-SPEC.md +++ b/docs/05-PROGRAM-SPEC.md @@ -155,7 +155,7 @@ Codes from 8000 (`error.rs`), disjoint from Floor (7000+), Anchor (6000+) and th ## 10. Compute (measured on the real runtime, through Harbor's CPI) -Measured 8 Oct 2026 by `compute_units_are_within_the_spec_targets` (`programs/tests/tests/vault.rs`), on a macOS (arm64) build of this tree. The canonical Linux build orders functions differently, so its figures can differ slightly. +Measured 8 Oct 2026 by `compute_units_are_within_the_spec_targets` (`programs/tests/tests/vault.rs`), on a macOS (arm64) build of that day's tree (#35, `25b5a87`); the Vault has changed since. Two figures are later and name their source: a first deposit's cost at a low bump (finding 75) and the full-ledger row (its own test, 14 Oct). The canonical Linux build orders functions differently, so its figures can differ slightly. | Path | Compute units | Target | |---|---|---| diff --git a/docs/09-VERIFICATION.md b/docs/09-VERIFICATION.md index ab8528f..9295484 100644 --- a/docs/09-VERIFICATION.md +++ b/docs/09-VERIFICATION.md @@ -21,17 +21,17 @@ The programs are checked against the core as a model (§7). Two implementations |---|---|---| | **Golden vectors** | 119 curated outputs (`reference/golden/`): every worked example in `03`, exact boundaries (including the backed-amount search where the valid set excludes 0), the abort binaries' bytes, loader-v3 layouts, restore and eject rules | 119/119 identical (Rust = Python) | | **Regression fixtures** | `reference/regressions/`: 36 cases from real past mismatches, each with its current expected output (`reference/golden/regressions-expected.txt`) | 36/36 reproduced by the Rust core in `cargo test` | -| **Differential runner** | Random cases for 31 commands (limiter, capacity, EMA, value, price, ledger, classification, ELF, governance, eject, Floor parsers, snapshot identity, murmur3, restore), weighted to edge values | 6,943,561 lines identical in the final 6-seed run (seeds 101–106, 200k cases each, 27 Sep, 29 commands then; ~1.15M lines per seed). Every pull request runs 300k cases, the weekly CI run 3M: identical (6 Oct) | +| **Differential runner** | Random cases for 33 commands (limiter, capacity, EMA, value, price, ledger, classification, ELF, governance, eject, guardian sets, Floor parsers, snapshot identity, murmur3, restore), weighted to edge values | 6,943,561 lines identical in the final 6-seed run (seeds 101–106, 200k cases each, 27 Sep, 29 commands then; ~1.15M lines per seed). Every pull request runs 300k cases, the weekly CI run 3M: identical (6 Oct) | | **Property tests** | 26 tests, seeded SplitMix64; `BULKHEAD_CASES` scales depth; coverage floors (proportional to depth) make vacuous passes fail | Pass at 1.5k, 3k, 20k and 100k cases each | | **Exhaustive checks** | 6 tests: every input of a small domain for the arithmetic properties (§5.2) | ~584M inputs | -| **Floor core tests** | 10 tests: the sBPF v3 strict ELF rules restated, murmur3 published vectors, loader-v3 headers, snapshot identity | Pass | +| **Floor core tests** | 12 tests: the sBPF v3 strict ELF rules restated, murmur3 published vectors, loader-v3 headers, snapshot identity, the ELF extent a snapshot must cover | Pass | | **Targeted unit tests** | 25 tests: 18 each written to kill a surviving mutant (§6, finding 45), `capacity_base`, the saturation case the limiter fuzzer found (finding 40), the guardian-set shape and the n-of-n refusal (finding 96), one request queuing a long window plus the delay (finding 102), and the guardian-set parser at every boundary and in its error order, with the bitmaps (finding 103) | Pass | | **Adversarial tests** | 7 tests (`crates/bulkhead-core/tests/adversarial.rs`): the attacks of `15` run through the real core, before and after each fix | Pass | | **Crash scenario** | 2 tests (`crates/bulkhead-core/tests/cascade.rs`): an honest liquidation cascade on a synthetic −30% day, through the real ledger, pricing and limiter; pins the numbers `08` S11 reports | Pass | | **Bounded proofs (Kani)** | 17 harnesses: logic at full width, 3 at small width (§5) | **17/17 verified** (14 Oct) | -| **Coverage-guided fuzzing** | 9 libFuzzer targets (`fuzz/`, cargo-fuzz, overflow checks and AddressSanitizer on): the Token-2022 mint policy, the token-account reader and the guardian-set parser Floor and the Vault share against independent statements of them; the Pyth `PriceUpdateV2` parser against the documented layout; load-store-load round trips of every Vault and Floor account; the loader-v3 parsers and snapshot identity; the limiter at full width (exact `available_now`, instant and queue rules, errors change nothing, window changes keep the used fraction); the ledger (backed ≤ requested, the same amount on a re-ask, `debit_backed` accepts it). Line coverage is measured (`cargo fuzz coverage`); seeds in `fuzz/seeds/` reach what a mutator rarely builds | 6 Oct: 10 min per target, 36M–277M executions each; one harness-oracle crash (finding 40), no program bug; every targeted parser at 100% of lines after the seeds (finding 41). CI: 60 s per target | +| **Coverage-guided fuzzing** | 9 libFuzzer targets (`fuzz/`, cargo-fuzz, overflow checks and AddressSanitizer on): the Token-2022 mint policy, the token-account reader and the guardian-set parser Floor and the Vault share against independent statements of them; the Pyth `PriceUpdateV2` parser against the documented layout; load-store-load round trips of every Vault and Floor account; the loader-v3 parsers and snapshot identity; the limiter at full width (exact `available_now`, instant and queue rules, errors change nothing, window changes keep the used fraction); the ledger (backed ≤ requested, the same amount on a re-ask, `debit_backed` accepts it). Line coverage is measured (`cargo fuzz coverage`); seeds in `fuzz/seeds/` reach what a mutator rarely builds | 6 Oct: 10 min per target, 36M–277M executions each; one harness-oracle crash (finding 40), no program bug; every targeted parser at 100% of lines after the seeds (finding 41). That run had 8 targets: the guardian-set parser's, added 14 Oct, has no recorded 10-minute run or coverage measurement. CI: 60 s per target | | **Mutation testing** | `cargo-mutants` on the core: 669 mutants (§6), and in CI on every pull request's changed lines | Every non-equivalent mutant killed; 22 equivalents, each explained and excluded by exact pattern | -| **Program tests** | Floor (43) and Vault (80) on the real agave runtime: LiteSVM, mainnet feature set, the mainnet token program | 123/123 | +| **Program tests** | Floor (45) and Vault (88) on the real agave runtime: LiteSVM, mainnet feature set, the mainnet token program | 133/133 | | **Account substitution** (`programs/tests/tests/common/sweep.rs`) | For every Vault and Floor instruction (28 of the Vault's 30, tags 0–29: all but the two read-only views, `attest` and `quote_release`; registration both by the caller PDA and by the upgrade authority; the Floor lifecycle end to end): a transaction that succeeds, then each account swapped in turn for a fresh address, every other account of the transaction, a second market's or a second protected program's accounts, a stranger's token accounts and buffers, other tickets, programs and sysvars, and for a signer every other known key and the key unsigned. Every swap must fail, except at the positions each program leaves free on purpose, each listed with its reason (a rent payer, a ticket account only used when part of a release queues, another guardian of the same set). The first version missed two of four bugs planted to test it (a settle that pays anyone, a cancel of another vault's ticket): it had no look-alike token account or ticket to offer. With them, all four are caught | 10 Oct, with a release to none signed by the owner (finding 95; with that check deleted, the sweep accepts the owner unsigned and fails): 4,499 Vault and 1,683 Floor substitutions (33 and 18 transactions), refused (9 Oct, with `close_buffer`, the signing release and the exit's account 6: 4,508 and 1,549; 8 Oct: 4,474 and 1,472). Found finding 53. Accepted and explained: a veto of a lifeboat exit where none is pending changes nothing (the next proposal clears it) | | **Program ↔ model differential** | Random operation sequences on-chain vs a host model built only from the core; invariants after every step (§7) | 12,000 steps across 24 seeds, identical; 24,000 steps across 24 seeds after the A1 fix (29 Sep), identical | | **Wide program differential** | The same, with a limit configuration drawn per seed (floors and caps far above the balance, a closed lane, a queue delay past the horizon) and every instruction that changes what can leave: protocol and vault freezes, quorum unfreezes, ticket extensions, backlog resets, lifeboat moves both ways, limit changes; `quote_release` against the release, every `attest` field against the core (§7) | 5 Oct: 200,000 steps across 400 configurations, then 100,000 with limit changes after finding 30, identical | @@ -74,7 +74,7 @@ The programs are checked against the core as a model (§7). Two implementations ## 3. Commands ```bash -scripts/check.sh # fmt, clippy, core tests, golden + fixtures, 150k differential, model, +scripts/check.sh # fmt, the docs' stated counts, clippy, core tests, golden + fixtures, 150k differential, model, # floor_check self-test, JavaScript tests scripts/check-programs.sh # build the programs, every LiteSVM test, then clippy on the harness (E2E=1: also cloud/e2e) BULKHEAD_CASES=100000 cargo test --release -p bulkhead-core # deeper properties @@ -84,7 +84,7 @@ scripts/mutants.sh # mutation testing, release profile (~45 scripts/fuzz.sh # libFuzzer, every target 60 s in parallel (SECONDS_PER=600 for longer) (cd fuzz && cargo clippy --all-targets -- -D warnings) # lint the fuzz crate (its own workspace; CI's fuzz job) scripts/build-programs.sh # Floor, Vault, Harbor for sBPF v3; checks e_flags == 3 -(cd programs/tests && cargo test --release) # 123 program tests + the program-vs-model differentials +(cd programs/tests && cargo test --release) # 133 program tests + the program-vs-model differentials (cd programs/tests && BULKHEAD_PROGRAM_SEEDS=24 BULKHEAD_PROGRAM_STEPS=500 cargo test --release --test differential) cloud/e2e/run.sh # validator + guardian + attacker node --test cloud/test/*.test.mjs # client, policy, JS-vs-Python differential @@ -371,7 +371,7 @@ One workflow (`.github/workflows/ci.yml`) runs on every pull request, and at ful | Job | When | What | |---|---|---| -| `check` | Always | `scripts/check.sh`: fmt, clippy, the core's tests, golden vectors and fixtures, the Python-vs-Rust differential (300k cases; 3M weekly), the model, the JavaScript tests | +| `check` | Always | `scripts/check.sh`: fmt, the counts README and docs state against the tree (`tools/claims.py`), clippy, the core's tests, golden vectors and fixtures, the Python-vs-Rust differential (300k cases; 3M weekly), the model, the JavaScript tests | | `programs` | The programs, the core, Cloud, the reference | Build (sBPF v3), LiteSVM tests (including the account-layout contract with `cloud/lib/solana.mjs`), clippy on the test harness, the end-to-end run, the deploy rehearsal; the binaries and `SHA256SUMS` kept 14 days | | `scenarios` | The guardian, its libraries, the e2e scripts | `cloud/e2e/guardian-scenarios.sh`: the guardian on a real validator, one scenario per bug it once had (the incident hold across a restart, a restart after a quorum's unfreeze, a dead primary RPC, a thief proposing ahead of an approved change, a replaced recovery key; weekly also a forgotten cursor) | | `fuzz` | The fuzz targets, the core, the Vault and Floor | Every libFuzzer target in parallel, 60 s each (25 min weekly); clippy on the fuzz crate as the last step; the corpus carries over between runs, and the weekly run minimizes it | diff --git a/docs/FAQ.md b/docs/FAQ.md index f635fc6..1967bf7 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -81,7 +81,7 @@ No. Tokens sit in accounts owned by the Vault program's PDA. No key can transfer **What if Bulkhead has a bug?** It is the top risk, so every rule is stated independently more than once and checked: - a Rust core against a Python reference, over 6.9M differential lines; -- property tests, 15 Kani harnesses, exhaustive tests and mutation testing; +- property tests, 17 Kani harnesses, exhaustive tests and mutation testing; - the programs against a model built from the core; - every finding listed in `09 §4`. diff --git a/scripts/check.sh b/scripts/check.sh index f29ff0f..2fd2128 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -10,6 +10,9 @@ export PATH="$HOME/.cargo/bin:$PATH" CASES="${CASES:-50000}" echo "== fmt"; cargo fmt --all -- --check +echo "== claims (the counts README and docs/ state, against the tree)" +python3 tools/claims.py --self-test +python3 tools/claims.py echo "== clippy"; cargo clippy --all-targets --release -- -D warnings echo "== tests"; BULKHEAD_CASES="${BULKHEAD_CASES:-5000}" cargo test --release --quiet echo "== golden"; python3 reference/golden.py --check diff --git a/tools/claims.py b/tools/claims.py new file mode 100644 index 0000000..baf6a22 --- /dev/null +++ b/tools/claims.py @@ -0,0 +1,468 @@ +#!/usr/bin/env python3 +"""claims: the counts the docs state, checked against the tree. + + python3 tools/claims.py # fail if any stated count is stale + python3 tools/claims.py --self-test # the matcher on planted drift + +The Markdown files, the CI workflow and the status page state counts that go +stale as the code moves: +Kani harnesses, tests per suite, golden vectors, fuzz targets, findings. Each +count here is computed from the files themselves (a `#[kani::proof]`, a +`#[test]`, a row of `09 §4`), and each phrasing of it listed in RULES must +equal it wherever it appears, across line breaks too. A count stated in a new +phrasing is not checked until a rule names it; a rule that matches nothing +fails, so a reworded sentence can't leave its rule checking nothing. A test +counted here is a test function in the source that is not `#[ignore]`d; `09 §3` +gives the commands that run them. + +A statement that is not of the current count (a dated past state, a subset) is +listed in KEPT with its reason and left as written. An entry there must match +exactly once: one that matches nothing fails, so the list can't go stale, and +one that matches twice would hide a current statement. + +Standard library only. +""" +from __future__ import annotations + +import os +import argparse +import re +import subprocess +import sys +import time + +ROOT = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")) +CORE = "crates/bulkhead-core" + + +def read(path: str) -> str: + with open(os.path.join(ROOT, path), encoding="utf-8") as f: + return f.read() + + +def files(directory: str, suffix: str) -> list: + """Every file under `directory` (recursively, never into build output) + ending in `suffix`, sorted.""" + out = [] + for d, dirs, names in os.walk(os.path.join(ROOT, directory)): + dirs[:] = [x for x in dirs if x not in {".git", "target", "node_modules"}] + out += [os.path.relpath(os.path.join(d, n), ROOT) for n in names if n.endswith(suffix)] + return sorted(out) + + +# A string (raw or not) or char literal, or a comment: matched as one token, +# so a "/*" in a string or a '"' never starts anything. (Rust nests block +# comments; this reads the first `*/` as the end, so don't nest them around tests.) +LITERAL_OR_COMMENT = re.compile(r'b?r(#*)".*?"\1|"(?:[^"\\]|\\.)*"|\'(?:[^\'\\\n]|\\(?:x[0-9a-fA-F]{2}|u\{[0-9a-fA-F]+\}|.))\'|//[^\n]*|/\*.*?\*/', re.S) +FN = re.compile(r'(?:pub(?:\([^)]*\))?\s+)?(?:(?:async|const|unsafe|extern(?:\s+"[^"]*")?)\s+)*fn\s+(\w+)') + + +def code(source: str) -> str: + """Rust source without its comments (doc comments and their examples too) + and literals (so no attribute holds a quoted bracket or word).""" + return LITERAL_OR_COMMENT.sub("", source) + + +def functions(source: str) -> list: + """(attributes, name) of each function that has attributes. An attribute's + brackets may nest to any depth; anything but whitespace between two + attributes, or between them and the `fn`, ends the run.""" + src, out, attrs, end = code(source), [], [], 0 + for m in re.finditer(r"#\[|\b(?:pub|async|const|unsafe|extern|fn)\b", src): + if m.start() < end: + continue + if src[end:m.start()].strip(): + attrs = [] + if m.group(0) == "#[": + depth, j = 0, m.start() + 1 + while j < len(src): + depth += {"[": 1, "]": -1}.get(src[j], 0) + if depth == 0: + break + j += 1 + attrs.append(src[m.start():j + 1]) + end = j + 1 + else: + f = FN.match(src, m.start()) + if f and attrs: + out.append((attrs, f.group(1))) + attrs, end = [], f.end() if f else m.end() + return out + + +def tests_in(source: str) -> int: + """Test functions that run by default: one marked `#[ignore]` is not counted. + A function's attributes may come in any order; comments don't count. + This reads the source, so a test under a `#[cfg]` counts whatever the cfg: + after adding one, compare with what `cargo test` runs.""" + return sum( + any(re.fullmatch(r"#\[\s*test\s*\]", a) for a in attrs) and not any(re.match(r"#\[\s*ignore\b", a) for a in attrs) + for attrs, _ in functions(source) + ) + + +def kani_names(source: str) -> list: + """The function name of each Kani harness, in order: `#[kani::proof]` or + `#[kani::proof_for_contract(..)]`, either under a `cfg_attr`; not in a comment.""" + proof = re.compile(r"#\[\s*(?:cfg_attr\(.*,\s*)?kani::proof(?:_for_contract)?\b", re.S) + return [name for attrs, name in functions(source) if any(proof.match(a) for a in attrs)] + + +def kani_table(text: str) -> tuple: + """`09 §5.1`: the harness names its table lists, and how many are SMALL-WIDTH.""" + if "\n### 5.1 " not in text or "\n### 5.2 " not in text: + raise SystemExit("claims: 09 has no '### 5.1 ' or '### 5.2 ' heading; update kani_table()") + section = text.split("\n### 5.1 ", 1)[1].split("\n### 5.2 ", 1)[0] + rows = re.findall(r"^\| `(\w+)` \| ([A-Z-]+)", section, re.M) + return sorted(name for name, _ in rows), sum(domain == "SMALL-WIDTH" for _, domain in rows) + + +JS_LITERAL_OR_COMMENT = re.compile(r"""("|')(?:(?!\1)[^\\\n]|\\.)*\1|`(?:[^`\\]|\\.)*`|//[^\n]*|/\*.*?\*/""", re.S) + + +def js_tests_in(source: str) -> int: + """`test(` or `it(` calls at the start of a line, less those skipped or todo + on that line; not in a comment or a string. A test registered in a loop + counts once, and options on a later line are not read: write such tests out.""" + # A string becomes its empty self (`skip: ""` still reads as skipped); a + # comment or a template literal, which may span lines, goes. + src = JS_LITERAL_OR_COMMENT.sub(lambda m: 2 * m.group(1) if m.group(1) else "", source) + return sum( + bool(re.match(r"\s*(?:test|it)\(", line)) and not re.search(r"""\b(?:skip|todo)\s*:\s*(?:true|['"])""", line) + for line in src.splitlines() + ) + + +def js_tests(*paths: str) -> int: + return sum(js_tests_in(read(p)) for p in paths) + + +def rust_tests(*paths: str) -> int: + return sum(tests_in(read(p)) for p in paths) + + +def findings(text: str) -> int: + """Rows of the `09 §4` table, which must be numbered 1..n without a gap.""" + if "\n## 4. " not in text or "\n## 5. " not in text: + raise SystemExit("claims: 09 has no '## 4. ' or '## 5. ' heading; update findings()") + section = text.split("\n## 4. ", 1)[1].split("\n## 5. ", 1)[0] + ids = [int(m) for m in re.findall(r"^\| ?(\d+) ?\|", section, re.M)] + for want, got in zip(range(1, len(ids) + 1), ids): + if got != want: + raise SystemExit(f"claims: 09 §4 is not numbered 1..n: row {want} is numbered {got}") + return len(ids) + + +# Each test file the docs count, by the count it feeds; and each one they +# don't, with the reason. A test file in neither list fails, so a new suite +# can't go uncounted. +TEST_FILES = { + f"{CORE}/tests/properties.rs": "property", + f"{CORE}/tests/exhaustive.rs": "exhaustive", + f"{CORE}/tests/floor.rs": "floor_core", + f"{CORE}/tests/units.rs": "targeted", + f"{CORE}/tests/adversarial.rs": "adversarial", + f"{CORE}/tests/cascade.rs": "cascade", + "programs/tests/tests/floor.rs": "floor_programs", + "programs/tests/tests/vault.rs": "vault_programs", +} +UNCOUNTED = { + "programs/tests/tests/differential.rs": "a program-vs-model differential, reported in steps (09 §2)", + "programs/tests/tests/differential_wide.rs": "a program-vs-model differential, reported in steps (09 §2)", + "programs/tests/tests/floor_differential.rs": "a program-vs-model differential, reported in steps (09 §2)", + "programs/tests/tests/curve_vectors.rs": "an #[ignore]d generator of the curve vectors, run by hand", + "programs/tests/tests/e2e_world.rs": "an #[ignore]d builder of the end-to-end world, run by hand", +} + + +def truth() -> dict: + """Every count a claim can name, computed from the tree.""" + # What cargo builds as a test target: tests/x.rs and tests/x/main.rs. + found = [ + p + for d in (f"{CORE}/tests", "programs/tests/tests") + for p in files(d, ".rs") + if os.path.dirname(p) == d or (os.path.basename(p) == "main.rs" and os.path.dirname(os.path.dirname(p)) == d) + ] + unmapped = sorted(set(found) - set(TEST_FILES) - set(UNCOUNTED)) + if unmapped: + raise SystemExit(f"claims: test files in neither TEST_FILES nor UNCOUNTED: {unmapped}") + t = {key: 0 for key in TEST_FILES.values()} + for path, key in TEST_FILES.items(): + t[key] += rust_tests(path) + t.update({ + # What `golden.py --check` compares: one output line per case, one block per fixture. + "golden": len([l for l in read("reference/golden/expected.txt").splitlines() if l.strip()]), + "regressions": len(re.findall(r"^== ", read("reference/golden/regressions-expected.txt"), re.M)), + "client": js_tests(*files("cloud/test", ".test.mjs")), + "fuzz": len(re.findall(r"^\[\[bin\]\]", read("fuzz/Cargo.toml"), re.M)), + "findings": findings(read("docs/09-VERIFICATION.md")), + }) + t["programs"] = t["floor_programs"] + t["vault_programs"] + names = sorted(n for p in files(f"{CORE}/src", ".rs") for n in kani_names(read(p))) + t["kani"] = len(names) + listed, t["kani_small"] = kani_table(read("docs/09-VERIFICATION.md")) + if names != listed: + missing, extra = sorted(set(names) - set(listed)), sorted(set(listed) - set(names)) + raise SystemExit(f"claims: 09 §5.1 must list each Kani harness once; missing {missing}, not harnesses {extra}") + t["kani_full"] = t["kani"] - t["kani_small"] + return t + + +# (pattern, the count each group must equal). Patterns are matched in every +# document, so a new sentence in an old phrasing is checked too. A space in a +# pattern matches any whitespace, a line break included; a number may have +# thousands separators. A result "passed/total" must be all passed: a run with +# failures is a red CI, not a state the docs describe. After raising a total, +# run the check before writing the new result. +D = r"(\d{1,3}(?:,\d{3})+|\d+)" +WHOLE = r"(? list: + """Every Markdown file in git's index (so a new doc counts once added, and + local notes never do), the CI workflow and the status page.""" + try: + out = subprocess.run( + ["git", "ls-files", "-z", "--cached", "--", "*.md", ".github/workflows/*.yml", "cloud/status/*.html"], + cwd=ROOT, check=True, capture_output=True, text=True, + ).stdout.split("\0") + except (OSError, subprocess.CalledProcessError): # not a git checkout (a source archive): every file + out = files(".", ".md") + out += files(".github/workflows", ".yml") + files("cloud/status", ".html") + return sorted(p for p in set(out) if p and os.path.exists(os.path.join(ROOT, p))) + + +def check(texts: dict, t: dict, kept_list: list) -> tuple: + """Every stated count that differs from `t`, as `file:line: message`, and + the rules that matched outside the kept statements.""" + errors, live, used, found_any = [], set(), set(), set() + kept = {} + for path, snippet, _ in kept_list: + flexible = WHOLE + r"\s+".join(re.escape(word) for word in snippet.split()) + r"(?!\w)" + found = list(re.finditer(flexible, texts.get(path, ""))) + hits = [(m.start(), m.end()) for m in found] + if hits: + found_any.add((path, snippet)) + if len(hits) != 1: + errors.append(f"{path}: kept entry {snippet!r} matches {len(hits)} times; it must name exactly one statement") + kept.setdefault(path, []).extend((a, b, snippet) for a, b in hits) + for path, text in texts.items(): + for pattern, keys in RULES: + for m in re.finditer(pattern, text): + # A kept statement exempts only the numbers inside it: another + # count the same rule matches (a current one in the same table + # row) is still checked. + check_keys = [] + for i, key in enumerate(keys, 1): + inside = [s for a, b, s in kept.get(path, []) if a <= m.start(i) and m.end(i) <= b] + if inside: + used.update((path, s) for s in inside) + else: + check_keys.append((i, key)) + if not check_keys: + continue + live.add(pattern.pattern) + wrong = [ + f"{m.group(i)} (the tree has {key} = {t[key]})" + for i, key in check_keys + if int(m.group(i).replace(",", "")) != t[key] + ] + if wrong: + line = text.count("\n", 0, m.start()) + 1 + errors.append(f"{path}:{line}: {m.group(0)!r} states " + ", ".join(wrong)) + for path, snippet, _ in kept_list: + if (path, snippet) in found_any and (path, snippet) not in used: + errors.append(f"{path}: kept entry {snippet!r} holds no statement a rule checks; remove it") + return errors, live + + +def self_test() -> int: + """The matcher and the test counter on planted text: right counts pass, a wrong one fails.""" + t = {key: 10 for _, keys in RULES for key in keys} + t.update(kani=17, kani_full=14, kani_small=3, floor_programs=4, vault_programs=6, findings=1103) + old = [("doc.md", "14 Kani harnesses", "a past state")] + cases = [ + ("15 Kani harnesses", [], 1), + ("15 Kani\n harnesses", [], 1), + ("17 Kani harnesses", [], 0), + ("### 5.1 Kani harnesses (17)", [], 0), + ("badge/Kani_proofs-17%2F15-x", [], 1), + ("badge/Kani_proofs-16%2F17-x", [], 1), + ("17/17 verified (14 for every value of their types)", [], 0), + ("(15 for every value of their types)", [], 1), + ("**17/17 harnesses verified**", [], 0), + ("| **Bounded proofs (Kani)** | 17 harnesses: x | **16/17 verified** (14 Oct) |", [], 1), + ("| **Bounded proofs (Kani)** | 17 harnesses: x | **17/18 verified** (14 Oct) |", [], 1), + ("| **Program tests** | Floor (4) and Vault (6) on x | 10/10 |", [], 0), + ("| **Program tests** | Floor (4) and Vault (6) on x | 10/11 |", [], 1), + ("| **Program tests** | Floor (5) and Vault (6) on x | 10/10 |", [], 1), + ("we found 1,103 findings", [], 0), + ("we found 1,102 findings", [], 1), + ("96–1,103 were found after the submission", [], 0), + # A sub-range (which of them are fixed) must not read as the latest. + ("96–99, 101 and 103 are fixed; 96–1,103 were found after the submission", [], 0), + ("96–99, 101 and 103 fixed; 1 at submission; 96–1,103 recorded 13–14 Oct", [], 0), + ("96–1,102 were\nfound after the submission", [], 1), + ("96-1,102 were found after the submission", [], 1), + ("196–1,102 were found after the submission", [], 0), + ("1 at submission; 96–1,103 recorded 13–14 Oct", [], 0), + ("1 at submission; 96–1,102 recorded 13–14 Oct", [], 1), + ("1 at submission; 96-1,102 recorded 13–14 Oct", [], 1), + ("1 at submission; 196–1,102 recorded 13–14 Oct", [], 0), + ("findings 96–1,102 recorded on devnet", [], 0), + ("1 at submission;\n 96–1,102 recorded", [], 1), + ("95 findings at submission; 96–1,102 recorded", [("doc.md", "95 findings at submission", "x")], 1), + ("then: 14 Kani harnesses", old, 0), + ("then: 14 Kani harnesses; now: 14 Kani harnesses", old, 1), + ("then: 14 Kani harnesses", [("doc.md", "then:", "x")], 2), + ("then: 14 Kani\nharnesses", old, 0), + ("then: 114 Kani harnesses", old, 2), + # A kept dated result does not hide a stale current count beside it. + ("| **Bounded proofs (Kani)** | 16 harnesses: x | **16/16 verified** (8 Oct) |", [("doc.md", "16/16 verified", "x")], 1), + ("| **Bounded proofs (Kani)** | 17 harnesses: x | **16/16 verified** (8 Oct) |", [("doc.md", "16/16 verified", "x")], 0), + ("Bounded model checking of the core's 15 harnesses", [], 1), + ("the 3 harnesses at small width", [], 0), + ("the S10 Floor tests and G2 property tests", [], 0), + ("now: 15 Kani harnesses", old, 2), + ] + failed = [] + for text, kept, want in cases: + got, _ = check({"doc.md": text}, t, kept) + if len(got) != want: + failed.append(f"{text!r}: {len(got)} errors, expected {want}: {got}") + if check({"doc.md": "then: 14 Kani harnesses"}, t, old)[1]: + failed.append("a rule matched only inside a kept statement counts as live") + if not check({"doc.md": "17 Kani harnesses"}, t, [])[1]: + failed.append("a rule that matched does not count as live") + sources = [ + ("#[test]\nfn a() {}\n#[test] fn b() {}\n", 2), + ("#[test]\n/// says why\n// and how\n/* and\n more */\nfn a() {}\n", 1), + ("#[test]\n#[ignore]\nfn a() {}\n#[ignore = \"slow\"]\n#[test]\nfn b() {}\n", 0), + ("#[test]\n#[should_panic(expected = \"index [0] must ignore\")]\n#[cfg_attr(miri, ignore)]\nfn a() {}\n", 1), + ("#[test]\npub(crate) async fn a() {}\n#[test]\nunsafe fn b() {}\n", 2), + ("#[cfg(test)]\nfn helper() {}\nfn b() {} // #[test]\n", 0), + ("#[test]\n#[case([1, 2])]\nfn a() {}\n", 1), + ("#[test]\n#[case(&[[1, 2], [3]])]\nfn a() {}\n#[test]\nlet x = 1;\nfn b() {}\n", 1), + ("#[derive(Debug)]\npub struct S;\nfn not_a_test() {}\n", 0), + ("/*\n#[test]\nfn old() {}\n*/\n/// #[test] fn x() {}\nstruct S;\n", 0), + ("const A: char = '\"';\n#[test]\nfn a() {}\nconst B: char = '\"';\n", 1), + ("const A: [char; 2] = ['\\x00','\"'];\n#[test]\nfn a() {}\nconst B: &str = \"x\";\n", 1), + ('const A: &str = r"\\";\n#[test]\nfn a() {}\nconst B: &str = "x";\n', 1), + ] + for source, want in sources: + if tests_in(source) != want: + failed.append(f"{source!r}: {tests_in(source)} tests, expected {want}") + harnesses = [ + ("#[kani::proof]\nfn a() {}\n#[kani::proof_for_contract(f)]\nfn b() {}\n#[cfg_attr(kani, kani::proof)]\nfn c() {}\n", 3), + ("// #[kani::proof]\n/* #[kani::proof]\n#[kani::proof] */\n#[kani::unwind(4)]\nfn a() {}\n", 0), + ('let s = "/*";\n#[kani::proof]\nfn a() {}\n// */\n', 1), + ] + for source, want in harnesses: + if len(kani_names(source)) != want: + failed.append(f"{source!r}: {len(kani_names(source))} harnesses, expected {want}") + names = kani_names("#[kani::proof]\n#[kani::unwind(3)]\nfn a() {}\n#[cfg_attr(kani, kani::proof)]\npub fn b() {}\n") + if names != ["a", "b"]: + failed.append(f"harness names {names}, expected ['a', 'b']") + table = "\n### 5.1 K\n| `a` | FULL | x |\n| `b` | SMALL-WIDTH (x) | y |\n| `c` | FULL (every byte) | z |\n### 5.2 E\n" + if kani_table(table) != (["a", "b", "c"], 1): + failed.append(f"09 §5.1 table read as {kani_table(table)}, expected (['a', 'b', 'c'], 1)") + js = [ + ("test('a', () => {});\n it('b', () => {});\ntest('c', { skip: true }, () => {});\n", 2), + ("test('d', { todo: 'later' });\n// test('e')\nconst f = test('g');\n", 0), + ("test('h', { skip: false }, () => {});\n", 1), + ("/*\ntest('old', () => {});\n*/\n", 0), + ("const g = 'cloud/test/*.mjs';\ntest('a', () => {});\ntest('b', () => { /* x */ });\n", 2), + ("const py = `\ntest(1)\n`;\ntest('c', { skip: \"why\" }, () => {});\n", 0), + ] + for source, want in js: + if js_tests_in(source) != want: + failed.append(f"{source!r}: {js_tests_in(source)} JavaScript tests, expected {want}") + start = time.monotonic() + tests_in("#[derive(Debug)]\n" + "// see https://x" * 40 + "\nstruct S;\n") + if time.monotonic() - start > 5: + failed.append("counting tests backtracks on a line with many `//`") + for pattern, keys in RULES: + if pattern.groups != len(keys): + failed.append(f"{pattern.pattern!r} has {pattern.groups} groups for {len(keys)} counts") + for f in failed: + print("self-test FAILED:", f) + if not failed: + print(f"self-test OK: {len(cases)} claim cases, {len(sources) + len(harnesses) + len(js)} count cases, {len(RULES)} rules") + return 1 if failed else 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description="Check the counts the docs state against the tree.") + ap.add_argument("--self-test", action="store_true", help="check the matcher and the counters on planted text") + if ap.parse_args().self_test: + return self_test() + t = truth() + texts = {p: read(p) for p in documents()} + errors, live = check(texts, t, KEPT) + # A rule whose phrasing left the docs checks nothing: reword the rule too. + errors += [f"rule {p.pattern!r} matches no current statement" for p, _ in RULES if p.pattern not in live] + for e in errors: + print(e) + if errors: + print(f"claims: {len(errors)} problems; a stated count must equal the tree") + return 1 + print(f"claims OK ({len(texts)} files): " + ", ".join(f"{k} {v}" for k, v in sorted(t.items()))) + return 0 + + +if __name__ == "__main__": + sys.exit(main())